The rapid integration of generative artificial intelligence (GenAI) into the professional practice of the legal industry has brought forth several instances of attorneys, experts, and judges knowingly or unknowingly submitting AI hallucinations in their work. This review focuses on a curated selection of such instances from across the United States that have “made the news,” and discusses how the courts are proposing to curtail the submission of AI hallucinations.
Currently, there is no explicit guidance for federal courts on how to address the misuse of GenAI, let alone address the submission of AI hallucinations in court filings. Broadly speaking, courts have applied existing rules and laws to attorney, expert, and judge misuse of AI. However, individual judges have taken their own approaches to punishing attorneys who submit AI hallucinations and have inconsistently applied punishments. This approach indicates a necessity for courts to respond but an uncertainty in how to respond. In other words, courts know they must act but lack consensus on how to appropriately do so in response to the submission of AI hallucinations in court filings.
What Do the Federal Rules of Civil Procedure (FRCP) Say?
Presently, the courts are enforcing violations under FRCP 11(b), which in relevant part states, “[B]y presenting to the court a pleading, written motion, or other paper—whether by signing, filing, submitting, or later advocating it—an attorney or unrepresented party certifies that to the best of the person’s knowledge, information, and belief, formed after an inquiry reasonable under the circumstances… (2) the claims, defenses, and other legal contentions are warranted by existing law.”
What Did We Find?
Our review uncovered the following themes:
1. Governance Through Existing Rules
The courts have mostly shoehorned the submission of AI hallucinations into other rules or laws, largely stemming from the attorney’s professional responsibility. The best example is FRCP 11(b).
2. Honesty Is the Best Policy
Courts seem to respond more harshly to the submission of AI hallucinations when use of GenAI is evident but such use is denied by the attorney. In contrast, when up front and apologetic, attorneys receive lesser punishments.
3. Proportionality of Actions
A brief with a few erroneous citations or arguments will receive reduced punishment compared to many erroneous citations in a single brief or several errant briefs.
4. Most Common Hallucinations
There appear to be three categories of AI hallucinations (although others can occur, they are less frequent). The categories are:
- Citations to fictitious cases.
- Fabricated citations to real cases or documents.
- Citations to real quotes from real cases that fail to support or that directly contradict a proposed legal proposition.
Exemplary Sanctions for Filings from Plaintiffs, Defendants, Experts, and Judges/Clerks
Plaintiff’s Motions
In Dehghani v. Castro (U.S. District Court for the District of New Mexico), the petitioner’s attorney purchased a brief from a freelance attorney. The freelance attorney likely used GenAI and then destroyed all notes. The purchasing attorney did not review the purchased brief before submitting it to the court. The court issued two show cause orders, but the petitioner’s attorney did not adequately respond to either request to clarify and provide copies of the cases. The court issued a sanctions order requiring a fine to be paid to the court, mandatory CLE training, and self-reporting to all appropriate state bars of this misconduct for the petitioner’s attorney and the freelance attorney.
In In re Loletha Hale (U.S. District Court for the Northern District of Georgia, Atlanta Division), the court found that “[a]n overwhelming majority of the cases cited by [the plaintiff’s attorney] either did not exist, did not support the proposition for which they were cited, or misquoted the authority.” The attorney blamed her daughter for drafting the brief, who was neither an attorney nor a paralegal. The court issued a sanctions order requiring her to notify all existing clients of the court’s decision and file a notice with the court along with a copy of the order in all pending and future cases in the Northern District of Georgia in which she appears for five years from the date of the order.
In Dubinin v. Papazian (U.S. District Court for the Southern District of Florida), the plaintiff’s attorney filed a response brief with fabricated quotes and relied on a “decision from this Court that does not exist.” The attorney blamed a subordinate who was not a member of the bar in the state, who in turn blamed her legal assistant. The court issued a sanctions order dismissing the case without prejudice, requiring that the attorney pay the other parties’ attorney fees and referring all attorneys involved to their respective state bars for discipline.
In Tercero v. Sacramento Logistics, LLC, et al. (U.S. District Court for the Eastern District of California), the plaintiff’s attorney filed a motion for reconsideration that included “non-existent, patently manufactured, or entirely inapplicable” case citations. Specifically, plaintiff cited to “two cases that simply do not exist,”
“[t]en other cases… [that] do not contain the language quoted by Plaintiff,” and 12 cases that “do not support the propositions for which [plaintiff] proffers them.” Instead, the plaintiff’s counsel argued to the court that they did not use AI. The court found that the attorney was lying about how she did her work. Despite each of the lies being exposed and three opportunities to come clean being offered, the attorney continued to give excuses, even going so far as telling the opposing counsel they were wrong for noting the incorrect citations. The court issued a sanctions order requiring a $1,500 sanction to the court; mandating service by the attorney of a copy of the order on her client; and directing the clerk of the court to serve a copy of the order on the State Bar of California, of which the attorney was a member.
In Idehen v. Stoute-Phillip (Civil Court of the City of New York, Queens County), the petitioner’s attorney submitted an affirmation that “contained citation to seven fake cases.” In response, the attorney submitted a second filing with an appendix described as “an incoherent document that is eighty-eight pages long, has no structure, contains the full text of most of the cases cited, shows distinct indications that parts of the discussion/analysis of the cited cases were written by artificial intelligence, cites multiple cases that do not have anything to do with the issues presented in Respondent’s motion to dismiss, and cites to multiple cases for propositions that are not remotely supported by the case cited.” The court found that “[the attorney’s] conduct at the hearing and the Appendix to the June 2, 2025 Affirmation escalated his conduct from merely frivolous to egregious misconduct that implicates his honesty, trustworthiness, and fitness to practice law”, and that “had [the attorney] simply come to the hearing acknowledging that he had used Copilot to conduct his research, was unaware that the software could produce fake cases, and apologized for his mistake, the Court would have likely determined that a sanction was sufficient to address the frivolous conduct.” The attorney is under a show cause order.
Defendant’s Motions
In Johnson v. Dunn, et al. (U.S. District Court for Northern District of Alabama, Southern Division), the defendant cited AI hallucinations in two motions relating to seeking discovery from an incarcerated person. At least one of the defendant’s attorneys used ChatGPT to prepare the motions. The court issued a sanctions order disqualifying defendants’ attorneys from the case and referring the matter to the state bar. The judge noted that “[i]f fines and public embarrassment were effective deterrents, there would not be so many cases to cite. And in any event, fines do not account for the extreme dereliction of professional responsibility that fabricating citations reflects, nor for the many harms it causes. In any event, a fine would not rectify the egregious misconduct in this case.”
In Mid Cent. Operating Eng’rs Health and Welfare Fund v. Hoosiervac LLC (U.S. District Court for the Southern District of Indiana), the defendant’s attorney submitted three briefs that contained citations to cases that did not exist. The attorney admitted to using GenAI and that he made no effort to verify the cases cited by the AI. Instead of arguing that he did not violate Rule 11 of the FRCP, the attorney argued that monetary sanctions were “moot” because of the reputational damage he incurred as a result of his erroneous citations coming to light. The judge was unsympathetic, noting that it is up to the court to determine the punishment and that “sanctions payable to the Court cannot become moot because of external consequences.” However, the judge did sanction the attorney for only $6,000 rather than the full $15,000 recommended by a magistrate judge.
Expert Reports
In Concord Music Group, Inc., et al. v. Anthropic PBC (U.S. District Court for the Northern District of California), music publishers, including Concord Music Group, brought suit against Anthropic for infringing copyrights of the plaintiffs. To support a brief on the plaintiffs’ requests for prompts and outputs from Claude that relate to song lyrics, Anthropic’s expert submitted a report that included at least one citation to an article that was identified as an AI hallucination. The court struck the part of her expert report that included this erroneous citation and took note of her credibility. Anthropic explained that it was an “honest mistake” and that the citation must have been accidentally added when it ran her declaration through Claude for “formatting.” The court was displeased with the expert for citing an AI hallucination and with the attorney for submitting the expert’s report without checking for AI hallucinations. In other words, the court did not care about the severity of the mistake and noted that the expert, and Anthropic’s attorneys, still had a duty to check the declaration for accuracy.
In Kohls v. Ellison (U.S. District Court for the District of Minnesota), an expert, testifying about AI and deepfakes, used AI to write his expert report, which was filled with AI hallucinations. To support a motion for preliminary injunction, the plaintiff’s attorney submitted two expert reports giving background on AI and deepfake technology. One of the expert reports cited two fictitious articles and a third article that was misattributed. The expert admitted to using ChatGPT to draft his expert report, but said that the substantive parts were still accurate. The court pointed out the irony of an expert relying heavily on AI to comment on the dangers of doing so. The court further commented that regardless of whether the substantive parts of the report were accurate, the fact that the expert had submitted false information (the citations) under penalty of perjury was problematic, especially since “[the expert] typically validates citations with a reference software when he writes academic articles but did not do so when submitting the… legal filing.” The court further commented that “[the expert’s] citation to fake, AI-generated sources in his declaration—even with his helpful, thorough, and plausible explanation (ECF No. 39)—shatters his credibility with this Court.” The court commended the honesty that the expert exhibited in acknowledging his mistake and explaining how the mistake occurred, but the court could not ignore the fake citations as simple mistakes. The court excluded the problematic expert report and did not allow the party to correct or replace the excluded expert report prior to ruling on the preliminary injunction.
Judges and Law Clerks
Unfortunately, violations related to AI are not limited to parties, as some judges and their clerks have also been reprimanded for submitting orders and decisions with AI hallucinations in them. For example, Henry T. Wingate of the U.S. District Court for the Southern District of Mississippi and Julien Xavier Neals of the U.S. District Court for the District of New Jersey recently admitted that their offices used AI when preparing opinions that were found to contain AI hallucinations.
In Mississippi Association of Educators, et al. v. Board of Trustees of State Institutions of Higher Learning, et al. (U.S. District Court for the Southern District of Mississippi), Federal Judge Henry T. Wingate issued a temporary restraining order in a civil rights case in Mississippi on July 20, 2025. The defendants, led by the state attorney general, filed a motion raising concerns about the accuracy of the order, pointing out that the order misnamed the parties in the case, misquoted state law, and made factually inaccurate statements that were not supported by the record. A few days later, Judge Wingate replaced his ruling with a corrected version and refused to clarify what happened, stating that the issues were merely “clerical errors referencing improper parties and factual allegations.” He refused to allow the original ruling to remain available on the public docket. A few months later, Senator Chuck Grassley of the U.S. Senate Committee on the Judiciary sent an inquiry to Judge Wingate requesting clarification on what occurred with the original order and speculated on the use of AI. In response, Judge Wingate acknowledged that his law clerk had used Perplexity, an AI program, to write the order. Wingate further admitted to inadequately reviewing the order before issuing it. While Judge Wingate claims that he will take steps to ensure this does not happen again, there appear to be no additional corrective measures for Judge Wingate’s AI hallucination-laden order.
In In re CorMedix, Judge Neals issued an opinion on a motion to dismiss in a biopharma securities case on June 30, 2025. In July, the defendants filed a letter raising their concerns about errors in the opinion, including fictitious quotations and incorrect case outcomes cited. In August, Judge Neals withdrew the previous opinion and issued a new opinion without explanation. In response to a letter similar to that written by Senator Grassley to Judge Wingate, Judge Neals admitted that a law school intern in his office had used ChatGPT to draft the opinion and that he had inadequately reviewed the opinion. Similar to Judge Wingate, Judge Neals also committed to implementing preventive measures to ensure this does not happen again, with no additional corrective measures taken.
New and Proposed Rules to Curtail the Misuse of AI
The courts and the federal bar are taking action to deter submission of AI hallucinations and the misuse of AI in filings. For example, the bar has proposed changes to Federal Rule of Evidence (FRE) 707, which establishes uniform standards for introducing evidence generated or significantly modified by AI systems, such as outputs from machine learning models or deepfakes. This is a response to the current lack of specific rules for AI evidence, which has led to confusion and inconsistency in court. Proposed FRE 707 would subject machine-generated evidence to the expert witness admissibility standards under the existing FRE 702 (which includes the Daubert standard for reliability). To admit AI evidence under the proposed rule, the parties need to show:
- The evidence is helpful to the trier of fact (judge or jury).
- The AI tool relied on sufficient facts or data.
- The AI tool used reliable principles and methods.
- These principles and methods were reliably applied to the specific facts of the case.
The proposed rule relies on the court’s role, requiring judges to scrutinize underlying technology, data inputs, and validation studies to ensure evidence is reliable before it can be presented to a judge or jury. The proposed rule was approved for publication for public comment in June 2025 by the U.S. Judicial Conference’s Advisory Committee on Evidence Rules, such that it is under review and has not been finalized or implemented.
We also looked into how the top patent courts are addressing use of AI through local rules, some of which are being tracked.
Some specific courts have taken action to change local rules.
The Eastern District of Texas modified Local Rule CV-11(g): Use of Generative Artificial Intelligence Technology. Local Rule CV-11(g) now states, “All litigants remain responsible for the accuracy and quality of legal documents produced with the assistance of generative artificial intelligence technology. Litigants are cautioned that certain generative artificial intelligence technologies may produce factually or legally inaccurate content. If a litigant chooses to employ generative artificial intelligence technology, the litigant continues to be bound by the requirements of Fed. R. Civ. P. 11 and must review and verify all content to ensure that it complies with all such standards. See also Local Rule AT-3(m).”
In the Northern District of California, no new rules or procedures have been issued, but individual judges have standing orders requiring disclosure of when GenAI is used in court filings.
The Delaware Supreme Court issued interim guidelines for the use of GenAI in 2024.
The authors will continue to monitor this important issue to see how the U.S. Patent and Trademark Office, federal courts, the U.S. International Trade Commission, and other patent venues will handle these circumstances.
This article appeared in the 2025 AI Intellectual Property: Analysis & Trends Year in Review report.
Related Industries
Receive insights from the most respected practitioners of IP law, straight to your inbox.
Subscribe for Updates