TLDR: The legal profession is grappling with the widespread adoption of generative artificial intelligence (AI) tools, leading to a surge in errors such as fabricated case citations and unreliable evidence. Judges and legal experts are expressing alarm over these mistakes, which have resulted in significant fines and sanctions. While some courts have begun implementing guidelines for AI use, a lack of uniform regulations across jurisdictions creates confusion. The increasing reliance on AI is partly driven by attorney burnout and heavy workloads, highlighting the urgent need for comprehensive oversight, transparency, and human review to maintain ethical standards and public trust in the justice system.
The legal landscape is currently navigating a turbulent period marked by the rapid integration of generative artificial intelligence (AI) tools, a development that has been met with both enthusiasm and growing apprehension. Recent incidents across the United States have brought to light a concerning trend of AI-generated errors in legal filings, including fabricated case citations and unreliable evidence, leading to substantial financial penalties and court sanctions.
This surge in AI adoption is largely attributed to the immense pressure faced by attorneys, particularly junior lawyers, who are contending with heavy workloads and stringent deadlines. Attorney Harshita Ganesh, an advisor on AI policy for her firm, noted that many associates are ‘burnt out’ and ‘exhausted,’ often turning to generative AI as a ‘shortcut’ to manage their demanding schedules. This reliance, however, can compromise the traditional multi-layered review process involving associates, paralegals, and partners, which ‘falls apart’ when legal professionals are stretched too thin.
Courts nationwide are responding to these challenges, albeit with varying degrees of clarity. While some jurisdictions have started issuing orders to define the permissible uses of generative AI in litigation, many others have yet to provide clear guidance, resulting in a ‘patchwork of expectations.’ Tennessee Attorney General Jonathan Skrmetti likened AI to a ‘high-energy, if somewhat naive, associate,’ emphasizing the critical need for transparency and robust human oversight. Skrmetti anticipates that as AI tools become more sophisticated and reliable, with built-in mechanisms to prevent ‘hallucinated cases,’ courts may adopt a more hands-off approach to how lawyers produce written work.
However, the current reality is far from this ideal. In May 2025, a California federal judge imposed a $31,000 fine on the law firm Ellis George after a filing contained false citations generated by Google Gemini and another legal AI tool. Similarly, the AI company Anthropic admitted that its Claude model fabricated a citation in a copyright lawsuit. Last month, a federal judge in Colorado ordered two lawyers representing MyPillow CEO Mike Lindell to pay $3,000 each for submitting a brief riddled with over two dozen ‘hallucinated case citations,’ noting their failure to provide a credible explanation or be forthcoming about their AI use. A prominent incident from late 2023 involved Michael Cohen, former attorney for Donald Trump, who unwittingly submitted AI-generated fake legal cases from Google Bard, believing it to be a ‘super-charged search engine’ rather than a generative text service capable of fabrication.
Judicial figures and bar regulators are actively weighing in on the issue. In 2023, U.S. District Judge Brantley Starr of Texas took a firm stance, mandating that all filings in his court must either be AI-free or meticulously reviewed by a human lawyer using ‘print reporters or traditional legal databases.’ Judge Starr issued a stark warning, stating that AI is ‘unbound by any sense of duty, honor, or justice,’ and threatened sanctions under Rule 11 of the Federal Rules of Civil Procedure for violations. David Coale, a constitutional law attorney, cautioned that beyond fabricated citations, the ‘next frontier’ of danger lies in ‘hallucinated facts,’ particularly from clients in emotionally charged cases like child custody disputes.
Legal ethicist Irina Raicu of Santa Clara University’s Markkula Center for Applied Ethics pointed to the marketing of AI tools as a contributing factor, suggesting that it often ‘downplays those limitations,’ leading many, including lawyers, to misunderstand generative AI’s capabilities. Raicu also highlighted a July 2023 opinion from the American Bar Association outlining basic responsibilities for ethical AI use, but stressed that ‘professional guidance must go deeper,’ citing concerns about AI models’ ‘sycophancy’ potentially endangering pro se litigants.
Also Read:
- Navigating the Veracity Challenge: AI Hallucinations in Legal Practice and the Path to Reliable Integration
- Singapore Pioneers Agentic AI for Corporate Compliance and Launches Advanced GenAI Search for Legal Professionals
Despite the challenges, a 2025 survey by the National Center for State Courts and the Thomson Reuters Institute revealed that 55% of court professionals view AI as having a ‘transformational’ impact on operations. However, only 17% of courts currently utilize generative AI, and a significant 70% prohibit their staff from using such tools. Harshita Ganesh underscored that judges and their clerks are ‘human,’ implying that AI-induced errors could potentially be missed. She, along with other experts, agrees that AI is an ‘inevitable’ part of the legal future, necessitating collective guidelines and responsible integration. The legal community finds itself in ‘uncharted territory,’ currently in a ‘data collection phase’ as it adapts to these rapid technological shifts.
