TLDR: The U.S. Food and Drug Administration’s recently launched generative AI tool, Elsa, intended to streamline clinical reviews, is reportedly hallucinating non-existent studies and misrepresenting real research, according to current and former FDA employees. Despite these concerns, FDA leadership maintains that the tool’s use is voluntary and they have not heard specific complaints.
The U.S. Food and Drug Administration (FDA) is facing scrutiny over its new generative AI tool, Elsa, which was unveiled last month with the aim of accelerating clinical protocol reviews and enhancing scientific evaluations. However, current and former FDA employees have raised significant concerns, telling CNN that Elsa frequently ‘hallucinates’ entire studies that do not exist or misrepresents actual research.
Sources within the agency describe Elsa’s output as confidently inaccurate. One employee stated, ‘Anything that you don’t have time to double-check is unreliable,’ adding, ‘It hallucinates confidently.’ This poses a considerable challenge for a tool designed to expedite critical processes and aid in making informed decisions that directly impact patient health.
Elsa, a large language model (LLM), was developed by Deloitte on Anthropic’s Claude platform within Amazon Web Services’ secure GovCloud environment. Its agency-wide rollout in June was reportedly ahead of schedule, following initial enthusiasm from FDA Commissioner Marty Makary, who had previously been ‘blown away by the success of [the agency’s] first AI-assisted scientific review pilot.’ FDA Chief AI Officer Jeremy Walsh even characterized Elsa’s launch as the ‘dawn of the AI era at the FDA.’
Despite these high hopes, employees have criticized the implementation as rushed. Reports indicate that Elsa has struggled with simple tasks, provided incorrect or partially accurate responses, and ultimately has not saved employees significant time. While the FDA emphasizes a ‘human in the loop’ approach, requiring reviewers to verify citations and sources, the pervasive inaccuracies raise questions about the tool’s practical utility.
Also Read:
- NIH Implements New Policy to Curb AI-Generated Grant Applications Amid Surging Submissions
- European Union’s Landmark AI Act Takes Effect: A New Era for Responsible AI Development
FDA Commissioner Marty Makary, when confronted with these concerns by CNN, stated, ‘I have not heard those specific concerns.’ He also underscored that the use of Elsa and participation in its training are currently voluntary within the agency. This comes as the White House simultaneously introduced an ‘AI Action Plan,’ advocating for accelerated AI development and the removal of ‘red tape and onerous regulation’ in the sector. The plan also calls for AI to be free of ‘ideological bias,’ which critics suggest could lead to the omission of crucial public health topics like climate change, misinformation, and diversity, equity, and inclusion efforts. Such directives further cast doubt on the ability of tools like Elsa to genuinely benefit the FDA and American patients.
