Clinical Concerns About ELSA

So before we dive in let's look at the basics of ELSA.

What is ELSA?

ELSA is a large language model (LLM) powered AI tool designed to assist with reading, writing, and summarizing. It can summarize adverse drug events to support safety profile assessments, perform faster label comparisons, and generate code to help develop databases for nonclinical applications.
Source: FDA Launches Agency-Wide AI Tool to Optimize Performance for the American People | FDA

Key Facts About ELSA:

Launch: The U.S. Food and Drug Administration (FDA) launched ELSA, a generative Artificial Intelligence (AI) tool designed to help employees of the FDA, from scientific reviewers to investigators, work more efficiently. Source: FDA Launches Agency-Wide AI Tool to Optimize Performance for the American People | FDA

Timeline: The tool, called ELSA, is already being used in clinical protocol reviews, scientific evaluations, and to identify high-priority inspection targets, the FDA officially announced ELSA to the world on June 2, 2025. Source: FDA, facing 4% budget cut, launches generative AI tool Elsa a month ahead of schedule

Security: Built within the high-security GovCloud environment, ELSA offers a secure platform for FDA employees to access internal documents while ensuring all information remains within the agency. The model does not train on data submitted by the regulated industry, safeguarding the sensitive research and data handled by FDA staff. Source: FDA Launches Agency-Wide AI Tool to Optimize Performance for the American People | FDA

So, while ELSA offers promising advantages, its rapid deployment and the inherent complexities of AI in critical regulatory functions have raised significant concerns that demand a critical perspective from patients and stakeholders alike.

1. "Black Box" Problem and Lack of Transparency: A core concern with generative AI is its inherent "black box" nature, where the specific reasoning behind its outputs is not always transparent. As Harvard Medical School's Isaac Kohane highlights regarding AI in medicine, understanding how an AI arrives at its conclusions is crucial for trust and accountability. For patients, this means less visibility into how regulatory decisions, impacting their health, are influenced by an AI system. The FDA's claims of a secure GovCloud environment are important for data security, but not for algorithmic transparency.
2. Accuracy, Hallucinations, and Outdated Information: Reports from within the FDA, as noted by sources observing the rollout, indicate that generative AI tools like ELSA can sometimes provide inaccurate information, "hallucinate" details, or rely on outdated knowledge bases. This poses a significant risk in a regulatory context where precision is paramount. A misinterpretation or hallucination by ELSA, even if caught by a human reviewer, could lead to delays or, more critically, flawed regulatory decisions if not adequately cross-referenced.
3. Bias and Equity Concerns: AI systems learn from data, and if that data reflects historical biases (e.g., underrepresentation of certain demographic groups in clinical trials), the AI's outputs may perpetuate or even amplify those biases. Stanford University's Institute for Human-Centered AI, through initiatives like its policy brief on stronger FDA approval standards for AI medical devices, emphasizes the need for rigorous, multi-site evaluations and mandated post-market surveillance to detect and mitigate bias in AI. For ELSA, this translates to a risk that its analyses could inadvertently disadvantage certain patient populations if not meticulously monitored for equitable outcomes.
4. Limitations of AI in Nuance and Context: While ELSA excels at data summarization and pattern recognition, it lacks the human capacity for nuanced judgment, ethical consideration, and understanding of complex, real-world context. Regulatory decisions often require weighing ambiguous evidence, considering ethical implications, and applying discretion, tasks where human oversight remains indispensable. There are concerns that ELSA might be used for substantive scientific review rather than purely administrative tasks, potentially leading to a de-emphasis on critical human insight.
5. Long-term Validation and Continuous Monitoring: As highlighted by research on AI in healthcare, including perspectives from Mass General Brigham on a "clinical trials informed framework" for AI deployment, continuous, large-scale prospective evaluations are essential for AI applications. AI models can "drift" over time, meaning their performance can degrade or become biased as they encounter new data or evolve. The lack of continuous, rigorous validation of ELSA's performance after launch could lead to unintended consequences that risk patient well-being.
6. Accountability and Legal Challenges: The increasing reliance on AI tools raises complex questions about accountability. If an error stemming from ELSA's analysis leads to a detrimental outcome, where does the responsibility lie? Unresolved legal questions surrounding data ownership and Freedom of Information Act (FOIA) exposure related to AI-generated content also present a critical area for scrutiny.
7. Human-in-the-Loop Challenges: While the FDA stresses that ELSA is an "assistive" tool with human oversight, the reality of integrating AI into workflows can lead to over-reliance, automation bias (where humans trust AI outputs too readily), or a desensitization to errors. Maintaining a genuinely critical "human-in-the-loop" and ensuring that FDA staff remain empowered to challenge AI outputs will be crucial.

These are very real concerns, and this is why I am advocating for the patient's voice to be heard in the implementation phase of ELSA. I realize this is a heavy request, but I will let you know as this discussion progresses with the team at the FDA.