The U.S. Food and Drug Administration (FDA) officially launched its new generative Artificial Intelligence (AI) tool, ELSA (no official acronym has been released yet), on June 2, 2025. ELSA is designed to assist FDA staff in various core functions. ELSA represents a significant step in integrating AI into regulatory science. This series of briefs will summarize ELSA's purpose, its anticipated impact on patients, including myself, and provide a critical perspective on its advantages and disadvantages, incorporating scientific viewpoints from leading institutions such as Harvard Medical School, Massachusetts General Hospital, and Stanford University.

What is ELSA?

ELSA is an internal generative AI tool developed by the FDA. Its primary objective is to enhance the efficiency and speed of regulatory processes by assisting FDA staff with a range of tasks, including:

• Reviewing clinical protocols: Expediting the initial assessment of study designs.
• Summarizing adverse event reports: Quickly synthesizing large volumes of safety data.
• Comparing product labels: Identifying discrepancies or similarities across different product information.
• Identifying high-priority inspection targets: Using data to inform where regulatory oversight is most needed.

The FDA states that ELSA operates within a secure GovCloud environment and is designed not to train on industry-submitted data, aiming to maintain data integrity and security.

About Dan Noyes

Dan Noyes operates at the intersection of healthcare AI strategy and governance. After 25 years leading digital marketing strategy, he is transitioning his expertise to healthcare AI, driven by his experience as a chronic care patient and his commitment to ensuring AI serves all patients equitably. Dan holds AI certifications from Stanford, Wharton, and Google Cloud, grounding his strategic insights in comprehensive knowledge of AI governance frameworks, bias detection methodologies, and responsible AI principles. His work focuses on helping healthcare organizations implement AI systems that meet both regulatory requirements and ethical obligations—building governance structures that enable innovation while protecting patient safety and advancing health equity

Want help implementing responsible AI in your organization? Learn more about strategic advisory services at Viable Health AI

The Hallucination Problem Nobody Mentioned at the Launch

Six weeks after ELSA went live, the cracks started showing. Six current and former FDA officials told CNN that the tool fabricates studies outright — a phenomenon researchers politely call “hallucination” and reviewers experience as something closer to sabotage. CNN investigation on ELSA's fabricated studies One employee put it bluntly: “Any information that you can't verify is not dependable. It hallucinates with confidence.” FDA employee quote via CNN Another said the tool ends up costing time rather than saving it, because every summary now requires a second pass to catch distortions. Engadget report on ELSA hallucinations

This isn't a hypothetical edge case. Employees documented ELSA citing a study that doesn't exist, misstating who worked at the agency, and getting simple counting questions wrong — how many pediatric drugs were approved in a category, how many products carried a specific label. CNN reporting on ELSA errors When FDA Commissioner Marty Makary was asked about these specific complaints, he said he hadn't heard them, and noted that using ELSA is currently voluntary at the agency. Makary's response reported by Engadget Voluntary is doing a lot of work in that sentence. If a tool that hallucinates confidently is optional for federal reviewers evaluating drug safety, what happens when a hospital makes a similar tool mandatory for clinicians evaluating patient risk?

I bring this up not to pile on the FDA — I bring it up because hospital systems are buying and deploying the exact same class of generative AI, often with far less scrutiny than a federal agency under press investigation. If ELSA can misrepresent a study inside a GovCloud environment built specifically to keep it accurate, your ambient documentation tool or your clinical summarization assistant can do the same thing inside your EHR. The lesson isn't “don't use AI.” The lesson is: assume hallucination is a baseline risk, not an edge case, and build verification into the workflow before you build efficiency into it.

What ELSA Actually Can't See — and Why That Matters for Your Vendors

Here's a detail that got less attention than it deserved: ELSA cannot access many of the documents it would need to answer basic regulatory questions, including industry submissions. It can't reliably tell a reviewer how many times a company has sought approval for a product, or details about what's already on the market. CNN on ELSA's data access limitations The FDA has since clarified that ELSA runs in a FedRAMP High secure Google Cloud Platform environment, doesn't train on submitted industry data, and isn't connected to the internet — all reasonable security choices. FDA press announcement on ELSA's technical architecture But security and completeness are different problems. A tool can be perfectly locked down and still be confidently wrong because it's reasoning from an incomplete picture.

That's the exact question hospital leaders should be asking every AI vendor pitch: what can't your model see, and what does it do when it doesn't know? Does it say “I don't have enough information,” or does it generate a plausible-sounding answer anyway? ELSA, by multiple accounts, does the latter. TechTimes on ELSA's confident errors If your ambient scribe, prior-auth assistant, or triage tool behaves the same way when it hits a gap in the chart, you have a patient-safety problem wearing an efficiency costume.

The FDA's own draft guidance on AI in regulatory submissions is instructive here, even though it's written for drug sponsors rather than hospitals. It lays out a risk-based credibility framework: the higher the stakes of a decision the AI is informing, the more rigorous the validation needs to be before you trust its output. FDA guidance on AI credibility assessment framework That same logic belongs in your hospital's AI governance committee. A documentation assistant that drafts a discharge summary for a clinician to review is lower risk. A tool that flags sepsis risk or recommends a dosing change without a human check in the loop is a different category entirely — and it deserves ELSA-level scrutiny before go-live, not after a reporter starts asking questions.

Three Questions to Ask Before Your Next AI Procurement Meeting

If the FDA's rollout teaches hospital leaders anything, it's that “the vendor says it's accurate” is not a governance answer. Bring these questions into your next procurement or credentialing review:

• What happens when the model doesn't have enough data to answer confidently — does it decline, flag uncertainty, or generate a plausible guess?
• Can staff independently verify every factual claim the tool produces, and how long does that verification realistically take in a busy shift?
• Is use of the tool voluntary or mandatory, and who is accountable when a hallucinated output makes it into a chart or a clinical decision?
• Does the tool's risk tier match the FDA's credibility-assessment logic — low-stakes drafting versus high-stakes clinical recommendation — and is your review process proportional to that risk? FDA AI credibility framework

Continue the Conversation

If this resonated, here is where to go next: Join the AI-in-Healthcare Workshop · Get the Books · Contact Dan