The Most Trustworthy AI Says I Don't Know
A woman in her 60s, two days out from a cardiac catheterization, opens her health system's app at 11:40 on a Tuesday night. She types: is it normal for the puncture site to be warm and a little swollen? The chatbot tells her the clinic is closed until 9 a.m. and links her to a parking page.
Steve Pappas, Head of Innovation at eGain, uses that scenario to name the actual failure mode in patient-facing AI — and it isn't inaccuracy. The bot didn't do anything wrong. It just didn't know it was in over its head.
Confidence is the dangerous part, not ignorance
AI can field billing questions and appointment scheduling without incident. What most deployments can't do is recognize the moment a question has crossed into territory where "close enough" becomes actively harmful. A system that answers a post-surgical warning sign with total composure isn't failing loudly. It's failing quietly, while the patient walks away believing she got a real answer.
Pappas's reframe here is the one worth sitting with: saying "I don't know" is not a defect. It's the single behavior that protects a patient from a confidently wrong answer — and it's exactly the behavior most systems get tuned to eliminate, because they're measured by how rarely they escalate to a human.
He calls the fix "negative knowledge" — a defined list of situations where the only correct response is a real person, regardless of how the question gets phrased. Post-surgery symptoms. Medication dosing. Any appearance of the word "fear." One organization he worked with built this as a literal "boundary page" — just over a page of plain-language rules, maintained by a nurse informaticist, reviewed quarterly by a physician, backed by more than two thousand supporting articles. Simple in its wording. Not simple in what it took to write.
Escalation done right is a handoff, not a transfer
Pappas names four things a real handoff requires: the new receiver already understands the full context, so the patient never repeats themselves. The patient is told, in plain language, why they're being moved. The receiver is the right caregiver, not just the available one. And if it's genuinely too late to reach the right person, the system says so honestly rather than pretending the loop is still working.
Most dashboards treat every escalation as a defect to engineer toward zero. Pappas argues the opposite: a patient-facing tool that never escalates is probably answering questions it has no business answering. The real failure signals are different — a patient contacting the system again after a supposed resolution, a human auditor flagging a missed escalation, a patient forced to re-explain themselves after being transferred.
And then the line that should end any debate about where responsibility lives: when a provider gives the wrong answer, that's a mistake, and the person who owns the knowledge owns the risk. AI tools should be held the same way. If a healthcare chatbot gives the wrong answer, the healthcare organization is accountable. Healthcare doesn't get an exception because software was involved.
That's not an argument against the technology — Pappas is explicit about that. It's an argument that the systems patients will actually trust aren't the ones that never say "I don't know." They're the ones built with enough humility to know exactly when to stop talking and call for help.
Source: Steve Pappas, "Patient-Facing AI Must Know When to Call for Help," Healthcare IT Today, October 14, 2026.