AI hallucinations force healthcare CIOs to rethink governance, validation and trust

Lior Eshel, CEO of TestDynamics

Artificial intelligence has quickly become part of everyday healthcare operations. Health systems are deploying AI to summarize office visits, prioritize imaging studies, assist clinicians with documentation, analyze scans and support clinical decision making.

The operational benefits are real, but so is a new category of enterprise risk. Unlike traditional software defects, generative AI can produce polished, authoritative answers that appear entirely credible even when they are incorrect.

That distinction is changing the conversation inside health systems. Early discussions centered on productivity, clinician burnout and return on investment.

Increasingly, healthcare CIOs, CMIOs and governance committees are asking a different question: How can a provider organization verify that AI remains accurate after deployment?

The issue no longer is simply selecting the right product. It is building an ongoing assurance process that measures performance, detects failures and protects patients as models evolve.

When confidence becomes the problem

"The biggest fear is an autonomous system making a diagnosis on its own," said Lior Eshel, CEO of AI-powered medical imaging company TestDynamics.

He argues that healthcare leaders should worry less about dramatic AI failures than about ordinary workflow tools that quietly become trusted. He points to generative scribes capable of introducing facts that never appeared in a patient encounter and predictive algorithms that continue influencing care for years before independent researchers evaluate their real-world accuracy.

For clinicians, inaccurate AI creates two opposite risks. Frequent false alerts contribute to alert fatigue, while confident, polished responses encourage overreliance. Patients may receive inaccurate information, and organizations inherit legal, financial and reputational exposure.

"A model cannot flag its own hallucination because the model that is wrong, by construction, is confident that it is right," Eshel explained.

He argues that detection therefore cannot depend on the model itself. Instead, hospitals need independent comparison against verified clinical data and, where possible, against other tools performing the same task.

Hallucinations represent only one dimension of AI risk. Bias may be embedded within training data, proxy variables or commercial optimization objectives long before a hospital purchases the technology.

"Evaluating biases before deployment is, therefore, concrete work, not a values statement," he said.

He recommends that buyers request detailed information describing training populations, demographic composition, validation methodology and participating clinical sites. Just as important, organizations should evaluate products using local patient populations rather than assuming vendor-reported performance will translate across different communities and workflows.

That reflects a broader lesson emerging across healthcare AI: validation is local, Eshel said. Models developed under one set of clinical conditions may perform differently once exposed to another institution's patients, documentation practices and disease prevalence.

Healthcare already monitors pharmaceuticals and medical devices after they reach the market. Eshel believes clinical AI deserves similar treatment.

Hospitals should validate systems before implementation, continuously monitor performance afterward and preserve the underlying source material needed to audit AI-generated outputs, he said. Without those records, organizations may never determine whether documentation accurately reflects the clinical encounter.

"Shared accountability is the only workable model, and it carries a precondition: shared visibility," he explained.

Shared visibility requires comprehensive logging, reproducible testing and independent measurement rather than accepting static validation performed elsewhere. Governance becomes a continuous operational discipline instead of a one-time procurement exercise.

"The steps to take now are not exotic," Eshel noted. "They are procurement disciplines applied to software that makes clinical claims."

He recommends that health systems begin with a complete inventory of AI models already operating across the enterprise, followed by contractual rights to independent testing, local validation and ongoing access to performance data.

As AI applications proliferate, evaluating products individually becomes increasingly impractical. Eshel instead envisions an independent evaluation layer capable of comparing competing systems using common benchmarks over time. Such a framework would give healthcare organizations objective evidence about how clinical AI performs after deployment instead of relying primarily on marketing claims.

Whether that model ultimately emerges through regulators, industry collaboratives or independent organizations remains uncertain. What appears increasingly clear, however, is that AI governance is becoming a core executive responsibility, he said.

For CIOs, success will be measured not simply by how quickly AI is implemented, but by whether leaders can demonstrate that systems remain transparent, auditable and clinically trustworthy throughout their operational life.

Follow Bill's health IT coverage on LinkedIn: Bill SiwickiEmail him: [email protected]Healthcare IT News is a HIMSS Media publication.

WATCH NOW: AI design lets health systems solve organizational problems