How Mayo Clinic works to ensure its AI algorithms are vetted for safety

Mayo Clinic Platform President Dr. John Halamka speaks at HIMSS26 in March.

Many hospitals and health systems are advancing their artificial intelligence initiatives, moving beyond low-hanging fruit such as financial, operational and other back office applications to AI models designed for clinical settings.

But as they deploy new tools for decision support, predictive analytics and other clinical use cases, provider organizations may not be doing all they can to ensure the algorithms they're using are fully vetted and tested for safety.

WHY IT MATTERSMayo Clinic leaders this past week described the process it has put into place to test and assess clinical AI tools.

Mayo Clinic Platform President Dr. John Halamka, writing on the health system's website alongside Paul Cerrato, Mayo's senior research analyst and communications specialist, noted that "every clinical AI application is reviewed and approved before being used by Mayo Clinic staff."

The review regimen is based around an "executive-led oversight process, rooted in trusted clinical experience, that recognizes the need for both speed and guardrails," they write.

So far, that process "has reviewed over 100 AI clinical applications this year alone, taking into account such factors as AI complexity, clinical setting, performance and patient safety, user training, workflow integration, privacy and security and life-cycle management."

They also cite other work at major health systems to ensure the safety and integrity of clinical AI apps, such as Stanford's ongoing development of its ChatEHR tool, which depends closely on feedback and vetting from human clinicians.

"Even the most sophisticated AI models are no match for years of clinical experience and the ability to sense problems," write Halamka and Cerrato.

THE LARGER TRENDHalamka has been interested for many years about the proper place for AI technology in care delivery and how it should be used by humans. As he said way back in 2018: "If your doctor could be replaced by AI, your doctor *should* be replaced by AI."

In 2023, Halamka had expressed some concern about the transparency and reliability of generative AI, which was taking healthcare by storm. "We have to be a little bit careful with the use cases we choose," he cautioned.

But he nonetheless recognized that AI was leading to fundamental changes in the care delivery process. And he offered an update to his previous statement. "Doctors and nurses who use AI will replace doctors and nurses who don't."

By early 2026, Halamka was making the case that AI "may soon become our standard of care," enabling providers to "treat more patients in more locations than ever before."

But none of that changes the fact that any AI model deployed in a clinical setting needs to be rigorously tested, governed and monitored – and subjected to the oversight of human clinicians with hard-won medical experience.

"We need to take care of our patients, our researchers and our employees," said Jane Moran, chief information and digital officer at Mass General Brigham at HIMSS26 earlier this year.

"AI in healthcare is moving from pilots to early-stage skills," she said. "Most healthcare organizations are approaching it this way, and I think that's an encouraging signal. We're not just pushing things into mass production within 90 days. Adoption needs to be cautious. We need to consider clinical safety, cybersecurity, privacy and regulatory concerns."

ON THE RECORD"AI tools which are complex and could affect decision-making in time-urgent or critical clinical scenarios are not allowed to be used at Mayo Clinic without proactive human review and decision-making by qualified clinical users," write Halamka and Cerrato.

"The early success of this governance model demonstrates that, done right, we can rapidly advance AI to improve patient outcomes without sacrificing patient safety, privacy, or security."

Mike Miliard is Executive Editor of Healthcare IT NewsEmail the writer: [email protected]Healthcare IT News is a HIMSS publication.