AI accountability is now healthcare's next big challenge

Angela Adams, CEO of Inflo Health

Artificial intelligence has quickly evolved from an emerging technology into an everyday clinical tool. Physicians increasingly rely on AI for documentation, decision support, triage and diagnostic assistance, while patients are arriving at appointments armed with chatbot-generated interpretations of imaging studies, laboratory results and potential diagnoses.

For health system leaders, the industry's central AI question has changed accordingly. The challenge is no longer whether clinicians will adopt AI. It is whether organizations can govern it well enough to earn lasting trust.

"For most of the last few years, the hard part was getting clinicians to use AI at all," she recalled. "That problem has largely solved itself. Clinicians now reach for AI in documentation, triage, decision support and detection as a matter of routine. The gap that has opened up in its place is traceability."

Adams argues that many organizations cannot confidently answer a deceptively simple question: Where exactly is AI affecting patient care? Internal inventories often fail to capture every AI-enabled application operating across an enterprise.

At the same time, patients are introducing another layer of complexity by bringing externally generated AI advice into clinical encounters. That combination creates governance blind spots that extend well beyond the technology health systems deploy themselves.

Accountability must be defined before deployment

Rather than debating responsibility after something goes wrong, Adams said health systems should establish clear ownership before activating an AI tool. Accountability, she argued, spans clinicians, vendors and provider organizations – but each carries a different obligation.

"Here is where I would push leaders hardest," she said. "The question of who is accountable should be answered before a tool is ever deployed, not reconstructed after a patient has been harmed. For every place AI touches care, a health system should be able to name who owns the output and what happens when it is wrong.

"If you cannot answer that, the tool is not ready to turn on," she continued. "And clinicians have to stay vigilant about automation bias, the very human pull to trust the machine a little more than the evidence warrants. AI raises the stakes on clinical judgment. It does not retire the need for it."

Clinicians remain responsible for medical decisions, she said, while health systems must validate AI performance against their own patient populations, rather than relying solely on vendor studies. Vendors, meanwhile, should provide transparent information about model development, limitations and expected performance.

Together, those responsibilities create a governance framework that is continuous rather than a one-time implementation exercise.

Scaling AI requires organizational trust

Many organizations successfully complete AI pilots but struggle when expanding them across an enterprise. One reason, Adams said, is that governance often becomes centralized control instead of collaborative decision-making.

"The most common mistake is treating governance as a control function: a central committee that approves or denies from above the people actually doing the work," she noted.

Instead, she advocates multidisciplinary teams that include clinicians, informaticists, quality leaders and operational stakeholders throughout implementation. Equally important, organizations should avoid assuming that a successful pilot automatically predicts enterprise-wide performance.

As AI encounters new workflows, patient populations and clinical edge cases, continuous validation becomes essential to maintaining clinician confidence, she added.

Protecting the human relationship

Despite widespread discussion about AI replacing clinicians, Adams sees the technology's greatest value elsewhere. Administrative automation can return attention to patients by reducing documentation burdens, while AI-driven follow-up can help ensure abnormal findings receive appropriate next steps instead of disappearing between departments.

"The replacement narrative gets the most airtime and teaches us the least," she said. "AI will change a great many clinical jobs. It will not replace the relationship at the center of care, and the organizations that internalize that will use the technology to protect that relationship instead of crowding it out."

She also believes patient use of consumer AI does not have to undermine clinicians. Properly handled, those conversations can create better-informed patients while reinforcing physicians as trusted interpreters of increasingly complex information.

"Measurement is where I would challenge most leaders because the easiest metrics are the most misleading," she explained. "Judge AI purely by throughput and minutes saved, and you can optimize your way straight into a faster, colder, less trusted experience.

"The numbers that actually signal a stronger relationship are quieter: time spent in direct patient interaction, how well patients understand their own care, whether follow-up loops reliably close, and whether clinicians trust the tools they are handed," she continued. "When efficiency climbs while those measures fall, the technology is costing you the thing healthcare exists to provide."

For healthcare executives, those measures may become the defining indicators of AI success. As adoption becomes routine across the industry, competitive advantage will increasingly come, not from deploying the newest model first, but from building governance structures that make AI transparent, accountable and worthy of clinicians' and patients' trust.

Follow Bill's health IT coverage on LinkedIn: Bill SiwickiEmail him: [email protected]Healthcare IT News is a HIMSS Media publication.

WATCH NOW: Confidence remains the biggest healthcare AI barrier