AI governance challenges need close attention and collaboration

Dr. Deepti Pandita of the University of California Irvine Health, Denisa Lambert of TriMedx, Erika Kim of ARPA-H and Dr. Leeda Rashid of FDA

BOSTON – Effective artificial intelligence governance efforts are too often blocked by a chaotic maze of overlapping regulations, legal challenges such as patient opt-out laws and, of course, fragmented and siloed datasets.

But health systems can build mature data governance initiatives and collaborate with all stakeholders to ensure AI safety and trust, said speakers on a panel at the HIMSS AI in Healthcare Forum this past week.

Clearing those roadblocks requires active buy-in and collaboration from clinical, operational and IT leaders; proactive data preparation, and a risk-based strategy to AI safety, said panelists during the data governance discussion.

"We need a multi-stakeholder approach," said panel moderator Dr. Leeda Rashid, from the U.S. Food and Drug Administration's Digital Health Center of Excellence.

The challenges around safe, effective and efficient deployment of AI are many.

For instance, navigating state regulations, such California's many recent AI laws, poses compliance challenges and even ethical considerations for providers – as Dr. Deepti Pandita chief medical information officer and VP of clinical informatics at the University of California Irvine Health pointed out.

Pandita described, for example, a patient who declines the use of AI in their care.

"What happens then?" she asked. "Does that automatically mean we are declining care to that patient? And where will they go? Because the next door system is also using AI, and the system next door to them is also using AI. So are [they] declining care at a state level then? We haven't figured this out.

Compliance and legal team members have explained that patients cannot specify "which size scalpel the surgeon should use for their surgery, so they can't tell us what AI you should and should not use," Pandita added.

"We are in this very, very nebulous zone of what does using AI in healthcare really mean from a compliance and regulatory perspective," she said.

"This is a very fast-moving space" that requires standard operating procedures to deploy AI systems into real-world applications, added Erika Kim, program manager at the Advanced Research Projects Agency for Health, or ARPA-H.

Kim is also a signal processing engineer working on cancer genomics and data management previously at the National Institutes of Health.

"We need the community who are battling this in the real world to help us to get the right requirements and right understanding of where the bottlenecks are and what things need to be really put in place for people to have trust," she said.

"As a federal government employee here, that's kind of my call to action to the community: Let us know, talk to us, talk to me," said Kim.

Playbooks help, but aren't plug-and-play

Explicitly tying data governance goals to positive outcomes – such as reducing clinician burnout, protecting patient data security and improving the accuracy of clinical tools – can help to foster the collaboration that makes effective governance a reality.

But governance cannot solely lie on the shoulders of a health system's IT or compliance teams – it requires a multi-stakeholder approach, said panelists. The successful management of AI requires oversight across its entire lifecycle, from procurement to retirement.

AI Forum panelists suggest building a formal data governance council that meets regularly, including data stewards or users, clinicians, researchers and operational staff who understand how data is collected and used.

It should also include data custodians, or IT leaders such as database administrators and cybersecurity experts, who understand storage, architecture and security protocols.

Then, add compliance and legal staff members to the governance committee.

Currently, AI governance often operates like a sequential "handoff" between different entities – manufacturers, developers and health systems.

With regulations unable to keep up with AI technology, healthcare organizations face a chaotic, layered compliance environment spanning federal agencies, such as the FDA, state-level mandates, local hospital policies and accreditation requirements.

There is some guesswork as to where one entity's governance responsibility ends, and another's begins.

"There are playbooks out there, but those playbooks are not plug and play," said Pandita. "They don't work out for every system in the same manner, because every system is resourced differently. Their problems are different. Their culture is different. Their acceptance of risk is different."

Governance, workflow and culture

AI governance cannot exist without mature data governance, however. As Pandita noted: "Bad data will lead to bad AI."

To achieve successful collaboration in data governance, organizations must move away from rigid, bureaucratic policies and move toward shared ownership.

Because healthcare data is largely unstructured, "messy" and rapidly expanding, organizations must fix their data integrity, translation and standardization layers first.

"It's not a fine science that anyone has mastered yet, having to pull all that in together and make sure that it's a cohesive process," said Denisa Lambert, senior vice president at TriMedx, the clinical asset management and engineering company.

"There's really no handing it off; you have to have your arms around all of it at the same time."

Healthcare data coding is already highly variable, but new layers from ambient technologies and patient-generated real-world data only add to the complexity.

Integrating all of that information into existing data elements to improve insights is a new frontier for data managers.

"I think it's really challenging because when you bring those devices in, they're looked at as an individual device itself, but everything is commingled, and one device functions with another device," said Lambert.

"And when you look at the structure from a manufacturer model perspective, you're looking at it as an individual, and I don't think anyone's mastered that yet of how do you pull all that in together, which makes it very messy from even a regulatory perspective."

The key is to connect data governance directly to workflow reality, said Pandita.

When onboarding new software or AI tools, treat the vendor as a collaborative partner, she suggested. Establish clear, upfront agreements regarding data ownership, how data drift will be monitored and how data will be securely handed off between systems.

Also, build the guardrails into the tools health systems already use because AI governance will fail if it's treated as a rigid checklist, Pandita suggested.

"When you do structure your AI governance, you have to look at it from not just a governance standpoint," she said. "It is a marriage of governance with workflow and culture.

"You have to incorporate your cultural nuances and your workflow reality into your governance structure. It is not just governance for the sake of governing."

The approach is to treat governance as an enabler, and shift the narrative from restricting data to safeguarding and maximizing its value.

"I am very passionate about how we really make data more usable, to really bring life from all the [patient] data that's sitting in our healthcare systems and many different repositories that have been funded by NIH to do a lot of research on," said Kim.

Multiple layers improve data quality and overlay the translational data layer.

"And this is where AI can really come in to help clean up the garbage potentially, or help it align with the standards that people are using to exchange health information and so forth," she said. "Those are different layers that we can implement and use really clean gold standard benchmarks to test your AI tools that you're applying to curate your data and clean up your data."

Andrea Fox is senior editor of Healthcare IT News.Email: [email protected]Healthcare IT News is a HIMSS Media publication.