Powered by MOMENTUMMEDIA
For breaking news and daily updates, subscribe to our newsletter

What’s up, Doc? AI in healthcare needs human oversight long after deployment, researchers warn

A Flinders University review has found clinicians are rarely involved in ongoing monitoring of clinical AI, despite their expertise being critical to keeping systems safe and effective.

What’s up, Doc? AI in healthcare needs human oversight long after deployment, researchers warn

AI systems in the healthcare sector risk drifting away from clinical needs without structured human oversight, according to new research from Flinders University.

Researchers examined 12 studies exploring how health professionals contribute to AI systems used to support clinical decisions.

While clinicians were commonly involved in developing and checking systems, none of the studies reported formal processes for their ongoing involvement.

 
 

Eleven studies described clinicians reviewing AI systems, while just two reported their involvement in maintenance. Development activities included labelling information and designing system rules.

The findings suggest that the concept of keeping a ‘human in the loop’ remains poorly defined in healthcare AI, making it difficult to establish what oversight actually involves and how clinician feedback influences systems post-deployment.

“We found much more detail about how clinicians helped build and check AI tools than about how they oversee them during everyday care,” said lead author Michael Bakker, a Flinders University PhD candidate, in a statement.

“We need to understand how clinicians can flag problems and how that feedback leads to changes in the system.”

Published in the International Journal of Medical Informatics, the review proposes a reporting checklist designed to make human involvement across an AI system's lifecycle clearer.

It asks researchers to specify who contributes to an AI tool, what they do, why their input is necessary, and how it impacts the system. It also calls for limitations and supporting documentation to be reported.

Co-author Professor Niranjan Bidargaddi, who leads Flinders University's Digital Health Research Lab, said the findings expose a gap in understanding how clinical AI remains safe and useful once deployed.

“The TGA and international regulators are increasingly focused on monitoring AI medical devices throughout their use. The challenge now is to establish how this works in real clinical settings, including who checks performance, who identifies problems and who is responsible for acting on them,” he said.

“Clinical expertise needs to remain part of the process throughout an AI tool’s use, with oversight built into routine clinical work.”

The researchers also cautioned that many of the studies relied on small datasets and input from only a handful of clinicians, potentially limiting how broadly their findings apply across healthcare settings.

Cyber DailyWant to see more stories from trusted news sources?
Make Cyber Daily a preferred news source on Google.
Tags: