As artificial intelligence (AI) makes its way into healthcare settings, it is revealing significant shortcomings in governance as these technologies come into use with patients. Although AI systems are increasingly employed for diagnostics, decision support, and hospital management, recent research indicates that many of these systems still present unmanaged risks to patient safety and equity. Rather than a lack of ethical considerations, these risks arise from inadequately implemented governance mechanisms.
The study, published in Sci, titled Governing Healthcare AI in the Real World: How Fairness, Transparency, and Human Oversight Can Coexist, reviews global evidence that highlights how issues such as bias, lack of transparency, and accountability gaps often arise post-deployment, during periods of minimal oversight.
Bias and Transparency Risks Persist After Deployment
Bias in healthcare AI is not a fixed issue that can be resolved during the design phase. Instead, it evolves over time as systems interact with diverse patient demographics, adapt to changing clinical practices, and utilize updated data inputs. While models developed on historical data may initially perform effectively, their accuracy and fairness can deteriorate after deployment—especially for underrepresented or vulnerable populations. The authors stress that this evolving aspect of bias is often neglected in governance frameworks that focus predominantly on pre-deployment validation.
In practical healthcare environments, patient demographics, disease prevalence, and care pathways rarely remain constant. AI systems deployed without ongoing monitoring may fail to reveal performance discrepancies until it is too late. The study underscores that many healthcare organizations lack the infrastructure, expertise, or contractual authority necessary to audit AI systems after they have been acquired. This results in a governance void, where tools continue to affect clinical outcomes even as their reliability shifts.
Transparency represents a similar challenge. Although explainability has become a cornerstone of ethical AI in healthcare, the authors found that explanation tools are often misaligned with the actual needs of clinicians and regulatory bodies. While AI-generated explanations may meet technical standards, they frequently do not provide the meaningful insights necessary for clinicians to justify decisions to either patients or regulators. Additionally, transparency mechanisms aimed at developers often fail to convert into effective accountability measures for healthcare institutions.
The study argues for differentiating transparency based on audience and purpose. Clinicians require explanations that enhance clinical decision-making, while patients need clear, understandable information about how AI affects their care. Furthermore, regulators and auditors need traceability and documentation that demonstrate compliance and effective risk management. When transparency is treated as a single, uniform criterion, the authors conclude, it results in systems that may seem explainable in theory but are inconsistent in practice.
Safety, Privacy, and the Accountability Gap
Safety has emerged as a recurring yet often overlooked risk in the governance of healthcare AI. The authors note that safety failures often stem not from major system errors but rather from gradual misalignments between AI outputs and clinical practices. Changes in staffing, protocols, or patient demographics can alter how AI recommendations are interpreted and enacted, leading to clinically significant harms without clear processes to reassess and adjust the systems.
The study reveals that responsibility for AI safety is often ambiguous. Developers may claim that systems operate as intended, while healthcare providers may assume that regulatory approval ensures ongoing safety. This diffusion of duty creates what the authors term an “accountability gap.” When adverse outcomes arise, it is frequently unclear who possesses the authority and obligation to intervene, suspend, or revoke an AI system’s use.
Privacy concerns further complicate the situation. The growing reliance on adaptive and generative models raises issues regarding data leakage, memorization, and inappropriate secondary usage of sensitive health information. The authors argue that traditional anonymization techniques are no longer adequate in an age where models can deduce or reconstruct personal data based on complex patterns. Although privacy-enhancing technologies like federated learning and differential privacy offer potential remedies, their effectiveness can vary widely depending on context.
The review emphasizes that privacy protections should not only be assessed on a technical level but should also relate to clinical outcomes and patient trust. Systems that technically comply with data protection regulations may still undermine confidence if patients do not comprehend how their data is utilized or if consent mechanisms remain unclear. The authors advocate for governance frameworks that integrate privacy considerations throughout procurement, deployment, and ongoing monitoring rather than treating them as a one-time compliance issue.
Human Oversight Must Be Operational, Not Symbolic
The study highlights a significant disparity between the theoretical concept of human oversight and its practical implementation in healthcare AI systems. Although many AI tools are marketed as decision-support systems designed to keep clinicians “in the loop,” the reality is that oversight is often poorly defined. Clinicians may lack the training, time, or authority to meaningfully question AI recommendations, particularly in high-pressure situations.
The authors contend that human oversight must be clearly defined through established decision-making rights, escalation protocols, and institutional support. Merely requiring a human to review an AI output does not guarantee effective oversight if that review is superficial or constrained by workflow pressures. Effective oversight requires clinicians to understand the system’s limitations, have access to performance metrics, and possess the authority to override or deactivate the system when necessary.
Training is also highlighted as a crucial yet frequently overlooked aspect. Many healthcare professionals receive insufficient education on AI system functionality, potential failures, and how to critically interpret outputs. Without this knowledge, human oversight risks becoming more of a symbolic gesture than a meaningful practice. The authors advocate for governance models that integrate training, competency assessment, and ongoing support as essential components of AI deployment.
The review also identifies procurement as a pivotal point for effective governance. Healthcare institutions often acquire AI systems through contracts that limit access to performance data or inhibit independent evaluation. The authors argue that governance must begin with procurement, including requirements for transparency, audibility, and ongoing monitoring in contracts. Without such stipulations, institutions may find themselves tied to systems that they cannot effectively manage.
Toward Governance by Design in Healthcare AI
The authors call for a transition from principle-based ethics to governance by design. This approach integrates fairness, transparency, safety, and oversight into the technical and organizational frameworks that dictate how AI systems will be used over time. Instead of treating governance as an add-on applied post-deployment, the authors argue for its incorporation at every stage of the AI lifecycle.
This includes pre-deployment evaluations that extend beyond accuracy metrics, deployment strategies that align with clinical realities, and continuous monitoring to observe performance variations across different patient populations. It also necessitates a clear assignment of responsibility among developers, healthcare providers, and regulatory bodies, backed by documentation that can withstand scrutiny.
The authors note that evolving frameworks in Europe and beyond are steering towards lifecycle-based governance of AI. However, they warn that mere regulation will not solve governance challenges unless healthcare institutions build the internal capacity to manage AI systems effectively. Compliance without operational capability risks resulting in formal adherence that fails to provide substantive protections.
