The Role of AI in Healthcare: Addressing Bias and Ensuring Fairness

Artificial intelligence (AI) is increasingly integrated into healthcare, enhancing the analysis of medical images, risk prediction, and remote patient monitoring. However, AI systems can also falter, particularly when they are trained on biased or unrepresentative datasets. A recent study led by Professor Courtney Lyles at UC Davis highlights the critical need for human oversight in AI decision-making to mitigate bias and enhance safety. This research was published in the journal Social Science and Medicine.

About the Research

Professor Courtney Lyles, who directs the UC Davis Center for Healthcare Policy and Research, is also a co-founder and co-director of UC S.O.L.V.E Health Tech. This initiative brings together researchers from UC Davis, UC Berkeley, UC San Francisco, and private digital health firms. In this interview, Lyles elaborates on the use of AI in healthcare, strategies to identify and prevent bias, and examples of how UC Davis Health is committed to developing fair and reliable AI systems to serve both patients and healthcare professionals.

What is this study about?

This study is a collaboration involving Google, along with researchers from the University of California and Northeastern University. We adopted a human-centered methodology to critically evaluate explainable AI models, identifying potential biases. An expert panel from various fields reviewed the AI interpretations to find factors contributing to bias.

Why can bias be a problem in AI healthcare systems?

Understanding the social and structural factors that shape health data is essential when interpreting AI models. Without this perspective, AI outputs may seem credible but can be incomplete, biased, or unsafe. As AI tools become more embedded in clinical practice, it is vital to combine algorithmic analyses with human expertise and explainable AI tools.

What is explainable AI and why is it important in evaluating AI models?

Explainable AI (XAI) focuses on understanding the rationale behind AI decisions. It sheds light on the AI’s reasoning process, allowing us to grasp how it arrives at its conclusions and predictions.

How would human reviewers assess bias in XAI models?

Our research underscores that an interdisciplinary panel can meticulously evaluate XAI outputs and provide contextual interpretations that align with real-world scenarios. This panel included experts from fields such as medicine, epidemiology, and data science, among others. Incorporating community members and patient advocates into this process ensures that AI tools cater to the actual needs of the communities they serve.

How does an interdisciplinary panel assess XAI results?

When an XAI tool reveals the basis for its predictions, it often uncovers significant patterns. Interdisciplinary experts can then interrogate these findings by asking:

• Could this pattern arise from dataset discrepancies?
• Is this result a function of patient interactions with medical devices?
• Does this reflect a broader social or structural issue instead of a purely medical one?

This approach helps identify where AI may be relying on misleading “shortcut features” that initially appear meaningful but ultimately reflect underlying disparities in the data.

How can you turn this XAI study into real-world practice?

Our research included a case study demonstrating how an interdisciplinary panel reviewed real-world XAI outcomes from medical imaging and suggested actionable steps for research and practice. By intertwining technical tools with human judgment, we can enhance accuracy and context in various applications. Forming teams that encompass diverse expertise ahead of time fosters better implementation and builds trust among data scientists, clinicians, patients, and communities.

How do private-public partnerships shape AI development in healthcare?

Going forward, intentional collaboration between the private and public sectors is crucial. For instance, UC S.O.L.V.E Health Tech emphasizes partnerships between UC researchers and the private sector to promote equity in their products. By facilitating structured collaborations, we can bridge the gaps between often isolated sectors.

What are some additional examples of how UC Davis Health uses AI?

UC Davis Health is a national frontrunner in AI integration across various clinical domains. Key initiatives include:

• AI Governance Committee: Led by Professor Jason Adams, this committee has been assessing and refining new AI models for years.
• Equitable Evaluation Process: Under the leadership of Professor Reshma Gupta, our IT team has devised methods to reduce bias in AI predictive models. For example, this involves identifying patients at higher risk for readmission by considering specific subgroups and eliminating potential barriers throughout the AI development and implementation phases.
• AI Scribe Implementation: UC Davis Health has adopted AI Scribe technology for generating medical notes during clinical interactions.

What is AI Scribe and how does it work?

In 2024, UC Davis Health introduced an AI Scribe program that employs AI to create notes for clinicians during consultations. After obtaining the patient’s approval, doctors can record their conversations, and the AI application summarizes the interactions into a standard clinical note format, streamlining the documentation process. A pilot study evaluated the accuracy of AI-generated notes.

The results, presented by principal biostatistician Sandra Taylor in the Department of Public Health Sciences, revealed that the AI-generated clinical notes achieved high quality, with a remarkable 94.7% free from significant errors. However, it also emphasized the necessity for clinicians to periodically review the AI-generated outputs to catch and rectify any minor inaccuracies—reinforcing the importance of human oversight in these tools.

About The Center for Healthcare Policy and Research

The mission of the Center for Healthcare Policy and Research (CHPR) is to facilitate research, promote education, and inform policies related to health and healthcare. Its goal is to enhance public health by contributing knowledge about access, delivery, cost, quality, and outcomes associated with healthcare, providing rigorous evidence to policymakers and other stakeholders. The center advances its mission through interdisciplinary collaboration, educational initiatives, and the synthesis and dissemination of research.

To learn more about CHPR’s weekly seminar series and research themes, visit our website.

Conclusion

The integration of AI in healthcare presents exciting opportunities, but it also brings challenges that must be addressed, particularly regarding bias and the importance of human oversight. By fostering collaboration among diverse experts and ensuring that the needs of the community are prioritized, we can create a future where AI systems are equitable, trustworthy, and truly beneficial for all.