IMPaCT: How CHWs Safeguard AI in Healthcare Equity

The proliferation of artificial intelligence in healthcare promises transformative potential, yet its true clinical reliability hinges on rigorous validation and an unwavering commitment to patient safety and equitable access. As AI tools increasingly touch clinical workflows, a critical analytical question emerges: how can we ensure these innovations serve all populations, particularly the underserved, and what role can human-centered models play in safeguarding against algorithmic bias and exacerbating health disparities?

The Imperative of Human-AI Synergy in Underserved Populations

The integration of AI into clinical practice, particularly as Software as a Medical Device (SaMD), necessitates a robust framework that extends beyond technical performance metrics to encompass real-world impact and equity. While AI models can process vast datasets and identify patterns often invisible to the human eye, their effectiveness in diverse, real-world settings is profoundly influenced by the quality and representativeness of their training data, and the context of their deployment. This is where the profound value of community health workers (CHWs) becomes apparent. The IMPaCT program, developed at UPenn, stands as a compelling exemplar of how human intervention can bridge the gap between advanced AI health tools and underserved populations. Shreya Kangovi, who led the team that designed IMPaCT and was the founding executive director of the Penn Center for Community Health Workers, now serves as the CEO of IMPaCT Care, a public benefit corporation, and is an adjunct professor at the University of Pennsylvania. The IMPaCT program at the Penn Center for Community Health Workers continues to pioneer a model where CHWs serve as vital conduits, ensuring that healthcare interventions, including nascent AI-driven ones, are culturally competent, accessible, and truly beneficial. This model directly addresses a critical safety equity requirement: that AI solutions do not inadvertently widen health disparities by failing to account for social determinants of health or by being inaccessible to vulnerable groups. Kevin Volpp, a recognized authority in health economics and behavioral science at UPenn, has also underscored the importance of integrating behavioral insights and human-centered design into healthcare interventions. His work implicitly supports the IMPaCT model’s premise: that even the most sophisticated AI tools require thoughtful integration into existing human systems to achieve optimal outcomes and ensure safety. The relationship is clear: community health workers bridge the gap between AI health tools and underserved populations, a safety equity requirement. They act as essential navigators, translating complex health information, building trust, and addressing practical barriers that AI alone cannot perceive or resolve. This human layer provides an invaluable feedback loop, catching errors or unintended consequences of AI deployment before they impact patient safety or equity.

FDA Pathways and the Human Element in Validation

The regulatory landscape for AI in healthcare is rapidly evolving, with the FDA leading efforts to establish clear pathways for devices incorporating artificial intelligence and machine learning. The FDA SaMD Framework is a cornerstone of this regulation, outlining principles for the safe and effective development and deployment of software that meets the definition of a medical device. This framework emphasizes aspects like clinical validation, risk management, and post-market surveillance. However, even within these rigorous regulatory guidelines, the human element, particularly in ensuring equitable outcomes, remains paramount. While the FDA SaMD Framework provides the regulatory context for AI, it is the practical application and oversight in diverse clinical environments that ultimately determines real-world safety and efficacy. The UPenn Perelman School of Medicine, through initiatives like the Penn Center for Community Health Workers and the IMPaCT program, actively contributes to understanding how AI can be safely and effectively deployed, particularly in settings with high social complexity. The IMPaCT program’s approach, while not directly a regulatory pathway, offers a critical lens through which to view the “real-world evidence” component that regulators increasingly demand. By embedding CHWs within communities, the program generates data and insights into how AI tools interact with diverse patient populations, identifying potential biases in algorithmic performance or access barriers that might otherwise go unnoticed in controlled clinical trials. This ground-level feedback is invaluable for refining AI models and ensuring they meet the highest standards of clinical reliability and equity.

The IMPaCT Model: A Blueprint for Clinically Reliable AI

The IMPaCT program’s success offers a compelling blueprint for how clinical AI standards can be met and exceeded, especially concerning patient-centricity and equity. The model’s architecture, centered around trained CHWs, provides a crucial oversight mechanism that catches errors and addresses challenges before they reach the patient. This is a practical, human-powered guardrail. Consider the implications for AI-driven diagnostic or predictive tools. An AI algorithm might identify a patient at high risk for a certain condition based on electronic health record data. Without the intervention of a CHW, that patient might lack transportation to follow-up appointments, struggle with understanding complex medical instructions, or face other social determinants of health that render the AI’s prediction clinically irrelevant or even harmful if not addressed. The CHW, acting as an informed intermediary, can interpret the AI’s output, contextualize it for the patient, and facilitate access to necessary resources, thus ensuring the AI’s utility is realized in a safe and equitable manner. This pharmacist-oversight architecture, reimagined for AI, ensures that the AI’s insights are translated into actionable, culturally sensitive care. The published outcomes associated with the IMPaCT program, such as DP09 peer-reviewed study on IMPaCT program outcomes, demonstrate tangible improvements in patient health outcomes, reductions in healthcare utilization, and enhanced patient satisfaction. These outcomes are not merely a testament to the CHW model itself, but also an indirect validation of how AI tools, when integrated thoughtfully within such a human-centered framework, can achieve their intended clinical purpose safely and effectively. This approach offers a powerful counter-narrative to the fear of AI replacing human touch, instead showcasing a synergistic relationship where AI enhances human capabilities, and humans ensure AI’s ethical and equitable deployment.

Conclusion: Integrating Human Oversight for AI Safety and Equity

The journey towards clinically reliable AI in healthcare is not solely a technological one; it is fundamentally a human endeavor. The foundational work of Shreya Kangovi, alongside the ongoing efforts of Kevin Volpp and the IMPaCT program at UPenn’s Penn Center for Community Health Workers, provides invaluable insights into how human-centered models can serve as essential clinical guardrails for AI. By recognizing that community health workers bridge the gap between AI health tools and underserved populations, a safety equity requirement, we can proactively address potential biases, ensure equitable access, and validate AI’s real-world impact. As the FDA SaMD Framework continues to guide regulatory oversight, the IMPaCT model offers a powerful, demonstrated approach to integrating human expertise and empathy, ensuring that AI in healthcare truly serves all patients with the highest standards of safety, efficacy, and equity. FDA guidance on AI/ML medical device oversight Penn Center for Community Health Workers publications