AI Governance and Regulation in Healthcare

AI governance and regulation in healthcare represents the most complex intersection of technology oversight and human safety in the AI era. With more than 1,450 FDA-authorized AI-enabled medical devices on the market, generative AI entering clinical documentation workflows, and AI agents beginning to handle administrative tasks like prior authorization and scheduling, the governance challenge spans federal agencies, 50 state legislatures, international regulators, hospital accreditation bodies, and clinical professional standards — all attempting to manage a technology whose capabilities are advancing faster than any regulatory system in history.

The FDA's Evolving Oversight Framework

The FDA remains the primary federal gatekeeper for clinical AI in the United States, applying its Total Product Life Cycle (TPLC) framework to AI-enabled Software as a Medical Device (SaMD). By the end of 2025, the agency had authorized over 1,450 AI-enabled medical devices — up from 950 in August 2024 — with radiology accounting for 76% of all clearances. The pace of authorization accelerated to 295 new AI/ML device clearances in 2025 alone, reflecting both industry momentum and the FDA's growing institutional capacity to evaluate algorithmic products.

A pivotal regulatory innovation was the December 2024 Final Guidance on Predetermined Change Control Plans (PCCPs), which allows manufacturers to pre-specify how AI algorithms will be updated post-market without requiring full resubmission for each change. This directly addresses one of AI's fundamental governance tensions: the need for continuous learning versus the need for regulatory certainty. In January 2026, the FDA further clarified that AI tools which summarize patient data or suggest options for independent clinician evaluation may fall outside device regulation — creating a consequential distinction between advisory AI and autonomous AI in clinical settings.

The FDA's 2026 alignment with ISO 13485:2016 under the Quality Management System Regulation (QMSR) update harmonizes U.S. device oversight with international standards, a move that simplifies compliance for companies seeking simultaneous FDA and EU market access. For AI-specific cybersecurity, the agency now requires manufacturers to demonstrate products are "secure by design" with embedded threat modeling, Software Bills of Materials (SBOMs), and mechanisms for ongoing security updates.

State-Level Regulation: A Patchwork of Requirements

While Congress has not passed comprehensive federal AI legislation, states have moved aggressively. By mid-2025, over 250 healthcare AI bills had been introduced across more than 34 states, with 33 becoming law in 21 states. This fragmentation creates significant compliance burdens for multi-state health systems and intersects with broader data privacy requirements.

Texas (effective January 1, 2026) requires written patient disclosure whenever AI is used in healthcare services or treatment — a transparency mandate that applies before or on the date of service. Colorado mandates disclosure when AI is used in high-risk decisions, annual impact assessments, anti-bias controls, and record-keeping for at least three years, with enforcement beginning June 30, 2026. California's AI Transparency Act (SB 942, effective January 2026) requires large AI providers to offer tools for detecting AI-generated content, while AB 489 specifically prohibits AI systems from implying they hold healthcare licenses or that care is being provided by a licensed human when it is not. Illinois has gone further in behavioral health, prohibiting AI therapy systems from generating treatment plans without licensed professional review.

This state-level patchwork mirrors the broader challenge of AI governance and regulation across jurisdictions: different philosophies about transparency, consent, autonomy, and liability producing a compliance matrix that grows more complex with each legislative session.

The EU AI Act and International Convergence

The EU AI Act's provisions for high-risk AI systems become enforceable in August 2026, with medical AI classified as high-risk by default under Article 6(1)(b) whenever it functions as a component of a medical device requiring Notified Body review. This triggers comprehensive requirements: conformity assessments, quality management systems, transparency documentation, bias auditing, human oversight mechanisms, and incident reporting. The European Commission's December 2025 proposal to harmonize AI Act requirements with the existing Medical Device Regulation (MDR) and In Vitro Diagnostics Regulation (IVDR) aims to reduce duplicative compliance — a simplification that could be finalized by summer 2026 or 2027.

For companies selling AI medical devices in both the U.S. and EU markets, the convergence of FDA's ISO 13485 alignment with the EU's harmonization proposal represents a potential reduction in regulatory friction. However, substantive differences remain: the EU's requirements for algorithmic transparency and bias documentation go beyond current FDA expectations, while the FDA's PCCP framework for iterative updates has no direct EU equivalent.

Accreditation-Driven Governance: CHAI and the Joint Commission

Perhaps the most consequential governance development of 2025–2026 is the partnership between the Joint Commission and the Coalition for Health AI (CHAI). In September 2025, they released initial guidance for responsible AI adoption covering seven governance domains: executive oversight, regulatory compliance, IT infrastructure, cybersecurity, patient safety, clinical quality, and workforce readiness. The Joint Commission — which accredits more than 22,000 healthcare organizations nationwide — is developing a voluntary AI certification program based on these frameworks, with detailed implementation playbooks expected in 2026.

The CHAI Applied Model Card standard gives health systems a structured way to document algorithm provenance, training data composition, known risks, bias mitigation approaches, and ongoing maintenance procedures. If AI certification becomes a condition of Joint Commission accreditation, it would effectively establish a national governance baseline that reaches further than any federal regulation — because hospitals cannot operate without accreditation. This accreditation-driven approach complements the regulatory landscape by addressing institutional governance practices that no single agency can mandate, particularly how AI agents are evaluated, deployed, and monitored within clinical workflows.

Algorithmic Bias and Health Equity as Governance Imperatives

Governance frameworks increasingly treat algorithmic bias not as a technical nuisance but as a patient safety crisis. Research has documented systematic performance degradation in AI diagnostic tools when applied to populations underrepresented in training data — dermatology models trained primarily on lighter skin tones missing melanoma in darker-skinned patients, sepsis prediction models underperforming in hospitals serving predominantly minority communities. Over half of published clinical AI models rely on data primarily from the United States or China, creating blind spots that map onto existing health disparities.

The regulatory response is multi-layered. Colorado's anti-bias controls require annual impact assessments. The CHAI framework mandates ongoing equity auditing. The EU AI Act requires bias documentation as part of conformity assessment. And the ECRI Institute's 2025 report ranked insufficient AI governance as the second-highest patient safety concern nationally — signaling that the accreditation and quality infrastructure is treating bias as a governance failure, not merely a model deficiency. These equity requirements connect directly to the broader question of how AI safety is operationalized when the stakes are measured in patient outcomes.