AI Governance in Pharma

Industry Application

AI Governance RegulationPharma & Life Sciences

The Regulatory Imperative for AI in Drug Development

Pharmaceutical and life sciences companies face a uniquely stringent AI governance environment because the stakes — patient safety, drug efficacy, and public health — are existential. Unlike most industries, pharma operates under decades-old statutory frameworks (the Food, Drug, and Cosmetic Act; ICH guidelines; GxP regulations) that predate modern AI entirely. Regulators are now racing to layer AI-specific oversight onto these foundations without triggering innovation paralysis in an industry where AI-assisted drug candidates are already entering Phase II and III trials.

The AI Governance Regulation landscape for pharma is shaped by three converging forces: the FDA's Software as a Medical Device (SaMD) framework and its AI/ML action plan, the EU AI Act's classification of most clinical AI as "high-risk," and the industry's own voluntary governance initiatives through bodies like the Pharmaceutical Research and Manufacturers of America (PhRMA) and the International Council for Harmonisation (ICH). By early 2026, the FDA has cleared over 950 AI/ML-enabled medical devices — the majority diagnostics and imaging tools — making it the world's most active regulator of deployed clinical AI.

FDA Framework: Predetermined Change Control and SaMD

The FDA's 2021 AI/ML-Based Software as a Medical Device Action Plan introduced the concept of the Predetermined Change Control Plan (PCCP) — a mechanism allowing manufacturers to pre-specify the types of AI model updates permissible without triggering a full 510(k) or PMA resubmission. This is critical because continuously-learning AI systems, such as those used in radiology triage or sepsis prediction, improve with real-world data in ways that traditional static-software review cycles cannot accommodate. The PCCP framework requires manufacturers to define in advance the performance boundaries, retraining triggers, and validation protocols for any planned model modifications.

Under 21 CFR Part 11 and Good Machine Learning Practice (GMLP) guidance finalized in 2023 (co-developed with Health Canada and the UK's MHRA), FDA-regulated AI systems in pharma must meet seven core principles: data management transparency, model explainability appropriate to intended use, bias detection and mitigation across demographic subgroups, real-world performance monitoring, human oversight provisions, clear labeling of AI-assisted outputs, and robust change management. Companies like Tempus AI and Veracyte have restructured their regulatory submissions to include dedicated GMLP annexes — a practice now effectively required for AI diagnostic submissions.

EU AI Act: High-Risk Classification and Pharma's Compliance Burden

Under the EU AI Act (Articles 6 and 10, fully applicable from August 2026), virtually all AI systems used in direct clinical decision support — including AI that recommends drug dosing, interprets diagnostic imaging, predicts patient deterioration, or screens drug-target interactions for clinical use — are classified as high-risk AI systems under Annex III. This triggers conformity assessments, CE marking requirements under the notified body regime, mandatory registration in the EU AI database, and ongoing post-market monitoring obligations that closely parallel the EU MDR/IVDR frameworks already familiar to device manufacturers.

For pharma companies operating in the EU, this means AI systems embedded in clinical decision support tools, companion diagnostics, or patient monitoring platforms require technical documentation covering training data governance, accuracy metrics disaggregated by relevant population subgroups, human oversight mechanisms, and incident reporting procedures. Roche's navify Algorithm Suite and Siemens Healthineers' AI-Rad Companion were among the first platforms to publish EU AI Act compliance roadmaps in late 2024, committing to notified body review cycles that mirror their MDR Article 10 obligations. The Act's prohibition on general-purpose AI systems that cannot provide meaningful explanations for high-stakes clinical outputs is reshaping how companies architect models — pushing adoption of inherently interpretable architectures and post-hoc explainability tools like SHAP and LIME in regulated contexts.

Drug Discovery AI: Governance Beyond the Clinic

While clinical AI bears the heaviest direct regulatory burden, AI systems used in preclinical drug discovery — target identification, molecular generation, ADMET prediction, synthetic route planning — currently occupy a governance gray zone. They are not classified as SaMD or high-risk under the EU AI Act because they do not directly inform patient care, but they feed into regulatory submissions (INDs, CTAs) where the provenance, validation, and reproducibility of AI-generated data is increasingly scrutinized.

The FDA's 2023 guidance on Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products explicitly stated that AI used to generate data supporting regulatory submissions must be documented with the same rigor as any other analytical method — including version control of models, training data provenance, and cross-validation against wet-lab results. Insilico Medicine, whose AI-designed fibrosis drug candidate INS018_055 entered Phase II in 2023, became a landmark case study: the FDA required a dedicated "AI Data Annex" in the IND describing the generative chemistry model's architecture, training corpus, and the human expert review process for generated candidates. Recursion Pharmaceuticals has similarly developed internal AI governance frameworks aligned with these evolving expectations, publishing their model cards and dataset documentation as part of their scientific transparency program.

Pharmacovigilance, Real-World Evidence, and Ongoing Monitoring

AI governance in pharma extends beyond initial approval into the post-market surveillance lifecycle. Pharmacovigilance — the detection, assessment, and prevention of adverse drug reactions — has been heavily automated using NLP and ML models that process adverse event reports from EHRs, social media, literature, and spontaneous reports. The EMA's guidance on electronic submission of safety reports and the FDA's Sentinel System both now incorporate AI-assisted signal detection, but with explicit governance requirements: AI-flagged signals must be reviewed by qualified pharmacovigilance physicians before any regulatory action, and the false-positive/false-negative rates of automated systems must be periodically validated and disclosed.

AstraZeneca has deployed large language models for automated medical coding of adverse events (MedDRA coding) across its global pharmacovigilance operations, processing millions of case reports annually. Their governance framework — published in their 2024 AI Ethics Annual Report — specifies mandatory human review rates, inter-rater reliability benchmarks for AI-versus-human coding agreement, and escalation protocols for ambiguous or serious cases. This model — AI as a triage and efficiency layer with structured human oversight gates — has become the de facto industry standard for pharma AI governance in post-market contexts, reflecting both regulatory expectations and the practical liability calculus of safety-critical automation.

Applications & Use Cases

AI-Designed Drug Candidates (IND Submissions)

Generative AI platforms used for de novo molecular design — as deployed by Insilico Medicine, Recursion, and Exscientia — now require dedicated AI methodology annexes in IND/CTA submissions. Governance frameworks specify model version control, training data provenance, and the human expert review process for AI-generated candidate selection, ensuring regulatory traceability from algorithm to clinical hypothesis.

AI/ML-Enabled Diagnostics (SaMD Compliance)

FDA-cleared diagnostic AI — covering radiology, pathology, genomics interpretation, and sepsis prediction — must comply with GMLP principles and, where continuously learning, operate under an approved Predetermined Change Control Plan. Companies like Tempus AI and PathAI embed governance artifacts (model cards, bias audits, real-world performance dashboards) directly into their FDA submission packages.

Clinical Trial Design and Patient Stratification

AI systems that propose adaptive trial designs, identify patient subpopulations, or optimize endpoint selection are subject to ICH E9(R1) estimand framework requirements and increasing FDA scrutiny on algorithmic bias. Pfizer and Novartis have established internal AI review boards that vet trial-design algorithms before protocol submission, assessing fairness across sex, age, race, and comorbidity subgroups.

Automated Pharmacovigilance Signal Detection

NLP and ML models processing adverse event reports from spontaneous reporting systems, EHRs, and social media must meet EMA and FDA validation requirements for sensitivity and specificity. AstraZeneca's AI-assisted MedDRA coding system and similar platforms at Pfizer operate under governance frameworks mandating periodic calibration audits, defined human override rates, and regulatory disclosure of system performance metrics.

Manufacturing Quality Control and Process AI

AI systems embedded in continuous pharmaceutical manufacturing — monitoring blend uniformity, real-time release testing, deviation detection — fall under FDA's Process Analytical Technology (PAT) framework and 21 CFR Part 11. Eli Lilly and Merck have implemented AI governance protocols for manufacturing AI that require change control documentation, equipment validation parity, and audit trail integrity equivalent to traditional analytical instruments.

Real-World Evidence Generation for Label Expansion

AI models analyzing EHR data, claims databases, and registries to generate real-world evidence for supplemental approvals must comply with FDA's RWE framework (2023 guidance) and the EU's DARWIN EU network standards. Governance requirements address training data representativeness, confounding adjustment transparency, and model reproducibility — with companies like Flatiron Health and IQVIA building regulatory-grade audit trails into their RWE AI pipelines.

Key Players

Insilico Medicine — Pioneer in end-to-end AI drug discovery with INS018_055 (fibrosis, Phase II), the first AI-designed and AI-triaged clinical candidate subject to a dedicated FDA AI methodology review; has published model cards for its generative chemistry platform.
Recursion Pharmaceuticals — Operates the OS platform integrating biological imaging AI with chemical space exploration; developed internal AI governance standards including dataset documentation and model transparency reports aligned with FDA's 2023 AI submission guidance.
AstraZeneca — Runs enterprise-scale AI governance across pharmacovigilance (automated MedDRA coding), clinical operations, and R&D; publishes annual AI Ethics Reports disclosing model performance metrics, human review rates, and bias audit outcomes.
Roche / Genentech — navify Algorithm Suite is among the first commercial clinical AI platforms to publish an EU AI Act compliance roadmap, with notified body engagement for high-risk classification under Annex III; also leads industry working groups on GMLP implementation.
Tempus AI — FDA-cleared genomic and multimodal diagnostic AI; governance framework includes prospective bias monitoring across patient demographics, PCCP filings for continuous-learning oncology models, and structured GMLP annexes in 510(k) submissions.
Novartis — Established one of pharma's first formal AI Ethics Boards in 2022; governance program covers clinical trial AI bias assessments, AI vendor due diligence standards, and a mandatory human oversight protocol for any AI system informing go/no-go development decisions.
Flatiron Health (Roche subsidiary) — Builds regulatory-grade real-world evidence pipelines with governance infrastructure (data provenance tracking, confounding documentation, reproducibility audits) meeting FDA RWE framework standards for oncology label expansions.
Veeva Systems — Provides regulated AI infrastructure (Vault AI) for clinical data management, regulatory submissions, and quality systems; governance features include 21 CFR Part 11-compliant audit trails, role-based access controls, and AI output versioning for submission packages.

Challenges & Considerations

Explainability vs. Predictive Performance — Regulators increasingly require that high-risk clinical AI provide human-interpretable explanations for outputs, but the most accurate models (large transformer architectures, deep ensembles) are inherently opaque. Companies face a fundamental tension between deploying best-in-class models and meeting explainability requirements under GMLP and the EU AI Act — a tradeoff that often results in deliberate architectural downgrading for regulatory contexts.
Continuous Learning and Static Approval Paradigms — Drug regulators built their frameworks around static, validated products. AI systems that retrain on real-world data — improving with deployment but drifting from their validated baseline — challenge the fundamental assumption of approval stability. The FDA's PCCP mechanism helps but requires companies to anticipate all future change types in advance, a constraint that limits adaptive AI architectures.
Cross-Jurisdictional Compliance Fragmentation — A global pharma company deploying an AI diagnostic tool faces overlapping and sometimes conflicting requirements: FDA GMLP, EU AI Act Annex III, Japan's PMDA AI guidance, China's NMPA AI regulations, and Canada's MHRA alignment. Building a single global compliance framework requires harmonization of documentation, audit trail formats, and human oversight protocols that no single regulatory framework mandates — creating enormous compliance overhead.
Algorithmic Bias in Clinical Populations — AI models trained on historically skewed clinical datasets (over-representing white male patients in many disease areas) risk perpetuating or amplifying health disparities when deployed. FDA's guidance on bias in AI/ML medical devices and the EU AI Act's non-discrimination requirements impose obligations to evaluate and document model performance across demographic subgroups — but the industry lacks standardized benchmarks for what "acceptable" disparity looks like.
Data Privacy at the Intersection of HIPAA, GDPR, and AI Training — Training high-quality clinical AI requires large volumes of real patient data, creating tension with HIPAA's minimum necessary standard, GDPR's purpose limitation principle, and the EU AI Act's training data governance requirements. Federated learning and synthetic data generation have emerged as partial solutions, but each introduces its own governance complexity around data fidelity validation and regulatory acceptance of synthetic training inputs.
Vendor AI Liability and Third-Party Governance — Pharma companies increasingly rely on AI tools from CROs, technology vendors, and SaaS platforms. Under FDA GMLP and EU AI Act Article 28 (obligations of deployers), the pharma company bears regulatory responsibility for AI outputs even when the underlying model is a black-box vendor product. Establishing contractual AI governance standards — requiring vendors to provide model cards, bias audits, change notifications, and audit trail access — is now a legal and regulatory necessity that most vendor contracts have not historically accommodated.