Evaluating AI Models for Medical Applications

Today, I released the 2026 DrGPT #AIHealthcareIndex. This is a 54-page, evidence-based analysis of more than 150 AI healthcare companies built to separate clinical impact from hype. AI in healthcare is projected to reach $543B. But 74% of tools lack meaningful clinical validation. Inside the report: • A composite scoring model weighted toward clinical outcomes • A breakdown of FDA clearance pathways and evidence strength • A “Theranos” red flag checklist for vendor claims • A governance risk matrix for hospital boards • A Top 25 AI Healthcare Leaders ranking • A procurement and implementation checklist for health systems • A bias & equity scorecard • A 19-slide visual summary designed for boardrooms and policy discussions This Index is clinician-led, safety-first, and explicitly flags data gaps. It is not vendor-sponsored. It is not capital-weighted. It is bedside-anchored. If you are a hospital executive, CMO, CIO, regulator, policymaker, investor, or clinician evaluating AI, this report was written for you. AI is already transforming imaging, workflow, revenue cycle, and pharma R&D. The question is whether we scale it safely, equitably, and with evidence. You can read the full 2026 DrGPT AI Healthcare Index here: and on my website #CHATGPTHEALTHdotcom under resources I welcome thoughtful feedback from clinicians, health system leaders, and AI developers committed to raising the standard. Harvey Castro, MD, MBA. Emergency Physician Chief AI Officer, Phantom Space Corp. #DrGPTAIHealthcareIndex

We just published in Nature Medicine: a framework for the next phase of clinical AI evaluation. The core idea — we should stop giving clinical AI written exams. We need to put it in a flight simulator. The medical AI community has an obsession with static benchmarks. We celebrate LLMs for passing the USMLE or diagnosing isolated, perfectly packaged text snippets. But real medicine isn't a multiple-choice test — it's a dynamic, resource-constrained environment where every choice creates a ripple effect. Our new Perspective (Luo et al.) proposes the Clinical Environment Simulator (CES): instead of static datasets, evaluate AI inside a digital hospital where every decision dynamically alters future states. Here's why this exposes the gaps in current AI tools: - The illusion of time. Static benchmarks ignore the clock. Patients deteriorate. If an AI orders a "gold standard" scan but the radiology queue is three hours long, what happens next? A simulator forces the AI to reason temporally and adapt. - The resource ripple effect. Decisions are zero-sum. An aggressive workup for one patient might exhaust the lab or bed capacity needed to stabilize another. AI must balance individual optimization with system-wide efficiency. - The interface bottleneck. Generating a text-based diagnosis is easy. Translating that into action — navigating EHR interfaces, placing orders, fitting into the workflow of a human care team — is where the friction lives. We've seen this lesson before. Aviation didn't achieve its safety record with paper tests — it built simulators that throw dynamic weather and system failures at pilots. Autonomous driving didn't get validated by passing a written DMV exam — it took millions of miles in simulation with unpredictable pedestrians, weather, and edge cases. Medicine needs the same shift. If we want AI capable of genuine collaboration on the hospital floor, we need to start testing it under the same operational realities. With my fantastic co-authors Luyang Luo (first author) with Sung Eun Kim, Xiaoman Zhang, Julius M. Kernbach, MD, Roshan Kenia, Julián Nicolás Acosta, Larry Nathanson, Adrian Haimovich, Adam Rodman, Ethan Goh, MD, Jonathan H. Chen, Nigam Shah, David Kim, James Zou, Faisal Mahmood, Jakob Nikolas Kather, Matt Lungren MD MPH, Vivek Natarajan, Eric Topol, MD

AI models in medical imaging often boast high accuracy, but are we measuring what really matters? 1️⃣ Many AI models are judged using metrics that do not match clinical goals, like relying on AUROC (area under the receiver operating characteristic curve, which shows how well the model separates classes) in imbalanced datasets where rare but critical findings are overlooked. 2️⃣ A single metric such as accuracy or Dice can be misleading. Multiple, task-specific metrics are essential for a robust evaluation. 3️⃣ In classification, AUROC can stay high even if a model misses rare cases. AUPRC (area under the precision-recall curve, which focuses on the model's performance on the positive class) is more useful when positives are rare. 4️⃣ For regression, MAE (mean absolute error, the average size of prediction errors) and RMSE (root mean squared error, which gives more weight to large errors) do not reflect how serious the errors are in real clinical settings. 5️⃣ In survival analysis, the C-index (concordance index, which measures how well predicted risks match actual outcomes) and time-dependent AUCs (area under the curve at specific time points) each reflect different things. Using the wrong one can mislead. 6️⃣ Detection models need precision-recall metrics like mAP (mean average precision, which combines detection quality and location accuracy) or FROC (free-response receiver operating characteristic, which shows sensitivity versus false positives per image). Accuracy is not useful here. 7️⃣ Segmentation metrics like Dice (which measures the overlap between predicted and true regions) and IoU (intersection over union, the overlap divided by the total area) can miss small but important errors. Visual review is often needed. 8️⃣ Calibration means checking if predicted risks match observed outcomes. ECE (expected calibration error, the average gap between predicted and actual risks) and the Brier score (the mean squared difference between predicted probability and actual outcome) help assess this. 9️⃣ Foundation models need extra checks: generalization (how well they perform across tasks), label efficiency (how few labeled examples they need), and alignment across inputs and outputs. Zero-shot means no examples were given before testing. Few-shot means only a few examples were used. 🔟 Metrics must fit the clinical context. A small error in one use case may be acceptable, but the same error could be dangerous in another. ✍🏻 Burak Kocak, Michail Klontzas, MD, PhD, Arnaldo Stanzione, Aymen Meddeb MD, EBIR, Aydin Demircioglu, Christian Bluethgen, Keno Bressem, Lorenzo Ugga, Nate Mercaldo, Oliver Diaz, Renato Cuocolo. Evaluation metrics in medical imaging AI: fundamentals, pitfalls, misapplications, and recommendations. European Journal of Radiology Artificial Intelligence. 2025. DOI: 10.1016/j.ejrai.2025.100030

Introducing Healthcare AI Model Evaluator: Open-Source Evaluation for Healthcare AI 🏥 The gap between AI capability and AI trust remains one of healthcare's biggest challenges. Generic benchmarks don't answer the questions that matter most: Will this AI work for our patients? In our workflows? With our use cases? Yesterday, at Microsoft Ignite, we unveiled Healthcare AI Model Evaluator—an open-source framework designed to help healthcare organizations evaluate AI systems on their own terms, with their own data, fully within their control. Healthcare AI Model Evaluator puts rigorous evaluation directly in the hands of healthcare organizations—enabling them to assess any AI system using their own clinical data, success criteria, and expertise. Key principles: ✅ Data sovereignty: Deploy within your secure infrastructure—your data stays in your control ✅ Built-in, no-code human evaluation: Intuitive workflows designed for clinicians without programming expertise to provide expert feedback and validate AI outputs ✅ Clinical task alignment: Define evaluations that reflect your real-world priorities—from diagnostic support to administrative workflows ✅ Model agnostic: Evaluate any AI system—commercial APIs, open-source models, or proprietary solutions ✅ Expert-driven: Leverage your clinical team's expertise to establish criteria, interpret results, and validate performance Built for collaboration: This is just the beginning. Healthcare AI evaluation is too important to solve alone, and we're committed to building this tool with the community—clinicians, data scientists, researchers, and healthcare leaders who understand these challenges first hand.   This would not be possible without our incredible team: Vincent Fitzgerald, Leonardo Schettini, Hao Qiu, Wen-wai Yim whose hard work, expertise and dedication made this possible. Also great thanks to our collaborators within HLS AI Frontiers and MSR research teams: Jameson M., Alberto Santamaria-Pang, PhD, Ivan Tarapov, Alexander Mehmet Ersoy, Erika Strandberg, Naiteek Sangani, Chris Burt, Harshita Sharma, Javier Alvarez Valle, Mu Wei and many others.   Get involved: 📁 Explore the repository: https://lnkd.in/eggbZz_T 💬 Share your thoughts, use cases, and feedback 🤝 Join us in making healthcare AI evaluation transparent, rigorous, and accessible The future of healthcare AI depends not just on building better models—but on evaluating them better. Let's build that future together. #HealthcareAI #OpenSource #AIEvaluation #HealthTech #ClinicalAI #DigitalHealth #HLSAIFrontiers

Thrilled to share our new The Lancet Digital Health Viewpoint on the chaotic universe of AI performance metrics colliding with the realities of clinical care. In this piece, we tackle a simple question: How should we actually evaluate predictive AI models intended for medical practice? With 32 different metrics circulating across discrimination, calibration, overall performance, classification, and clinical utility, it’s no wonder the field is confused and sometimes misled. Our analysis shows why selecting the right performance measures is not just a statistical preference but a clinical imperative. We highlight two essential characteristics that truly matter: 1. whether a metric is correct (optimized only when predicted probabilities are correct), and 2. whether it reflects statistical vs. decision-analytical performance in a way that aligns with real clinical consequences. The results are striking: some of the most widely used metrics, including the beloved F1 score, fail spectacularly when evaluated through a clinical lens. We offer clear recommendations: report AUC, calibration plots, net benefit with decision curve analysis, and probability-distribution plots. These metrics together provide the transparency and rigor required for safe, reliable deployment. Proud of this work, proud of the team Ben Van Calster Ewout Steyerberg Gary Collins Andrew Vickers Laure Wynants Maarten van Smeden Karandeep Singh and many others, and deeply hopeful that this brings more clarity, accountability, and clinical grounding to how we evaluate AI in healthcare.

Healthcare AI evaluation has a problem: we're still asking "what's the accuracy?" as if a single number could tell us whether a model is safe for clinical deployment. It can't. Every other high-stakes field has learned this lesson. Banks, schools, and the military all discovered that complex systems require multiple evaluation approaches because they fail in complex ways. While none of these are perfect approaches, they've all moved toward suites of evaluations. Federal banking regulators require three distinct validation components: Conceptual soundness (is the design theory-based?), outcome analysis (does it produce accurate results under various conditions?), and ongoing monitoring (does it keep working as conditions change?). A model might pass one and fail another. That's why you need all three. Education moved beyond standardized tests: While standardized exams measure basic skills efficiently, they miss creativity, collaboration, and complex problem-solving. Schools using portfolio-based assessment (combining tests, portfolios, projects, and teacher observations) show higher graduation rates than those relying on standardized tests alone. Defense requires operational testing: Lab tests show if technical specs are met. Operational Test & Evaluation shows if systems work in combat-realistic conditions with operational forces. A radar system might meet all specifications in controlled tests but prove unreliable when operated by soldiers under stress. Healthcare AI faces identical challenges. A model can excel at discrimination (AUROC) but have poor calibration. It can work well on common cases but fail catastrophically on edge cases. It can perform equitably for some demographics while showing significant bias for others. We need evaluation suites that test: Statistical performance (does it discriminate and calibrate well?) Clinical performance (does it work across diverse real-world cases?) Equity (does it work for all patient populations?) Edge cases (what happens with missing data or ambiguous situations?) Workflow fit (can clinicians actually use it in practice?) Ongoing performance (does it keep working over time?) The question is whether we'll build these frameworks proactively or wait until failures force us to. Read more in my Substack on why healthcare AI evaluation needs the same multi-method approach that works for every other high-stakes field https://lnkd.in/gg5DvKPS

Stanford-Harvard study tested 31 AI tools on 100 real physician-to-specialist consultations. The best AI outperformed board-certified internists by 15+ percentage points. Human generalist physicians ranked 31st out of 33. NOHARM study just published the most rigorous evaluation of medical AI to date. Here are three findings that matter for deployment: 1. The safety-restraint paradox is real. 22% of cases had potential for severe harm. But 77% of those instances came from AI failing to suggest important actions (not from recommending something dangerous). 2. Making AI extremely cautious (adding disclaimers, limiting recommendations, defaulting to "consult a doctor") paradoxically increases harm by causing critical omissions. 3. The safest models occupy a middle ground. Too little restraint = reckless recommendations. Too much restraint = missed critical interventions. Multi-agent configurations outperform solo models. When one AI made recommendations and additional AIs reviewed them (automated second opinion), configurations did 6x better on safety. Three-agent setups outperformed two-agent ones. Combining models from different organizations (open-source + proprietary + medical knowledge system) beat using multiple versions of the same model. Best performing combo: Meta's Llama 4 Scout, Google's Gemini 2.5 Pro, AMBOSS LiSA 1.0. Medical knowledge-grounded systems beat general LLMs. AMBOSS LiSA 1.0 topped overall (62.3%). Built on curated medical knowledge base, not just internet text. Systems that scored highest on board exam questions had mediocre performance on real clinical cases. Passing medical licensing exams and safely managing real patients are different skills. The deployment implications: One in five physicians consults AI for patient care questions. Two in three use LLMs regularly in some form. But performance gaps between best and worst models are massive… worst made 3x more severe errors than best. The AI tools physicians are using may not be the ones validated for clinical decision support. If organizations deploy cautious models to minimize liability, they may increase harm through omissions. If they deploy aggressive models to maximize completeness, they increase harm through reckless recommendations. Multi-agent architectures appear to be the safest approach, but almost no one is deploying them clinically. Full leaderboard available at NOHARM public website. Top 5 were statistically similar: AMBOSS LiSA 1.0, Gemini 2.5 Pro, Glass Health 4.0, GPT-5, Claude Sonnet 4.5. *** Which AI tools are you or your organization using for clinical questions? Do you know where they rank on safety metrics? — Source: NOHARM study - Stanford/Harvard

The NYT just reported that patients are uploading entire medical records into chatbots - but the risks are not what most people think. Patients are pasting labs, imaging, clinical notes, and oncology reports directly into LLMs. • 26-year-old told her labs “most likely” indicated a pituitary tumor. MRI: normal • 63-year-old advised to escalate to catheterization. Found ~85% LAD stenosis Because of how the chatbot responds, many assume the AI reasons about their symptoms and medical record the same way a clinician does. But AI systems are capable of both meaningful help and serious error, without any calibration signal visible to the user. Most worry about wrong AI recommendations. But the bigger risk is what the AI does not say. 📊 Harm preprint study A new Stanford-Harvard study (David Wu, MD, PhD, Fateme (Fatima) Nateghi, Adam Rodman, Jonathan H. Chen et al.) evaluated 31 models on 100 real outpatient eConsult cases across 10 specialties: - 4,249 management actions - 12,747 expert ratings Severe harm per 100 cases: - Best models: ~12–15 - Worst models: ~40 ~77% of severe harms were omissions: - Not ordering a critical test - Missing a needed referral - Neglecting follow-up suggestions 🔷 Additional findings: 1) Top models outperformed generalists using conventional resources (though these were difficult eConsult cases that PCPs were posing to specialists) 2) No link between safety and model size, recency, “reasoning modes,” or standard benchmarks 3) Multi-agent + RAG approaches reduced harm; heterogeneous ensembles had ~6× higher odds of top-quartile safety 📌 Implications When a patient asks AI for medical advice, the primary risk is not incorrect recommendations. It's neglecting critical actions a clinician might suggest (notably, humans also make a lot of mistakes). ⚠️ Why this matters 1) 2/3 of US physicians report using LLMs, and millions of patients. Errors will become more subtle as models get better. Both harms of omissions and commission will become harder for clinicians (and especially patients) to detect. 2) Sampling a few outputs is not enough: clinical AI evaluation needs explicit, systematic harm measurement on real cases, not just performance or accuracy on knowledge benchmarks. 3) If we don’t measure omission harms, we will systematically underestimate risk. 🔴 Open Call: State of Clinical AI Report (Jan 2026) The ARISE Network (Stanford + Harvard) is compiling a State of Clinical AI Report for 2026. Audience: health system leaders, clinicians, researchers, tech/pharma, media, investors 2025 peer reviewed and preprint studies within scope: • Clinical AI (doctor- or patient-facing) • Benchmarks, evaluations, real-world deployments, prospective trials • Workflow, outcomes, and implementation studies 📅 Submission deadline: Dec 21, 2025 - Comment with study link + 1–2 sentences on key findings and why it matters - We will follow up with a one-slide reference example for invited submissions

Half of radiology AI models were rejected Emory’s Radiology AI Council just published an internal audit of 13 AI models used or proposed for deployment - and only 4 were fully approved. Why?    Here’s the reality check in numbers: - 9 out of 13 improved diagnostic accuracy     - 6 out of 13 didn’t provide any time savings     - 3 showed financial benefits within radiology     - 3 showed net financial losses    And here’s an interesting observation: even the model for Spine MRI Degenerative Changes, which demonstrated decreased turnaround time (by 1 hour to 1 day), ended up bringing no financial benefit. That perfectly echoes one of the points made at the recent RSNA Spotlight in Barcelona - we still need to learn how to monetize time reductions. On their own, in isolation, they don’t necessarily translate into value. Final recommendations: 4 for deployment, 3 for shadow deployment, 3 deferred, and 3 discontinued. For deployment: 1. Brain volume and segmentation 2. Coronary artery calcification 3. Aortic valve calcification 4. Breast cancer risk Another major issue: none of the evaluated models included post-deployment quantitative performance tracking. But perhaps that should actually be part of the hospital’s or teleradiology provider’s own infrastructure. Because if every vendor comes in with its own monitoring solution, you end up with a chaotic zoo of fragmented systems. Overall, fascinating statistics - it would be great to start seeing more of these analyses for comprehensive AI models. Will the numbers look any different there? Team: Hari Trivedi Bardia Khosravi Judy Gichoya Laura Benson Damian Dyckman, M.D., Ph.D. James Galt Brian Howard Elias Kikano, MD Jean Kunjummen Neil Lall, MD CIIP Xiao T. Li Sumir Patel, MD, MBA Nabile Safdar Ninad Salastekar Colin Segovis, MD, PhD Marly van Assen Peter Harri DOI: 10.1016/j.jacr.2025.05.016

𝗪𝗵𝗲𝗻 "𝗙𝗮𝗶𝗿" 𝗜𝘀𝗻'𝘁 𝗘𝗻𝗼𝘂𝗴𝗵: 𝗖𝗼𝘃𝗮𝗿𝗶𝗮𝘁𝗲 𝗕𝗶𝗮𝘀 𝗶𝗻 𝗛𝗶𝘀𝘁𝗼𝗽𝗮𝘁𝗵𝗼𝗹𝗼𝗴𝘆 𝗙𝗼𝘂𝗻𝗱𝗮𝗍𝗶𝗼𝗻 𝗠𝗼𝗱𝗲𝗹𝘀 A model can achieve equal accuracy across demographic groups and still encode harmful biases. Here's why that matters for medical AI. Research from Abubakr Shafique et al. examines a subtle but critical problem in histopathology foundation models: covariate bias. Traditional fairness metrics focus on whether models perform equally well across different patient subgroups. But what if the model is making predictions based on spurious correlations—technical artifacts, scanning differences, or institutional patterns—rather than genuine biological signals? Why this is overlooked: Most fairness assessments in medical AI check whether accuracy, sensitivity, or specificity are balanced across demographic groups. If those metrics look good, we assume the model is fair. But this misses a fundamental issue: the model might be relying on the wrong features entirely, even if its predictions happen to be correct. The covariate bias problem: - Foundation models can inadvertently learn correlations between protected attributes (like demographics) and technical confounders (like staining protocols or scanner types) - When certain patient populations are overrepresented at specific medical centers with distinct technical characteristics, the model may conflate biological differences with institutional artifacts - This creates brittle models that fail when deployed in new settings, disproportionately affecting underrepresented groups What this means for deployment: A histopathology model might show "fair" performance metrics in validation but still perpetuate inequities. If the model learned to associate certain demographic groups with specific scanning artifacts, it could fail catastrophically when those technical conditions change—creating unpredictable performance gaps that traditional fairness audits wouldn't catch. The path forward: We need to look beyond surface-level fairness metrics and examine what features our models actually rely on. This requires probing representation spaces, testing robustness across technical variations, and ensuring models generalize based on biology rather than institutional fingerprints. Fairness in medical AI isn't just about equal outcomes—it's about equal reliability and trustworthiness across all populations we serve. Read the paper: https://lnkd.in/e8JasjWm #MedicalAI #AIFairness #DigitalPathology #MachineLearning #ComputationalPathology #FoundationModels — Subscribe to 𝘊𝘰𝘮𝘱𝘶𝘵𝘦𝘳 𝘝𝘪𝘴𝘪𝘰𝘯 𝘐𝘯𝘴𝘪𝘨𝘩𝘵𝘴 — weekly briefings on making vision AI work in the real world → https://lnkd.in/guekaSPf