Foundation Models for Vision: From Vision to Clinical Reality

Event: Conference Talk 2025 · Duration: 0 min · ▶ Watch on YouTube

Abstract

This talk explores the integration of AI in medicine, focusing on the challenges and successes of human-AI collaboration, particularly in medical imaging. It highlights that while AI can achieve superhuman performance in specific tasks, simply providing AI assistance to clinicians does not always improve diagnostic quality. The speaker proposes a shift from forced collaboration to role separation, leveraging the distinct strengths of AI and human experts. The presentation also delves into new models and frameworks for generalist medical AI, multimodal generative AI, and real-time clinical applications, emphasizing the need for robust validation and addressing challenges like AI hallucinations and reimbursement.

Speakers

  • Pranav Rajpurkar — Associate Professor, Harvard Medical School; Co-founder, a2z Radiology AI

Talks (1)

  • 00:00:00 — Pranav Rajpurkar: Foundation Models for Vision: From Vision to Clinical Reality
    • An exploration of the challenges and successes of AI integration in medicine, advocating for role separation and introducing new generalist AI models and clinical applications.

Key Takeaways

  • Direct AI assistance does not consistently improve physician performance; instead, role separation and personalized implementation are crucial for successful AI integration in medicine.
  • AI confidence, not just raw capability, dictates the effectiveness of AI assistance, with uncertain AI predictions potentially harming human performance.
  • Generalist AI models and multimodal generative AI (like MedVersa and GenMI) are emerging as a promising paradigm to handle diverse medical tasks and data types, moving beyond narrow, specialized AI.
  • New frameworks like CRAFT-MD and ReXplain are being developed to evaluate AI in more realistic clinical scenarios, such as patient-AI conversations and patient-friendly report generation.
  • Addressing critical challenges like detecting and mitigating AI hallucinations (RADFLAG, FACTCHECKER) and establishing clear reimbursement pathways are essential for the widespread adoption and clinical impact of advanced medical AI.

Methods / Models / Datasets Mentioned

  • NBER Working Paper No. 31422
  • Nature Medicine (Yu, F. et al., 2024)
  • MASAI Trial
  • PRAIM Study
  • The Robot Doctor Will See You Now (Rajpurkar and Topol, 2023)
  • CRAFT-MD (John, S. et al., 2024)
  • GPT-4
  • GPT-3.5
  • LLaMA-2
  • Mistral
  • ReXplain (Luo, L. et al., 2024)
  • Foundation models for generalist medical artificial intelligence (Zhou, H. et al., 2023)
  • GenMI (Rajpurkar, Topol et al., 2025)
  • MedVersa (Zhou, H. et al., 2024)
  • REXRANK LEADERBOARD
  • MIMIC-CXR
  • CheXpert
  • MC-MED (PhysioNet, 2025)
  • REXGRADIENT-160K SCALE (Zhong, X. et al., 2023)
  • REXVQA (Pai, A. et al., 2023)
  • Gemini
  • Qwen-2.5
  • Phi-3.5
  • LLaVA
  • RADFLAG Black-Box Method (Zhong, S. et al., 2024)
  • FACTCHECKER (Heimann, A. et al., 2024)
  • Coordinated AI Agents for advancing healthcare (Hortz, M. et al., 2023)
  • MASH Healthcare AI Agent Network

Topics

AI in medicine · human-AI collaboration · AI integration challenges · role separation · generalist AI · multimodal AI · medical imaging · patient interaction · AI hallucinations · medical AI reimbursement

Notes

Open for commentary — connections to other work, critiques, follow-up reading.