Learning the Language of Patients

Event: CVPR 2025 Workshop on Foundation Models for Text-Guided 3D Biomedical Image Segmentation · Duration: 0 min · ▶ Watch on YouTube

Abstract

This presentation delves into the transformative potential of AI, particularly generative AI and multimodal foundation models, in revolutionizing precision health. The speaker argues that technology can democratize high-quality healthcare, making it accessible to everyone and even surpassing the care available to the most privileged today. By ‘learning the language of patients’ from diverse real-world data, including radiology, pathology, and genomics, the goal is to create high-fidelity ‘virtual patients’ or digital twins. This approach promises to automate mundane tasks, accelerate medical discovery, and enable population-scale in silico clinical trial simulations, ultimately leading to improved patient care and emergent capabilities in biomedical research.

Speakers

  • Hoifung Poon — Microsoft Health Futures

Talks (1)

  • 00:00 — Hoifung Poon: Learning the Language of Patients
    • This talk explores how generative AI and multimodal foundation models can ‘learn the language of patients’ from real-world data to democratize precision healthcare, accelerate discovery, and enable population-scale clinical trial simulations.

Key Takeaways

  • Generative AI can democratize high-quality healthcare by automating complex data analysis and making it accessible, potentially leading to better outcomes for all patients.
  • Structuring unstructured real-world health data using ‘Universal Medical Abstraction’ with frontier models can drastically reduce the cost and time associated with tasks like clinical trial eligibility abstraction.
  • Learning the ‘language of patients’ involves integrating and interpreting diverse multimodal data (e.g., radiology, pathology, genomics) to build comprehensive patient embeddings or ‘digital twins’.
  • Text can serve as an ‘interlingua’ to bridge different modalities, allowing for cross-modal reasoning and enabling applications like ‘talking to the image’ for medical image analysis.
  • Population-scale Real-World Evidence, combined with advanced AI, opens new avenues for hypothesis generation and testing through ‘in silico’ clinical trial simulations, accelerating medical discovery and improving patient care.

Methods / Models / Datasets Mentioned

  • GPT-4
  • RAG
  • GigaPath
  • BiomedParse
  • Spotlight
  • LLaVA-Med
  • BiomedCLIP
  • Transformer
  • Dilated Attention
  • X-Reasoner
  • TrialScope
  • MIMIC-IV

Topics

Precision Health · Generative AI · Multimodal Data · Real-World Evidence (RWE) · Clinical Trial Matching · Digital Pathology · Patient Embedding · Foundation Models · Agentic Systems · In Silico Clinical Trials

Notes

Open for commentary — connections to other work, critiques, follow-up reading.