Multimodal, Generative, and Agentic AI for Pathology

Event: Unknown Conference 2024 · Duration: 0 min · ▶ Watch on YouTube

Abstract

This talk presents an update on the development of multimodal, generative, and agentic AI for computational pathology. It covers recent advancements in foundation models like UNI and CONCH for whole slide image analysis, and TITAN for multimodal slide-level representation learning. The presentation also introduces PathChat, a generative AI copilot for human pathology, and Kronos, a spatial proteomics foundation model. Finally, it previews Apollo, a multimodal and temporal foundation model for comprehensive patient data, aiming to unlock capabilities across clinical care and discovery.

Speakers

  • Faisal Mahmood, Ph.D. — Associate Professor, Harvard Medical School, Department of Pathology, BWH and MGH, Cancer Data Science Program, Dana Farber Cancer Center, Broad Institute of Harvard and MIT

Talks (1)

  • 00:00:00 — Faisal Mahmood, Ph.D.: Multimodal, Generative, and Agentic AI for Pathology
    • An overview of foundation models for computational pathology, including unimodal, multimodal, and generative AI approaches, and their applications in diagnosis, drug discovery, and patient care.

Key Takeaways

  • Foundation models like UNI and CONCH have achieved significant success in computational pathology, demonstrating strong performance in diverse tasks and widespread adoption.
  • Multimodal approaches, integrating histology with text (PathChat), immunohistochemistry (MADELINE), transcriptomics (TANGLE), and genomics (THREADS), significantly improve representation learning and predictive capabilities for complex tasks like treatment response.
  • Generative AI, exemplified by PathChat, is evolving into autonomous agents capable of triaging cases, ordering additional tests, and generating pathology reports, streamlining clinical workflows.
  • New foundation models like Kronos are being developed for advanced biomedical discovery in spatial proteomics, enabling phenotyping, artifact classification, and spatial biomarker discovery.
  • Future directions include building comprehensive multimodal and temporal patient foundation models (Apollo) that integrate all available patient data to enable early diagnosis, dynamic risk scores, and clinical trial matching.

Methods / Models / Datasets Mentioned

  • UNI
  • CONCH
  • DINOv2
  • CoCa
  • OpenAI CLIP
  • PLIP
  • BiomedCLIP
  • PANTHER
  • TITAN
  • PathChat
  • MADELINE
  • TANGLE
  • THREADS
  • HuatuoGPT-Vision-7B
  • PA LLaVA
  • Qwen2.5-VL-7B
  • LLaVA-0V-7B
  • Quilt-LLaVA
  • LLaVA-Med 1.5
  • Gemini-2.0-Flash
  • Llama3.2-11B
  • GPT-4o
  • Claude-3.5-Sonnet
  • Kronos
  • CODEX
  • COMET
  • Apollo

Topics

Computational Pathology · Foundation Models · Multimodal AI · Generative AI · Agentic AI · Whole Slide Imaging (WSI) · Drug Discovery · Patient Stratification · Spatial Proteomics · Clinical Decision Support

Notes

Open for commentary — connections to other work, critiques, follow-up reading.