AI agents in cancer research and oncology

Event: CVPR 2025 · Duration: 0 min · ▶ Watch on YouTube

Abstract

The presentation traces the evolution of artificial intelligence in biomedical research, highlighting key milestones from classical machine learning to the latest advancements in large language models (LLMs) and AI agents. It emphasizes the shift from single-purpose deep learning models to generalist models capable of reasoning and interacting with various tools. The speaker discusses the application of LLMs in medical reasoning, structuring unstructured data, and even outperforming human doctors in certain diagnostic tasks. A significant portion of the talk is dedicated to AI agents, which combine LLMs with external tools and memory to perform complex, multi-step tasks autonomously, with a focus on their potential to revolutionize cancer research, drug discovery, and clinical decision-making.

Speakers

  • Jakob Nikolas Kather — TU Dresden, Germany

Talks (1)

  • 00:00:00 — Jakob Nikolas Kather: AI agents in cancer research and oncology
    • This talk provides an overview of the evolution of biomedical AI, from classical machine learning to deep learning, foundation models, generalist models, and the emerging field of AI agents, focusing on their applications and implications in cancer research and oncology, including medical reasoning, data structuring, and autonomous decision-making.

Key Takeaways

  • Biomedical AI has evolved from classical ML to powerful generalist models and AI agents, with LLMs demonstrating superior reasoning capabilities in medical diagnosis compared to human clinicians.
  • LLMs and Vision-Language Models (VLMs) can effectively process and structure unstructured medical data, and their performance can be significantly enhanced through few-shot in-context learning by providing relevant examples or guidelines.
  • AI agents, by integrating LLMs with external tools, knowledge bases, and memory, can automate complex, multi-step tasks in healthcare and research, including oncology decision-making and even building new software tools.
  • While AI agents offer immense potential, they also introduce new challenges, such as susceptibility to malicious or incidental prompt injection attacks, which require careful investigation and mitigation.
  • The future of medicine will likely involve humans collaborating with AI agents as teammates, with the ultimate vision of a fully agentized economy where AI agents autonomously perform tasks that currently require human computer interaction.

Methods / Models / Datasets Mentioned

  • Classical Machine Learning methods
  • Deep Learning models
  • Foundation models
  • Self-supervised learning
  • Generalist models
  • Large Language Models (LLMs)
  • AI agents
  • GPT-4
  • GPT-4o
  • O1-preview
  • IsabelHealth
  • PEPID
  • ISABEL
  • DXPlain
  • Diagnosis Pro
  • PathChat
  • PatchCamelyon
  • MHIST
  • ResNet-18
  • ResNet-50
  • Tiny ViT
  • Small ViT
  • GPT-4V
  • Claude 3.5 Sonnet
  • GPT-4
  • GPT-4o
  • Haiku Core
  • OpenAI O1
  • DeepSeek
  • Gemini Thinking
  • Google
  • PubMed
  • OncoKB
  • Python Interpreter
  • Calculator
  • Imaging AI
  • Radiology AI
  • Pathology AI
  • Specialist AI
  • MedSAM
  • MSI KRAS BRAF Segmentation
  • Toolmaker project
  • STAMP

Topics

AI agents · Cancer research · Oncology · Large Language Models (LLMs) · Biomedical AI evolution · Medical reasoning · Vision-Language Models (VLMs) · Prompt injection attacks · Agentized workflows · Clinical decision-making

Notes

Open for commentary — connections to other work, critiques, follow-up reading.