Scalable Autonomous Driving via Fully Data-driven Simulation

Event: CVPR 2025 · Duration: 0 min · ▶ Watch on YouTube

Abstract

This presentation introduces a framework for scalable autonomous driving through fully data-driven simulation. It details a modular end-to-end AV stack that integrates learned perception, prediction, and planning modules, emphasizing long-range multimodal capabilities crucial for highway trucking. The talk highlights the development of physically-grounded generative world simulators (Gen 2) and a novel scenario generation model called Scenario Dreamer, which synthesizes diverse and challenging driving environments. This approach aims to overcome limitations of real-world data collection and enable robust, closed-loop training and validation of autonomous driving systems.

Speakers

  • Felix Heide — Princeton / Torc Robotics

Talks (1)

  • 00:05:16 — Felix Heide: Scalable Autonomous Driving via Fully Data-driven Simulation
    • This talk presents a data-driven approach to scalable autonomous driving, focusing on end-to-end AV stacks, long-range multimodal perception, behavior simulation, and generative world simulators for robust training and validation.

Key Takeaways

  • Data-driven simulation is crucial for scalable autonomous driving, enabling end-to-end training and validation of complex AV stacks.
  • Modular AV stacks can be trained end-to-end by passing gradients through differentiable components, including conventional algorithms, to optimize the entire system.
  • Long-range multimodal perception (beyond 1km) is essential for highway driving, requiring efficient sparse BEV representations that scale linearly with range, unlike dense representations which scale quadratically.
  • Physically-grounded generative world simulators can create diverse and challenging scenarios, including rare crash events and adversarial behaviors, for robust planner training and validation.
  • Generative models like Scenario Dreamer can synthesize limitless, realistic, and diverse test scenarios, augmenting limited real-world data and enabling closed-loop training and evaluation of AV planners.

Methods / Models / Datasets Mentioned

  • nuScenes
  • Waymo
  • Waymax
  • DriveSceneGen (GT Raster)
  • SLUDGE (DIT-L)
  • SLUDGE (DIT-XL)
  • Ctrl-Sim
  • Scenario Dreamer
  • PPO
  • Radiance Fields
  • Transformer
  • Latent Diffusion Model
  • Autoencoder
  • Diffusion Inpainting

Topics

Autonomous Driving · Data-driven Simulation · End-to-end Learning · Multimodal Perception · Long-range Vision · Behavior Prediction · Planning · Reinforcement Learning · Generative Models · World Simulators · Scenario Generation · Sparse BEV Representations · Trucking

Notes

Open for commentary — connections to other work, critiques, follow-up reading.