Perception and simulation for self-driving vehicles

Event: CVPR 2025 Workshop on Autonomous Driving · Duration: 29 min · ▶ Watch on YouTube

Abstract

The presentation explores the intersection of perception and simulation in autonomous driving. It begins by discussing data-driven simulators like NuPlan and Waymax, emphasizing the importance of reactive agent models for realistic evaluation of planning algorithms. The talk then delves into motion understanding, presenting methods for accurate behavioral forecasting using techniques like Neural Scene Flow Prior and distillation into feedforward models. Finally, it introduces the concept of spatio-temporal scenario mining, leveraging large language models and explicit code generation to identify safety-critical and interesting scenarios for robust AV testing and validation.

Speakers

  • Deva Ramanan — Carnegie Mellon University

Talks (1)

  • 00:04 — Deva Ramanan: Perception and simulation for self-driving vehicles
    • This talk covers recent developments in data-driven simulators for autonomous driving, focusing on how to build and evaluate reactive agent models, understand motion through scene flow, and mine interesting scenarios for testing.

Key Takeaways

  • Data-driven simulators, particularly those with reactive agents, are crucial for evaluating autonomous vehicle planning algorithms, moving beyond simple log replays.
  • City-specific driving behaviors can be modeled and integrated into simulators using inverse optimal control to tune reactive agent parameters, leading to more realistic and accurate simulations.
  • Understanding and predicting motion at scale can be achieved through self-supervised methods like distilling complex optimization-based scene flow models into efficient feedforward networks.
  • Object-centric approaches (detector + tracker) can outperform traditional scene flow methods for small objects, challenging conventional high-level vs. low-level vision paradigms.
  • Spatio-temporal scenario mining, leveraging large language models and programmatic instantiations, is essential for curating diverse and safety-critical test cases to validate AV stacks.

Methods / Models / Datasets Mentioned

  • NuPlan
  • Waymax
  • Intelligent Driver Model (IDM)
  • PID controller
  • Adaptive Cruise Control (ACC)
  • Model Predictive Control (MPC)
  • Inverse Optimal Control (IOC)
  • BehaviorNet
  • Neural Scene Flow Prior (NSFP)
  • FastFlow3D
  • ZeroFlow
  • TrackFlow
  • Eulerian Scene Flow
  • Referential Multi-Object Tracking (RMOT)
  • Large Language Models (LLM)

Topics

Autonomous Driving · Simulation · Perception · Planning · Reactive Agents · Model Predictive Control · Motion Understanding · Scene Flow · Self-Supervision · Scenario Mining · Safety Testing · Behavioral Modeling

Notes

Open for commentary — connections to other work, critiques, follow-up reading.