End-to-end Autonomous Driving: Past, Current and Onwards

Event: CVPR 2025 Workshop on Autonomous Driving · Duration: 26 min · ▶ Watch on YouTube

Abstract

This talk provides a comprehensive overview of end-to-end autonomous driving, tracing its evolution from early imitation learning methods to the current era of foundation models. It introduces the concept of ‘End-to-end 2.0’ centered around a ‘World Engine’ designed to address the challenges of rare corner cases and data scarcity. Recent progress in adaptive scene generation (Nexus) and multi-traversal Gaussian splatting (MTGS) for realistic sensor simulation is highlighted. The presentation concludes by discussing the critical need for scalable reinforcement learning and pseudo-simulation techniques to reliably benchmark and improve autonomous vehicles, aiming for robust and shippable solutions.

Speakers

  • Hongyang Li — The University of Hong Kong

Talks (1)

  • 00:04 — Hongyang Li: End-to-end Autonomous Driving: Past, Current and Onwards
    • An overview of end-to-end autonomous driving, covering its historical roadmap, the concept of a ‘World Engine’ for E2E 2.0, recent progress in scene generation and simulation, and key challenges like scalable reinforcement learning.

Key Takeaways

  • End-to-end Autonomous Driving (E2E AD) has advanced significantly, with the ‘World Engine’ representing the next generation of this paradigm.
  • Nexus offers improved controllability and reactivity for scene behavior, crucial for generating and handling corner cases.
  • MTGS ensures multi-traversal realism in large spaces, enhancing the fidelity and utility of simulated environments.
  • The combination of Nexus and MTGS forms a powerful data engine capable of generating challenging corner cases for robust AD development.
  • Scalability in E2E AD, particularly for reinforcement learning, will emerge from efficient ‘sampling’ strategies facilitated by advanced data engines and pseudo-simulation.

Methods / Models / Datasets Mentioned

  • CNN E2E
  • BDDV
  • CILS
  • CILRS
  • Darb
  • AgileAD
  • SafeDagger
  • NEAT
  • NMP
  • BDD-X
  • PlanT
  • PPGeo
  • SelfD
  • ACO
  • UniAD
  • P3
  • MP3
  • ST-P3
  • DriveDreamer
  • GenAD & Vista
  • DriveLM
  • Lingo-2
  • Cosmos-predict1
  • Gala-2
  • NVM
  • SimLingo
  • DriveVLM
  • Nexus
  • MTGS
  • NAVSIM v2
  • Diffusion Policy
  • Diffusion Planner
  • GPT-2
  • GUMP
  • IDM
  • SceneD

Topics

End-to-end Autonomous Driving · World Engine · Scene Generation · Gaussian Splatting · Pseudo-Simulation · Reinforcement Learning · Corner Cases · Foundation Models · Autonomous Driving Evaluation · Sensor Simulation

Notes

Open for commentary — connections to other work, critiques, follow-up reading.