Event-based Visual Odometry: A Short Tutorial

Event: CVPR 2021 Event-based Vision Workshop · Duration: 16 min · ▶ Watch on YouTube

Abstract

This tutorial introduces event-based visual odometry (VO), beginning with the fundamental principles and properties of event-based cameras, highlighting their advantages over traditional cameras. It then delves into the challenges of processing event streams and provides a comprehensive literature review of existing event-based VO methods, categorizing them by mapping and camera pose tracking techniques. The tutorial also presents ESVO, an event-based stereo visual odometry system, detailing its mapping and tracking modules, including the use of time-surface maps and probabilistic fusion. Finally, it showcases evaluation results on various datasets and concludes with key takeaways regarding latency, computational complexity, and power consumption in event-based vision.

Speakers

  • Dr. Yi Zhou — HKUST-DJI Joint Lab, Dept. ECE at HKUST

Talks (1)

  • 00:00:00 — Dr. Yi Zhou: Event-based Visual Odometry: A Short Tutorial
    • This tutorial provides an overview of event-based cameras, their advantages and challenges, and a literature review of event-based visual odometry (VO) methods, focusing on mapping and camera pose tracking, before introducing the ESVO system.

Key Takeaways

  • Event-based cameras offer high speed, low latency, high dynamic range, and ultra-low power consumption, making them advantageous over traditional frame-based cameras.
  • The core problem in event-based visual odometry is data association on events, which involves establishing a measurement model for recursive state estimation.
  • The ESVO system leverages stereo event cameras and time-surface maps for robust 3D reconstruction and camera pose tracking, running in real-time on a standard CPU.
  • There is a trade-off between latency and computational complexity in event-based solutions, with ‘bash’ processing often preferred over ‘event-by-event’ for higher accuracy and real-time performance.
  • Many current event-based solutions still rely on high-power platforms, indicating a need for more compact and energy-efficient solutions to fully realize the potential of neuromorphic engineering.

Methods / Models / Datasets Mentioned

  • DVS (Dynamic Vision Sensor)
  • PTAM [ISMAR 07]
  • EVO [RAL 17]
  • ESVO [T-RO 21]
  • H. Kim et al. [ECCV 16] (Real-time 3D reconstruction and 6-DoF tracking with an event camera)
  • G. Gallego et al. [T-PAMI 18] (Event-based, 6-DOF camera tracking from photometric depth maps)
  • H. Rebecq et al. [BMVC 17] (EMVS: Event-based multi-view stereo—3D reconstruction with an event camera in real-time)
  • Ultimate SLAM [RAL 18]
  • D. Weilerdsorfer et al. [ICVS 2013] (Simultaneous localization and mapping for event-based vision systems)
  • H. Kim et al. [BMVC 2014] (Simultaneous mosaicing and tracking with an event camera)
  • G. Gallego et al. [RAL 2017] (Accurate angular velocity estimation with an event camera)
  • S. Bryner et al. [ICRA 2019] (Event-based, direct camera tracking from a photometric 3D map using nonlinear optimization)
  • Extended Kalman Filters (EKFs)
  • Time-Surface Maps
  • Forward Compositional Lucas-Kanade method
  • upenn_indoor_flying1 (dataset)
  • hkust_lab (dataset)
  • hkust_HDR (dataset)
  • Dsec (Zurich_City_04_a) dataset

Topics

Event-based cameras · Visual odometry · SLAM · 3D reconstruction · Camera pose tracking · Stereo vision · Time-surface maps · Probabilistic fusion · Low latency · High dynamic range

Notes

Open for commentary — connections to other work, critiques, follow-up reading.