v2e: From Video Frames to Realistic DVS Events

Event: Research Presentation · Duration: 3 min · ▶ Watch on YouTube

Abstract

This presentation introduces v2e, a novel toolbox designed to convert conventional video frames into realistic event camera data. The speakers first debunk common myths about event cameras, specifically regarding motion blur and latency, highlighting that event cameras do exhibit motion blur under certain conditions and their latency is often higher than microsecond claims. The v2e toolbox incorporates models for various DVS non-idealities, including Gaussian event threshold mismatch, finite intensity-dependent bandwidth, leak events, and temporal noise, to produce highly realistic synthetic event streams. The utility of v2e-generated events is demonstrated through transfer learning experiments on object recognition (N-Caltech 101) and car detection (MVSEC) tasks, showing that models pre-trained on v2e data can outperform or significantly improve upon supervised baselines, especially in challenging low-light conditions.

Speakers

  • Yuhuang Hu — ETH Zurich
  • Shih-Chii Liu — University of Zurich
  • Tobi Delbruck — Institute of Neuroinformatics

Talks (1)

  • 00:00:00 — Yuhuang Hu: v2e: From Video Frames to Realistic DVS Events
    • This talk introduces v2e, a toolbox for generating realistic event camera data from standard video frames, and demonstrates its utility for transfer learning in object recognition and detection tasks.

Key Takeaways

  • Event cameras are not entirely free of motion blur, especially in low-light conditions, contrary to common belief.
  • Practical event camera latency is often in the millisecond range, not microseconds, due to real-world conditions.
  • The v2e toolbox generates highly realistic event camera data by modeling various DVS non-idealities.
  • Synthetic event data from v2e is highly effective for transfer learning, improving performance in object recognition and detection tasks.
  • Models pre-trained on v2e events can significantly outperform traditional frame-based methods, particularly in challenging environments like night driving.

Methods / Models / Datasets Mentioned

  • v2e toolbox
  • ResNet34
  • YOLOv3
  • N-Caltech 101 dataset
  • MVSEC dataset
  • EST (Gehrig, et al., 2019)

Topics

Event cameras · DVS (Dynamic Vision Sensor) · Synthetic data generation · Motion blur · Latency · Transfer learning · Object recognition · Car detection · Non-idealities modeling · Low-light conditions

Notes

Open for commentary — connections to other work, critiques, follow-up reading.