Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancy

Event: CVPR 2020 · Duration: 9 min · ▶ Watch on YouTube

Abstract

Event cameras are novel vision sensors that sample brightness increments asynchronously with low latency and high temporal resolution. While event streams are valuable for high-speed motion estimation, reconstructing brightness images from them allows bridging the gap with existing frame-based computer vision literature. Current learning-based reconstruction methods rely on supervised training with synthetic data, leading to a sim-to-real gap. This work proposes the first self-supervised learning approach for event-based image reconstruction, leveraging the fundamental optical flow-image brightness relation (photometric constancy) to learn from real unlabeled event data. The framework consists of two independent neural networks, FlowNet and ReconNet, which share information during training to estimate optical flow and reconstruct images, respectively. This method demonstrates robustness across various scenes and generalizes well to different event cameras and resolutions, outperforming prior self-supervised methods and achieving competitive results with state-of-the-art supervised approaches.

Speakers

  • Federico Paredes-Valles — MAVLab, TU Delft
  • Guido C. H. E. de Croon — MAVLab, TU Delft

Talks (1)

  • 00:00:00 — Federico Paredes-Valles: Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancy
    • This paper presents the first self-supervised learning approach for event-based image reconstruction, leveraging the optical flow-image brightness relation to learn from real unlabeled event data without relying on synthetic data or ground truth.

Key Takeaways

  • The paper introduces the first self-supervised learning-based approach for event-based image reconstruction, eliminating the need for ground-truth or synthetic data.
  • The framework leverages the fundamental optical flow - image brightness relation (photometric constancy) inherent in event camera operation.
  • It employs a two-network architecture (FlowNet for optical flow estimation and ReconNet for image reconstruction) that are trained collaboratively.
  • The method demonstrates robustness across diverse scenes and generalizes effectively to event sequences recorded with different event cameras or resolutions.
  • The proposed FireFlowNet offers a lightweight alternative for event-based optical flow estimation, contributing to computational efficiency.

Methods / Models / Datasets Mentioned

  • Rebecq et al. (TPAMI'19)
  • Stoffregen et al. (ECCV'20)
  • Kim et al. (JSSC'08, ECCV'16)
  • Cook et al. (IJCNN'11)
  • Bardow et al. (CVPR'16)
  • Reinbacher et al. (IJCV'18)
  • Scheerlinck et al. (ACCV'18)
  • Zhu et al. (CVPR'19)
  • FlowNet
  • ReconNet
  • EV-FlowNet (Zhu et al., RSS'18)
  • FireFlowNet
  • E2VID (Rebecq et al., TPAMI'19)
  • FireNet (Scheerlinck et al., WACV'20)
  • UZH-FPV Drone Racing Dataset (Delmerico, ICRA'19)

Topics

Event cameras · Image reconstruction · Self-supervised learning · Photometric constancy · Optical flow · Neural networks · Brightness constancy · Sim-to-real gap · Motion blur compensation

Notes

Open for commentary — connections to other work, critiques, follow-up reading.