Fusing Frame and Event data for High Dynamic Range Video

Event: CVPR 2021 Workshop on Event-based Vision · Duration: 33 min · ▶ Watch on YouTube

Abstract

Event cameras offer advantages like asynchronous events, high temporal density, no image blur, and high dynamic range, while frame cameras provide synchronous, spatially dense images with adjustable exposure and absolute intensity. This work proposes fusing these complementary sensor modalities to achieve images that are both spatially and temporally dense, possess high dynamic range in absolute intensity, and are free from motion blur in both static and highly dynamic scenes. The core of the approach is an Asynchronous Kalman Filter (AKF) that adaptively tunes its gain based on the reliability of frame data, effectively combining low-frequency information from frames and high-frequency details from events. The algorithm is shallow, avoiding deep learning, and demonstrates robust performance in HDR reconstruction and temporal interpolation, particularly in scenarios where event-only learning-based methods may fail.

Speakers

  • Robert Mahony — The Australian National University (ANU)

Talks (1)

  • 00:00:00 — Robert Mahony: Fusing Frame and Event data for High Dynamic Range Video
    • This talk presents an Asynchronous Kalman Filter (AKF) for fusing frame and event camera data to reconstruct high dynamic range (HDR) video with improved temporal and spatial density, reduced blur, and robust performance in challenging lighting conditions.

Key Takeaways

  • Fusing event and frame data can overcome individual sensor limitations, yielding spatially/temporally dense, HDR, and blur-free video.
  • Modeling event data as continuous-time Dirac delta functions allows the use of linear systems theory for image reconstruction and filtering.
  • A complementary filter effectively combines low-frequency information from frame data and high-frequency information from event data.
  • An Asynchronous Kalman Filter (AKF) adaptively tunes the fusion gain based on noise models for both sensor types, providing robust performance in varying lighting conditions, including extreme under/over-exposure.
  • The AKF is a shallow algorithm that solves continuous-time ODEs exactly between events, making it computationally efficient and asynchronous, with performance expected to improve as event camera sensor quality advances.

Methods / Models / Datasets Mentioned

  • E2VID
  • ECNN
  • DSEC dataset
  • Shapes dataset
  • Event-based Double Integral (EDI)
  • Direct integration
  • High pass filter
  • Complementary filter
  • Riccati Equation Solver
  • Gaussian blur
  • Laplace operator
  • Sobel operator
  • CRF (Camera Response Function)

Topics

Event cameras · Frame cameras · Sensor fusion · High Dynamic Range (HDR) · Asynchronous Kalman Filter (AKF) · Image reconstruction · Temporal interpolation · Motion blur · Noise modeling · Linear systems theory

Notes

Open for commentary — connections to other work, critiques, follow-up reading.