Differentiable Event Stream Simulator for Non-Rigid 3D Tracking

Event: CVPR 2021 Workshop on Event-Based Vision · Duration: 3 min · ▶ Watch on YouTube

Abstract

This work addresses the challenge of tracking non-rigid 3D objects using event cameras, which offer advantages like micro-second resolution, low latency, and high dynamic range compared to conventional cameras that suffer from motion blur and redundancy. The proposed method utilizes an analysis-by-synthesis approach, combining computer graphics rendering with an event generation model. A differentiable event stream simulator is developed to generate synthetic event frames, which are then compared against real event data from a DAVIS 240C camera within an optimization framework to estimate optimal object parameters.

Speakers

  • Jalees Nehvi — Saarland University
  • Vladislav Golyanik — Max Planck Institute for Informatics, SIC
  • Franziska Mueller — Max Planck Institute for Informatics, SIC
  • Hans-Peter Seidel — Max Planck Institute for Informatics, SIC
  • Mohamed Elgharib — Google Inc.
  • Christian Theobalt — Max Planck Institute for Informatics, SIC

Talks (1)

  • 00:00:00 — Jalees Nehvi: Differentiable Event Stream Simulator for Non-Rigid 3D Tracking
    • This paper introduces a differentiable event stream simulator for tracking non-rigid 3D objects using event cameras, leveraging an analysis-by-synthesis optimization framework.

Key Takeaways

  • Event cameras are highly advantageous for tracking fast-moving and deforming objects due to their high temporal resolution and lack of motion blur.
  • A differentiable event stream simulator can synthesize events from 3D models, enabling an analysis-by-synthesis approach for non-rigid tracking.
  • The proposed method uses a differentiable thresholding function to approximate the event generation model, making the entire pipeline end-to-end differentiable.
  • Quantitative comparisons on synthetic data demonstrate that the method outperforms existing RGB-based approaches in 3D error and standard deviation.
  • The framework successfully tracks non-rigid objects like hands and deformable paper both in synthetic and real-world scenarios.

Methods / Models / Datasets Mentioned

  • HandGraphCNN
  • DDD
  • Tien Ngo et al.
  • IsMo-GAN

Topics

Non-rigid 3D tracking · Event cameras · Differentiable rendering · Analysis-by-synthesis · Event stream simulation · Parametric models · Optimization framework · High dynamic range

Notes

Open for commentary — connections to other work, critiques, follow-up reading.