Event-based Feature Tracking and Visual Inertial Odometry

Event: ICRA 2017 Workshop on Event-based Vision · Duration: 24 min · ▶ Watch on YouTube

Abstract

This presentation introduces an event-based visual-inertial odometry (EVIO) system that leverages the unique properties of event cameras for robust feature tracking and camera pose estimation. The system addresses challenges faced by traditional frame-based cameras in scenarios with very low lighting, high-speed motion, and high dynamic range. By defining features as sets of events with consistent optical flow and employing an expectation-maximization (EM) algorithm for data association and flow estimation, the EVIO system can accurately track features and estimate camera motion. The pipeline integrates IMU observations using a Multi-State Constraint Kalman Filter (MSCKF) and incorporates RANSAC for outlier rejection, demonstrating superior performance in various challenging environments.

Speakers

  • Kostas Daniilidis — University of Pennsylvania
  • Alex Zhu — University of Pennsylvania
  • Nikolay Atanasov — University of Pennsylvania

Talks (24)

  • 0:00 — Kostas Daniilidis: Event-based Feature Tracking and Visual Inertial Odometry
    • Introduction to the presentation and the research topic.
  • 0:28 — Kostas Daniilidis: Night Scene, Very Low Lighting
    • Demonstration of event-based feature tracking in extremely low light conditions, where traditional cameras fail due to motion blur and darkness.
  • 1:29 — Kostas Daniilidis: Truck Passing 3m from the Camera at 60 miles/hr, 0.06x Realtime
    • Comparison showing event-based cameras’ superior performance in high-speed scenarios with extreme optical flow, outperforming high-frame-rate traditional cameras.
  • 2:49 — Kostas Daniilidis: Frame-based Cameras / Event-based Cameras
    • Explanation of the fundamental differences between frame-based and event-based cameras, emphasizing the asynchronous, microsecond-resolution output of event cameras.
  • 3:48 — Kostas Daniilidis: What is a feature in classic vision?
    • Discussion on how features are defined in classic vision using spatial neighborhoods and motion, and the limitations of this approach for event-based systems.
  • 4:48 — Kostas Daniilidis: Speed is dealt with multiple scales
    • Overview of traditional multi-scale optical flow methods and the challenge of defining features in event-based cameras due to the absence of frames.
  • 6:05 — Kostas Daniilidis: A feature is a set of 2D events induced by the same point in 3D.
    • Definition of an event-based feature as a cluster of noisy 2D events originating from a single 3D point, introducing the data association problem.
  • 8:13 — Kostas Daniilidis: A feature is a set of events with same flow
    • Features are identified by events exhibiting consistent optical flow, and the challenge lies in defining this association without prior knowledge.
  • 9:39 — Kostas Daniilidis: But we do not know the association so we will take the expectation
    • An expectation-maximization (EM) approach is used to estimate optical flow by handling the unknown data association between events and features.
  • 10:05 — Kostas Daniilidis: Optical Flow Estimation
    • Presentation of the optical flow estimation formula, which minimizes the difference between warped events to align them in the spatiotemporal domain.
  • 11:20 — Kostas Daniilidis: E-Step / M-Step (linear least squares)
    • Detailed explanation of the E-Step (calculating event-to-feature probabilities) and M-Step (solving linear least squares for optical flow) of the EM algorithm.
  • 12:09 — Kostas Daniilidis: How long temporal window?
    • Discussion on the impact of different fixed temporal windows on event visualization and the need for adaptive window selection.
  • 12:44 — Kostas Daniilidis: How do we choose the right temporal window?
    • Explanation of a dynamic temporal window selection method based on optical flow magnitude, grouping events with similar flow.
  • 14:29 — Kostas Daniilidis: Over longer time: Monitor quality of feature with an affine motion model!
    • Introduction of feature quality monitoring over time using an affine motion model and drift correction for stabilization.
  • 16:31 — Kostas Daniilidis: Results: KLT Comparison
    • Presentation of results and comparison with KLT, using KLT as ground truth for slow sequences.
  • 17:46 — Kostas Daniilidis: Sparsify and deal with aperture problem: FAST corner selection in the aggregation of warped images
    • Method for sparsifying features and addressing the aperture problem using FAST corner selection on aggregated warped images.
  • 18:12 — Kostas Daniilidis: Visual Inertial Odometry
    • Explanation of the VIO approach, combining event-based feature tracks with IMU data and MSCKF, and enforcing 3D rotation in 2D tracking.
  • 19:17 — Kostas Daniilidis: Outlier Rejection
    • Description of the outlier rejection strategy using two RANSAC steps (pure translation and triangulation) to handle EKF’s susceptibility to outliers.
  • 19:51 — Kostas Daniilidis: EVIO Summary
    • Overview of the complete EVIO pipeline, detailing its various components and their interactions.
  • 20:06 — Kostas Daniilidis: Results: General Scene
    • Demonstration of the system’s performance in a general scene, showing accurate camera trajectory and 3D feature tracking.
  • 21:00 — Kostas Daniilidis: Results: HDR Scene
    • Results showcasing the system’s robust performance in challenging High Dynamic Range (HDR) lighting conditions.
  • 21:39 — Kostas Daniilidis: Results: Motion Independent of Camera
    • Demonstration of the system’s ability to track features effectively in scenes with motion independent of the camera.
  • 21:48 — Kostas Daniilidis: Live Demo
    • A live demonstration illustrating the system’s real-time capability in reconstructing 3D paths from noisy events generated by a rotating striped cylinder.
  • 23:16 — Kostas Daniilidis: The future of robot vision is event-based!
    • Concluding remarks emphasizing the transformative potential of event-based vision for future robotics applications.

Key Takeaways

  • Event-based cameras offer significant advantages over traditional frame-based cameras in challenging scenarios like low light, high speed, and high dynamic range.
  • Features in event-based systems are defined by clusters of events exhibiting consistent optical flow, requiring novel data association techniques.
  • An EM-based approach effectively solves the data association problem and estimates optical flow in event streams.
  • The EVIO system integrates event-based feature tracking with IMU observations and robust outlier rejection for accurate camera pose estimation.
  • The future of robot vision is strongly tied to the advancements and capabilities of event-based sensing.

Methods / Models / Datasets Mentioned

  • KLT
  • Bayesian Multi-Scale Differential Optical Flow
  • MSCKF
  • RANSAC
  • FAST corner selection
  • EM algorithm

Topics

Event-based cameras · Visual-inertial odometry (VIO) · Feature tracking · Optical flow estimation · Expectation-maximization (EM) algorithm · Data association · Outlier rejection · High-speed motion · Low-light conditions · High dynamic range (HDR)

Notes

Open for commentary — connections to other work, critiques, follow-up reading.