Neuromorphic vision for humanoid robots

Event: CVPR 2025 Workshop on X · Duration: 46 min · ▶ Watch on YouTube

Abstract

This presentation explores the application of neuromorphic vision for humanoid robots, specifically using the iCub platform. It highlights the advantages of event-driven sensors for operating in dynamic and unconstrained environments due to their efficient information encoding, low latency, and low-power computation. The talk details various vision functionalities developed, including bio-inspired attention models, high-throughput online corner detection, and trajectory prediction for dynamic objects. It concludes by outlining future directions, emphasizing the integration of multi-modal neuromorphic sensing (vision, audio, touch) and asynchronous communication for advanced humanoid robotics.

Speakers

  • Chiara Bartolozzi — Italian Institute of Technology (IIT)

Talks (15)

  • 00:00:00 — Chiara Bartolozzi: Neuromorphic vision for humanoid robots
    • Introduction to neuromorphic vision for humanoid robots, focusing on the iCub platform and the advantages of event-driven sensors for dynamic, unconstrained environments.
  • 00:24:00 — Chiara Bartolozzi: The scenario…
    • Imagining future robots interacting autonomously in dynamic environments, leveraging event-driven vision for efficient information encoding, low latency, and low-power computation.
  • 01:21:00 — Chiara Bartolozzi: Neuromorphic iCub
    • Description of the neuromorphic iCub platform, featuring binocular event-driven vision (ATIS Gen1/Gen3) integrated into the eyes, and various computational platforms (FPGA, CPU, Neuromorphic HW like Spinnaker) connected via a modular YARP-based system.
  • 03:21:00 — Chiara Bartolozzi: Event-driven vision on the iCub
    • Overview of key event-driven vision functionalities developed for the iCub, including attention and selection, feature detection, tracking moving objects, and prediction for dynamic environments.
  • 06:48:00 — Chiara Bartolozzi: Attention: 3D Proto-object Saliency Map
    • Detailed explanation of a bio-inspired attention model based on stimulus-driven saliency computation, incorporating proto-object detection and depth information from RGB-D images (Hu et al. 2016 model).
  • 11:28:00 — Chiara Bartolozzi: Attention: ED Proto-object Saliency Map
    • Adaptation of the saliency map model for event-driven cameras, leveraging edge detection inherent in ED sensors and using Von Mises filters for grouping, achieving scale invariance and high speed (Iacono et al. 2019).
  • 13:35:00 — Chiara Bartolozzi: Attention: ED Proto-object Saliency Map with disparity
    • Extension of the event-driven saliency map to include disparity information from binocular ED cameras, using a cooperative stereo matching model (Mohsen & Conradt 2016) to select closer objects and static elements when the robot’s eyes move.
  • 18:26:00 — Chiara Bartolozzi: Feature detection – online corner detection
    • Introduction to online corner detection as a crucial task for robotics, highlighting the need for accuracy and real-time performance with event cameras.
  • 19:08:00 — Chiara Bartolozzi: Look-up event Harris corner detection (luvHarris)
    • Presentation of the luvHarris algorithm, which decouples data representation (Threshold Ordinal Surface - TOS) from the Harris algorithm, allowing for high event-throughput and low latency by querying a pre-calculated Look-up Table.
  • 24:51:00 — Chiara Bartolozzi: Million Events / Second
    • Performance comparison of luvHarris against other event-based corner detectors (eHarris, ARC, FAST), demonstrating its superior event-throughput and minimal delay even at very high event rates.
  • 26:23:00 — Chiara Bartolozzi: Prediction – Bouncing Ball
    • Investigation into using event-driven vision for predicting the trajectory of dynamic objects, specifically a bouncing ball, to enable proactive robot interaction.

Key Takeaways

  • Event-driven vision sensors offer significant advantages for humanoid robots in dynamic environments, including efficient information encoding, low latency, and low-power computation.
  • Bio-inspired attention models and specialized event-driven algorithms like luvHarris enable robust and high-throughput perception tasks such as saliency detection and online corner detection.
  • Asynchronous (space-based) sampling in event-driven systems can significantly improve the accuracy and efficiency of trajectory prediction for dynamic objects compared to traditional synchronous (time-based) methods.
  • The neuromorphic iCub platform serves as a versatile research tool for integrating and testing novel event-driven technologies across various sensory modalities like vision, audio, and touch.
  • Future advancements in humanoid robotics will likely involve the integration of multiple neuromorphic sensory modalities and asynchronous communication protocols to achieve more human-like perception and interaction capabilities.

Methods / Models / Datasets Mentioned

  • ATIS Gen1
  • ATIS Gen3
  • YARP
  • Spinnaker
  • Hu et al. 2016
  • Mohsen & Conradt 2016
  • Iacono et al. 2019
  • eHarris
  • luvHarris
  • Vasco et al. 2016
  • Alzugaray & Chli 2018
  • Mueggler et al. 2017
  • Glover et al. 2021
  • Threshold Ordinal Surface (TOS)
  • Sobel filter
  • Harris score
  • OpenCV
  • ARC
  • FAST
  • LSTM Encoder-Decoder
  • Stateful LSTM
  • Panda robot

Topics

Neuromorphic Vision · Humanoid Robots · Event-Driven Sensors · iCub Platform · Saliency Maps · Corner Detection · Trajectory Prediction · Asynchronous Sampling · Robotics Perception · Low Latency Computing

Notes

Open for commentary — connections to other work, critiques, follow-up reading.