Event-based vision and processing for tiny drones

Event: CVPR 2025 Workshop on X · Duration: 26 min · ▶ Watch on YouTube

Abstract

This presentation addresses the critical challenge of achieving autonomous flight for tiny, lightweight, and agile drones, which are severely limited by size, weight, and power (SWaP) constraints. Inspired by the remarkable capabilities of fruit flies, the talk explores how event-based vision and neuromorphic computing, particularly spiking neural networks (SNNs), can enable complex behaviors with minimal computational resources. A case study on optical flow-based landing demonstrates the successful implementation of SNNs on Intel’s Loihi neuromorphic chip for robust and smooth drone control. The research highlights the potential for fully neuromorphic pipelines to unlock advanced autonomy for micro air vehicles.

Speakers

  • Guido de Croon — Full Professor, MAVLab, Faculty of Aerospace Engineering, TU Delft

Talks (6)

  • 00:00:00 — Guido de Croon: Event-based vision and processing for tiny drones
    • Introduction to tiny, lightweight, agile drones and the challenges of achieving autonomy due to size, weight, and power (SWaP) constraints, contrasting with traditional SLAM memory requirements.
  • 00:32:39 — Guido de Croon: How to make tiny drones autonomous?
    • Drawing inspiration from nature, particularly fruit flies, to develop simple, linked behaviors for complex tasks, demonstrated with the DelFly Explorer and a swarm of Crazyflie drones.
  • 07:44:00 — Guido de Croon: Neuromorphic sensing and computing
    • Introduction to event-based cameras and spiking neural networks (SNNs) as a promising, energy-efficient approach for processing high-dimensional sensory data in tiny drones, despite training challenges.
  • 09:45:00 — Guido de Croon: Case study: optical flow landing
    • Demonstration of optical flow-based vertical landing inspired by honeybees, explaining the concept of constant optical flow divergence and the limitations of a ‘naïve’ control approach.
  • 13:22:00 — Guido de Croon: Our approach
    • Transitioning from traditional vision pipelines to event-based vision and neuromorphic processing, including unsupervised learning of optical flow with SNNs and implementing control on Intel’s Loihi neuromorphic chip.
  • 24:06:00 — Guido de Croon: Conclusion
    • Summary of the promise of neuromorphic sensing and processing for insect-inspired autonomous flight in tiny drones, outlining future directions for fully neuromorphic pipelines and complex task learning.

Key Takeaways

  • Tiny drones face significant SWaP constraints, making traditional SLAM and deep learning approaches impractical for onboard autonomy.
  • Nature, particularly insects like fruit flies, offers inspiration for achieving complex autonomous behaviors with highly efficient, low-power neural architectures.
  • Event-based cameras and spiking neural networks (SNNs) are promising technologies for enabling autonomous flight in tiny drones due to their speed, sparsity, and energy efficiency.
  • Evolving SNNs in simulation with robust training methods (like randomized unpredictable elements) and transferring them to neuromorphic hardware (like Intel Loihi) can successfully bridge the reality gap for complex control tasks.
  • Future work aims for fully neuromorphic pipelines, improved SNN learning mechanisms, and tackling increasingly complex autonomous tasks beyond basic optical flow control.

Methods / Models / Datasets Mentioned

  • DelFly Nimble
  • STM32F4
  • NVIDIA Jetson TX2
  • SLAM
  • DelFly Explorer
  • Crazyflie drones
  • Swarm bug algorithm
  • Event-based cameras (DVS)
  • Lucas-Kanade optical flow
  • RANSAC
  • Spiking Neural Networks (SNNs)
  • STDP (Spike-Timing-Dependent Plasticity)
  • Intel Loihi chip
  • Kapoho Bay
  • Parrot Bebop 2
  • Paparazzi autopilot
  • Optitrack
  • ROS
  • Up2 Board
  • NxSDK

Topics

Tiny drones · Autonomous flight · Event-based vision · Neuromorphic computing · Spiking Neural Networks (SNNs) · Optical flow landing · Insect-inspired robotics · SWaP constraints · Simulation-to-reality transfer · Unsupervised learning

Notes

Open for commentary — connections to other work, critiques, follow-up reading.