Neuromorphic computing hardware and event-based vision: a perfect match?

Event: CVPR Workshop “Event-based vision” · Duration: 40 min · ▶ Watch on YouTube

Abstract

This talk explores the synergy between neuromorphic computing hardware and event-based vision. It delves into the core elements of neuromorphic computing, such as spiking neurons, flexible network topologies, and local synaptic plasticity, and introduces Intel’s Loihi research chip. The presentation discusses various approaches to implementing deep learning and other algorithms on neuromorphic hardware for event-based vision tasks, highlighting potential benefits in performance, latency, and power efficiency. It also showcases applications in gesture recognition, multimodal sensing, continual learning for robotics, and ultra-fast vision-based control, emphasizing the importance of co-design between algorithms and hardware.

Speakers

  • Yulia Sandamirskaya — Neuromorphic Computing Lab, Intel

Talks (1)

  • 00:00:03 — Yulia Sandamirskaya: Neuromorphic computing hardware and event-based vision: a perfect match?
    • An exploration of how neuromorphic computing hardware can enhance event-based vision by leveraging its unique temporal processing, flexible network topologies, and on-chip learning capabilities for real-world applications.

Key Takeaways

  • Neuromorphic hardware, particularly Intel’s Loihi, offers significant advantages in energy efficiency and latency for event-based vision tasks due to its inherent temporal processing, event-driven communication, and on-chip learning capabilities.
  • Different approaches exist for mapping and training neural networks on neuromorphic hardware, ranging from converting pre-trained ANNs to directly training SNNs with event-based data or enabling online continual learning.
  • Novel SNN algorithms and biologically inspired architectures, such as those for kNN search, SLAM, and continual learning, demonstrate superior performance on neuromorphic platforms compared to conventional hardware.
  • The field faces challenges in developing NmC-compatible vision algorithms, establishing relevant benchmarks, and creating robust software tools, emphasizing the need for strong community collaboration and hardware-algorithm co-design.
  • Neuromorphic computing hardware is not static; its development is an iterative process driven by exploring what works best for specific tasks and continuously improving the underlying architecture.

Methods / Models / Datasets Mentioned

  • SLAYER networks
  • LSNN
  • SNN BPTT
  • NengoDL
  • SNNTB
  • kNN search
  • Sparse coding
  • NLCM
  • RatSLAM model
  • Hough Transform
  • PID controller
  • Backpropagation
  • SGD
  • Passive-Aggressive method
  • Naive Bayes Multinomial Distribution
  • HNSW
  • PCA
  • TrueNorth architecture
  • DVS Gesture dataset
  • NMNIST dataset
  • TIDIGITS dataset
  • CIFAR-10 dataset
  • DAVIS240C sensor
  • DAVIS5128 sensor
  • iCub robot simulator

Topics

Neuromorphic computing · Event-based vision · Spiking Neural Networks (SNNs) · Loihi chip · On-chip learning · Continual learning · Robotics · SLAM · Vision-based control · Benchmarking · Hardware co-design

Notes

Open for commentary — connections to other work, critiques, follow-up reading.