High Speed Perception-Action Systems with Event-Based Cameras

Event: CVPR 2021 Workshop on Event-Based Vision · Duration: 22 min · ▶ Watch on YouTube

Abstract

The presentation from Samsung Research at CVPR 2021’s Event-Based Vision Workshop introduces the Samsung AI Center in New York and its work on high-speed perception-action systems using event-based cameras. It covers three main research areas: efficient event-based classification for always-on applications, near-chip low-bandwidth event-based classification for IoT systems, and fast motion understanding for mobile robotics. The talk details novel methods for noise reduction, event-stream compression, and spatiotemporal feature extraction using binary and convolutional neural networks, demonstrating their application in pedestrian detection and rapid obstacle avoidance. An ongoing project on a perception-action testbed explores system latencies and real-time dart dodging.

Speakers

  • Volkan Isler — Samsung AI Center New York
  • Anthony Bisulco — Samsung AI Center New York
  • Daewon Lee — Samsung AI Center New York
  • Daniel D. Lee — Samsung AI Center New York
  • Fernando Cladera Ojeda — Samsung AI Center New York
  • Daniel Kepple — Samsung AI Center New York

Talks (5)

  • 00:00:00 — Volkan Isler: Introduction to Samsung AI Center New York and Robotics Research
    • An introduction to Samsung’s AI Center in New York, its focus on robotics and real-time perception, and examples of their work in manipulation in clutter, grasping, and multimodal perception for collision detection and tactile control.
  • 00:02:56 — Anthony Bisulco: On-Device Event Filtering with Binary Neural Networks for Pedestrian Detection Using Neuromorphic Vision Sensors
    • Presents a low-complexity architecture for always-on, energy-constrained pedestrian detection using event-based cameras, combining a Point Process Filter (PPF) with a Binary Neural Network (BNN) to reduce noise and increase detection accuracy.
  • 00:07:33 — Anthony Bisulco: Near-chip Dynamic Vision Filtering for Low-Bandwidth Pedestrian Detection
    • Addresses the high bandwidth issue of DVS for IoT applications by developing on-chip event-stream compression algorithms coupled with edge compute classification using Binary Neural Networks, achieving significant bandwidth reduction and F1 score increase.
  • 00:10:31 — Anthony Bisulco: Fast Motion Understanding with Spatiotemporal Neural Networks and Dynamic Vision Sensors
    • Focuses on high-speed motion understanding for mobile robot systems in dynamic environments, proposing a method that uses event-based sensors to encode event history with exponential filters, extracts spatiotemporal features using a CNN, and estimates time to collision and impact location for fast-moving objects.
  • 00:17:00 — Anthony Bisulco: Ongoing Work: High Speed Perception-Action Systems with Event-Based Cameras
    • Describes a perception-action testbed designed to evaluate the full system latency for fast obstacle avoidance using event-based cameras, including analysis of observability, communication, serial, and action latencies, and demonstrating real-time dart dodging.

Key Takeaways

  • Event-based cameras offer significant advantages for high-speed, low-latency perception-action systems in robotics and IoT.
  • Novel architectures combining event filtering (like PPF) and binary neural networks can achieve high accuracy in pedestrian detection with low computational complexity and memory footprint, suitable for always-on and embedded applications.
  • On-chip event-stream compression techniques can drastically reduce bandwidth requirements for DVS in IoT systems while improving detection performance.
  • Event-based sensors can effectively estimate impact location and time-to-collision for fast-moving objects, enabling rapid avoidance maneuvers in mobile robotics.

Methods / Models / Datasets Mentioned

  • Point Process Filter (PPF)
  • Binary Neural Network (BNN)
  • Convolutional Neural Network (CNN)
  • Coincidence Detection
  • Aggregation
  • Subsampling
  • Huffman Code
  • Affine Warp
  • Motion Capture System
  • Accelerometer
  • Opto-coupler
  • Arduino
  • libusb

Topics

Event-based cameras · Dynamic Vision Sensors (DVS) · High-speed perception · Robotics · Real-time perception · Low-latency systems · Pedestrian detection · Motion understanding · Obstacle avoidance · IoT applications

Notes

Open for commentary — connections to other work, critiques, follow-up reading.