Hardware and Algorithm Co-design with Event Sensors

Event: ONR Program Review November 5, 2015 · Duration: 26 min · ▶ Watch on YouTube

Abstract

This presentation explores the paradigm of computational imaging, emphasizing joint hardware and software co-design, particularly with event sensors. The talk covers various applications across different physical scales, from nanoscopy to astronomy, with a focus on photography-scale problems like 3D scanning and high-speed video. Key projects include the MC3D camera system for robust 3D scanning in bright ambient light conditions and event-driven techniques for high-speed video frame synthesis and guided event filtering. The presentation concludes by discussing future directions in bio-inspired computational imaging using spiking and hybrid neural network models.

Speakers

  • Oliver Cossairt — Associate Professor, ECE/CS Departments, Northwestern University

Talks (1)

  • 00:00:00 — Oliver Cossairt: Hardware and Algorithm Co-design with Event Sensors
    • Oliver Cossairt introduces his work on hardware and algorithm co-design with event sensors, highlighting projects from his Computational Photography Lab and the Image and Video Processing Lab (IVPL) at Northwestern University.

Key Takeaways

  • Computational imaging leverages joint hardware-software design to address challenges across various physical scales and applications, including 3D imaging and high-speed video.
  • Event sensors offer significant advantages over traditional intensity cameras due to their sparse data streams, enabling faster motion sensing and robustness to high ambient illumination.
  • The MC3D camera system demonstrates effective 3D scanning in challenging bright light conditions by combining a laser projector with an event sensor.
  • Event-driven video frame synthesis and Guided Event Filtering (GEF) can enhance the quality of high-speed video, enabling applications like motion deblurring, HDR imaging, and improved corner detection and tracking.
  • Future directions involve bio-inspired computational imaging and hybrid neural network models (combining analog and spiking networks) to achieve better end-to-end hardware-software performance with lower power consumption.

Methods / Models / Datasets Mentioned

  • Remote Fourier Ptychography Imaging
  • Synthetic Wavelength Holography
  • MC3D
  • Pulsed LIDAR
  • Time-of-flight camera
  • Optical interferometry
  • Compressed Sensing Video
  • Snapshot 3D Microscopy
  • Multifocal Microscopy (MFM)
  • Event-driven video frame synthesis
  • SepConv
  • DMR
  • Residual learning
  • Residual Nets
  • Guided Event Filtering (GEF)
  • ANN
  • SNN
  • DNN
  • MNIST Classification

Topics

Computational Imaging · Hardware-Software Co-design · Event Sensors · 3D Scanning · High-Speed Video · Bio-inspired Imaging · Spiking Neural Networks · Motion Deblurring · HDR Imaging · Corner Detection & Tracking

Notes

Open for commentary — connections to other work, critiques, follow-up reading.