Bringing computation in the focal plane: Sensors with in-pixel processing to enable novel algorithms and systems

Event: CVPR 2019 Workshop on Event Based Vision and Smart Cameras · Duration: 15 min · ▶ Watch on YouTube

Abstract

The presentation discusses the potential of in-pixel processing in modern sensors to revolutionize computer vision. It highlights the concept of a ‘visual code’ generated by sensors and decoded by vision algorithms, emphasizing that different visual quantities (like light intensity, temporal derivative, optic flow, polarization, depth, etc.) can be sensed or inferred. By integrating more intelligence directly into the pixel, it’s possible to mitigate sensor-processor bottlenecks and enable adaptive sensing strategies, such as HDR imaging, coded-exposure imaging, and dynamic vision sensing. The ultimate goal is to holistically co-optimize optics, sensors, and algorithms to derive desired visual quantities more efficiently and effectively.

Speakers

  • Julien N.P. Martel — Stanford Computational Imaging Lab

Talks (1)

  • 00:00:00 — Julien N.P. Martel: Bringing computation in the focal plane: Sensors with in-pixel processing to enable novel algorithms and systems
    • This talk explores how in-pixel computation in focal plane sensors can be leveraged to enable novel algorithms and systems, moving beyond simple low-level vision tasks to co-optimize hardware and algorithms for extracting complex visual quantities.

Key Takeaways

  • Modern sensors with in-pixel processing offer significant opportunities to sense visual quantities beyond simple light intensity, directly providing correlates of temporal derivatives, optic flow, or polarization.
  • Bringing more ‘intelligence’ (i.e., transistors) into the pixel allows for adaptive sensing strategies, such as controlling integration time for HDR or coded-exposure imaging, and even performing local computations like tracking or solving Poisson equations.
  • The traditional vision pipeline suffers from sensor-processor bottlenecks; in-pixel intelligence can mitigate this by processing data closer to the source and transmitting more compact, task-relevant ‘visual codes’.
  • A holistic approach to designing vision systems, involving the co-optimization of optics, sensors (including their in-pixel intelligence), and vision algorithms, is crucial for overcoming current limitations and achieving advanced capabilities like super-resolution or monocular depth imaging.

Methods / Models / Datasets Mentioned

  • APS cameras
  • Event-based sensors
  • Optical mouse
  • Polarization sensors
  • SCAMP-5 pixel
  • Joint HDR imaging and Tone Mapping
  • Coded-exposure imaging on the focal plane
  • Dynamic Vision Sensing
  • Depth from focus
  • High-speed tracker
  • Pixel solving a Poisson equation
  • Gradient based imager
  • End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging
  • Deep Optics for Monocular Depth Estimation and 3D Object Detection

Topics

In-pixel processing · Focal plane computation · Visual quantities of interest · Sensor-processor bottleneck · Co-optimization of hardware and algorithms · Adaptive sensing strategies · Event-based sensors · Computational imaging

Notes

Open for commentary — connections to other work, critiques, follow-up reading.