Image Reconstruction from Neuromorphic Event Cameras using Laplacian- Prediction and Poisson Integration with Spiking and Artificial Neural Networks

Event: CVPR 2021 Workshop on Event-based Vision · Duration: 3 min · ▶ Watch on YouTube

Abstract

Neuromorphic event cameras offer significant advantages over traditional frame-based cameras, including high temporal resolution, high dynamic range, and absence of motion blur. However, reconstructing frame-like images from the sparse event stream remains a fundamental challenge. This paper introduces a novel method that first predicts the Laplacian of the image using a simple convolutional neural network, which can be easily converted into a spiking neural network (SNN). Subsequently, a feed-forward SNN performs Poisson integration to reconstruct the full image from its predicted Laplacian. The proposed architecture is highly efficient, achieving good image reconstruction results with a minuscule parameter space, making it suitable for neuromorphic hardware implementation.

Speakers

  • Hadar Cohen Duwek — The Open University of Israel, Neuro-Biomorphic Engineering Lab
  • Albert Shalomov — The Open University of Israel, Neuro-Biomorphic Engineering Lab
  • Elishai Ezra Tsur — The Open University of Israel, Neuro-Biomorphic Engineering Lab

Talks (1)

  • 00:00 — Hadar Cohen Duwek, Elishai Ezra Tsur: Image Reconstruction from Neuromorphic Event Cameras using Laplacian- Prediction and Poisson Integration with Spiking and Artificial Neural Networks
    • This work presents a novel approach for image reconstruction from neuromorphic event cameras using a small convolutional neural network for Laplacian prediction, convertible to a spiking neural network, followed by a Poisson integration SNN.

Key Takeaways

  • Neuromorphic event cameras provide high temporal resolution and dynamic range, but image reconstruction from their event streams is a complex task.
  • A two-stage approach is proposed: Laplacian prediction using a small convolutional neural network (CNN) that can be converted to a spiking neural network (SNN), followed by Poisson integration using a feed-forward SNN.
  • The Poisson integration SNN is designed to solve the Poisson equation (a reduced form of the diffusion equation) and does not require trainable parameters.
  • The method achieves good image reconstruction quality with very small networks, utilizing less than 300 parameters for SNNs and under 100 parameters for the non-spiking CNN.
  • The Mish activation function is demonstrated to outperform the classic ReLU activation for this image reconstruction task.

Methods / Models / Datasets Mentioned

  • Convolutional Neural Network
  • Spiking Neural Network
  • Laplacian's Prediction
  • Poisson Integration
  • Poisson Equation
  • Diffusion Equation
  • Mish activation
  • ReLU activation
  • N-MNIST dataset
  • Caltech101 dataset

Topics

Neuromorphic Event Cameras · Image Reconstruction · Spiking Neural Networks (SNN) · Artificial Neural Networks (ANN) · Laplacian Prediction · Poisson Integration · Event-based Vision · Low-parameter models · Mish activation

Notes

Open for commentary — connections to other work, critiques, follow-up reading.