NEuroMorphic hardwarE for Smart vIsion Sensor – NEMESIS

Smart vision circuits are expected to extract high-level information from visual scenes. The main focus of these intelligent retinas is not to provide the highest image quality, but high level information on the image, and be a building block for object recognition and image understanding with many applications in areas such as robotics, security, automotive, home automation, medicine…
Since these applications require a very high bandwidth and a dense and highly efficient data processing technology, leading to use a high level of parallelism, a close integration of the processing capabilities together with the detection components is needed.

Classical approaches use complex algorithms that are difficult to implement using parallel implementations. Such approaches invariably lead to high processing latencies and high power consumption.
In contrast, biological visual systems are very efficient at performing complex recognition tasks using highly parallel “hardware”. For this reason, biologically-inspired spike-based neural networks are promising technical solutions. However, the currently available solutions suffer from the limited interconnection options available with purely 2D integrated circuits.
3D stacking technology seems to be an ideal solution to cope with both bandwidth and interconnect issues encountered by neuromorphic vision chips.

This project aims at exploring the potential of biologically-inspired spike-based image processing supported by the realization of massively parallel yet scalable hardware thanks to 3D stacking of integrated circuits.

The goal of the project is to develop and build a smart vision sensor relying on neuromorphic hardware to perform image and/or video processing. Hardware design will follow a unified and efficient approach: both imaging and processing layers will be based on spiking elements to reach a high degree of power efficiency, parallelism, and robustness. Pixels within the imaging layer will asynchronously send spike-events according to local image features such as light intensity, contrast and potentially other simple processing operations such as convolutions. The processing layer will also be composed of spiking neurons allowing biologically inspired processing. Specific sets of synaptic weights will be developed to allow the implementation of a range of high-level processing tasks. 3D stacking will be used to connect the different layers (imaging and processing) together in order to maximize parallelism without decreasing the fill factor of the retina.
This hardware will allow to address real-world scenarios that combine image pre-processing together with high-level real-world tasks. The following application example will be used as a benchmark to validate the approach: localization, counting, and motion monitoring of people in a crowd. The project also aims at showing that the spike-based approach is more energy-efficient and robust than classical processing approaches.