Light Adaptive Color Image Sensors – LACIS

The technology of spectral filters in thin layer consists of creating a stacked layer of metal/insulator of few nanometers thick. By modulating this relative thickness we can obtain different filters with different spectral transmission properties. Also we can create several such filters on a layer of glass and lay them in a mosaic fashion such each filter covers a single pixel on the sensor (approximately 15 micrometer). Further we lay this glass surface containing filter mosaic on the surface of the CMOS sensor (originally in black and white) such that now each pixel has a color sensitivity.
In general CMOS sensors are passive, i.e. each analog signal captured by the photodiode is directly converted to digital and readout. To enable the “local adaptation” mode we have added calcul blocks (either directly at the pixel level or between the pixel and readout circuit) by modifying the architecture of the classic CMOS sensor circuit which automatically modifies by the digital value readout according to the local luminance level around a pixel.
In the project we manufactured stacked layer of thin deposits allowing us 3, 4 and 9 different colors filters. We distributed these filters in mosaic form both regular and random. The filter characteristics were optimized to give the best rendering of captured images. We also have defined algorithms which allow us to reconstruct color images (demosaicing) from the data being recorded by the sensor, irrespective of the layout of the mosaic or the number of colors in it.

The manufacturing of the glass layer which contains the mosaic of filters wherein each filter is of size 15 x 15 micrometers was successful. We were also able to put these filters on a CMOS sensor with “local adaptation” mode (of 256 x 256 pixels), also manufactured in this project. The sensor was put in use and images captured were successfully demosaiced and rendered with the algorithms developed.
This first prototype is encouraging and validation of our method/process and shows the maturity of this technology. In terms of flexibility gained in color resolution and adaptation to local luminance, the applications foreseen for this sensor relate to all digital imagery, photography, agriculture – food industry, aerial imagery, robotic, machine vision system for autonomous vehicles.

--

Amba, P., Thomas, J. B., & Alleysson, D. (2017). N-LMMSE Demosaicing for Spectral Filter Arrays. Journal of Imaging Science and Technology, 61(4), 40407-1.

The goal of the LACIS's project is to demonstrate the validity of a new approach for color and spectral imaging sensor and camera systems. The demonstration will be given by building one or two prototypes showing the functionality of the novel approach and measuring the improvement compared to the state of the art. The novel approach is based on two principles inspired from the human visual system.

First, human retina consist of a mosaic of cone photoreceptors (LMS) but the mosaic arrangement of cones is changing from individual to individual without impinging on color vision capability of the individual. A generalization of this principle would say that we can build a color sensor with any arrangement of color samples in the color filter array that cover the camera. This flexibility of sensor colorization allows optimizing the sensor for many type of application, particularly those that need multispectral encoding. Our prototypes would be therefore equipped with different color filter array and the performance of these different sensors will be tested.

Second, instead of being perfectly linear with light intensity, the human retina response is non-linear and adaptive. Adaptation to light allows the human visual system to be sensitive to a large range of light value despite the noisy nature of the retina cells. We will implement this property on the prototypes in analog, before the analog to digital converter to prevent from noise amplification due to digitalization. A previous prototype have already been build and tested favorably by two members of the project. A new implementation has been proposed for a patent and will be implemented in the project.

The general goal of the project is to build a demonstrator composed by (1) new filters, either pseudo-random 6x6 RGB, or multispectral based on COLOR SHADE technology, (2) a locally adaptive color CMOS sensor and (3) a motherboard including embedded processing for color or spectral image reconstruction optimized for spatio-spectral information. The demonstrator will be given by a functioning prototype that will deliver images of size 256x256 and showing the properties of the new approach for color or spectral sensor.

The consortium is composed on three entities, two laboratories (LPNC, TIMA) and a company (SILIOS Technologies). The two laboratories have already worked together on a first prototype of light adaptive sensor. TIMA is well recognized in microelectronic and have a long achievement in sensor building. LPNC has developed several models for spatio-spectral representation and demosaicing method as well as high dynamic range and tone mapping inspired from human vision. SILIOS is a SME that develops technology and know-how on micro-optics and more specifically on multispectral filters for spectrometry and multispectral imaging. The project will open new products and skill for the company and new intellectual property for the consortium.