Super-resolution for visible camera system – SURECAVI

Following Jérémy Anger's thesis work (funded by DGA), the team of the Centre Borelli (CB) ENS Paris Saclay has developed a fast super-resolution pipeline to process series of push-frame satellite images from the PLANET SKYSAT. satellites. This pipeline, which is currently being industrialized by KAYRROS, fully exploits the GPU expertise at CB, enabling the efficient production of images with a resolution higher than those provided by PLANET. This illustrates the maturity of super-resolution and image and video restoration techniques, as well as the importance of their deployment in commercial and defense applications.

LERITY is a player in the field of optronics, specializing in very high definition vision systems in the visible spectrum, operating in particular in degraded visibility conditions. LERITY pursues a dynamic research and innovation policy, anticipating breakthrough developments in the field, namely: advanced real-time signal processing algorithms, constrained optronic design, and integration of image sensors and laser illuminators.
Among the products designed by LERITY, the improvement brought by the new algorithms concerns systems integrating the capacity of remote resources such as CATEYE System and CATEYE System XLR. The implementation of these algorithms is not planned yet for embedded architectures because of their computational constraints.

The goal of the project is to significantly improve the resolution and image quality of these cameras through software and hardware improvements, at (almost) the same hardware price, by increasing the ranges (by a factor of at least 1.5). The project will rely in particular on super-resolution techniques by merging several raw images acquired by the camera. The image produced will be super-resolved and its image quality will be improved by real-time deblurring and denoising at the best state of the art. All of these operations require an increasing computing power. But thanks to the advances and popularization of GPUs and algorithms that take advantage of the massive parallelism of these low-cost devices, it is now possible to perform these operations in real time. In addition, CB disposes of a wide range of algorithms of increasing complexity enabling the adaptation of a processing chain to the computational constraints. The quality improvement will in turn result in better detection and recognition, as well as in the reduction of false alarms.

We detail in this project the hardware and algorithmic adaptations necessary to bring these advances to CATEYE System cameras. The main hardware adaptations concern the optimization of the PSF of the optics, the generation and the control of the sensor micro displacements. Other hardware adaptations are envisaged such as the optimization of the bandwidth of the data flow and the integration of a wide field color camera in place of the existing monochrome camera.

Some elements of the satellite super-resolution chain need to be adapted: deblurring and robust recalibration in the presence of noise, image fusion and denoising, management of saturated pixels, and motion detection for fusion without "ghosts". Optical imaging also presents specific challenges, such as increased noise, color matting (for some products), vignetting and uncompensated residual motion or vibration in the scene. The expected result at the end of the project is a prototype of a super-resolved visible observation system and a real-time processing chain with a TRL of 6.

This project led by a team gathered around CB will have a duration of two years using the current dynamics of its collaborations. The partnership between CB and LERITY has already worked successfully in an initial collaboration in 2019 on the design of denoising and optical flow algorithms, currently ported to GPU or and CPU. The requested contribution will be used to fund the work of a post-doc and an engineer CDD for two years for the CB and an engineer for two years for LERITY.