Embedded multi-sourCE analySis for the exploration of the underwAteR environment (CESAR) – CESAR

Human knowledge of the underwater environment are limited and does not allow industrial exploitation nor ecosystem monitoring. This project foccus on the exploration of complex environments by cross-referencing multimodal data for decision-making support and knowledge improvement. More specifically, the ambition of this project is to fit out underwater drones with embedded intelligence for observation and analysis of the underwater scene that is invariant on a spatio-temporal scale.

The deficiency in human knowledge of the underwater environment prevents industrial exploitation and monitoring of the marine ecosystem. Indeed, a drone must maintain high decision-making performance despite changes in deployment zones, testing seasons, and operational conditions. These performances are linked to those of semantic segmentation algorithms upon which the drone relies to operate. However, experiments conducted by the Vision-AD team show that semantic segmentation rates of underwater images depend on the observation zone and time. This spatio-temporal dependency constitutes a major obstacle that must be adressed to facilitate drone missions.

The main hypothesis on which the project relies is that by including environmental data within the semantic segmentation algorithm of images, a link is created between the environment and the image quality, which can subsequently be exploited by the decision-making algorithm.

The observation system to be designed allows for semantic segmentation of underwater scenes based on the integration of heterogeneous sources embedded on an underwater drone. Reviewing recent state-of-the-art for machine learning algorithms that could be used in the project has allowed us to extract promising methods such as "contrastive learning" to enhance learning, "curriculum learning" for gradual incorporation of environmental data, and adapting "adaptive neural networks" to transformers to control decision-making based on image complexity and the impact of environmental data.

The CESAR project focuses on use cases where drone reactivity is necessary. This includes not only security applications to detect unknown objects but also autonomous navigation, inspection, and real-time positioning applications. For these applications, rapid decision-making within the drone offers several advantages. However, this poses the problem of the difficulty of embedding intensive computation on embedded targets. We aim to reduce these constraints through the application of algorithm-architecture suitability and more particularly through knowledge distillation, which has not yet been examined in the underwater domain.

Thus, the project's four objectives are:

- Identify parameters of the underwater environment that influence image quality.
- Enhance the mathematical modeling of AI algorithms by integrating heterogeneous data inspired by innovative techniques such as "contrastive learning" and "curriculum learning", and by adapting the architecture of algorithms like "adaptive neural networks" to transformers.
- Accelerate the process of scene interpretation through the application of KD techniques.
- Explore the design space of reconfigurable architectures for implementing high-performance computations.