Multimodal generative Artificial Intelligence for Situational Awareness – ASIAM

The ASIAM project (Multimodal generative Artificial Intelligence for Situational Awareness) aims to develop an innovative and sovereign solution of multimodal artificial intelligence which can empower an operator analysing a situation by significantly increasing his situational awareness.

Generative Artificial Intelligence seems like a game changer technology for information processing and operational decisions. However, it still presents major limitations in the context of critical and/or military applications. These are essentially its lack of precision and reliability, along with limitations in understanding and advanced reasoning. Furthermore, it is today limited to processing text and images. On top of this, there are major concerns over sovereignty and confidentiality of data processed.
In order to overcome these limitations, Thales and the CEA will combine their complimentary expertise to develop sovereign MM-LLM (Multimodal Large Language Models) with three key properties:
- Adaptation of the models to the target domain,
- Extended multimodality, covering text and images, but also cartographic and structured datasets,
- Reasoning capabilities covering detailed descriptions and time-space analysis.

The situational analysis will make use of a large spectrum of possible information: field observations (text messages, audio transcriptions, images and videos…), other sensor data (infrared, radar…), private databases, information available on the web, cartographic and also structured data.

The key challenges addressed by ASIAM are of managing to: i) efficiently fuse heterogeneous data, ii) produce precise synthetic and reliable data which are relevant to the operational context, iii) generate high level comprehensible information by integrating reasoning mechanisms at different time and spatial scales.

The major breakthroughs and innovations expected in the project are:
- New tools for intelligent data annotation and reasoning.
- A first of its kind multimodal model, MapCLIP, enabling the alignment of map data with text in order to extract semantic information from maps.
- New approaches for geospatial reasoning.
- The first MM-LLM for defence, mastered from end to end, capable of exploiting all the mentioned modalities and searching for information with precision, notably using Graph-RAG techniques.
- An objective evaluation of the performance of the MM-LLM developed.
- A demonstrator of the developed functionalities in the scope of the generative AI used to assist an operator.

ASIAM aims to develop a Generative AI which will fulfil the role of a trusted intelligent assistant to the operator, enabling him to easily dialogue efficiently with data sources with the aim of facilitating and accelerating human decision-making. For instance, applied to the intelligence gathering domain, the multimodal generative AI will enable the simultaneous extraction, processing, correlation and interpretation of various types of information coming from multiple sources in order to produce summaries and accelerate the production of reliable briefings. Applied to command and control, it will update the situation map with data gathered on the field and highlight differences to the planned or expected situation.

The ASIAM project is part of the “6.15 Artificial Intelligence” theme, and particularly multimodal artificial intelligence methods for critical applications (6.15.1 and 6.1.2).