Fusion of Heterogeneous Components by Articial Intelligence – FUSCHIA

INPIXAL specializes in collecting and processing images from drones for ISR (Intelligence, Surveillance, Reconnaissance) missions for military and civil purposes. The experience accumulated during experiments or operational missions for the DGA or institutional clients shows that, while the performance of ISR missions by drone is now well under control, the use of the payload in real time induces a considerable cognitive load. on the operator, who must reconcile divergent objectives:
• Ensure complete coverage of a large area and detect objects of interest
• Observe these objects of interest in a narrow field, classify them, quantify the interest or the potential threat they may represent.
• Maintain an overall tactical situation of the area to be monitored
• Provide real-time data to a response team if the situation warrants it.
• Submit a report of the observations made within a very short time after the mission
Drone flights often last 3 to 12 hours, operator stress and fatigue can lead to errors: undetected objects, poor classification, focus on an object while others are present in the area to be observed.
The objective of the project is to explore methods of assistance to the operator based on artificial intelligence, which would allow him in the future (1) to focus on his tactical objectives instead of manual operations. payload control and object annotation and (2) semi-automatically generate a mission report at the end of the mission.
The images captured during aerial surveillance missions by military or professional drone are accompanied by metadata giving the context of the shooting (complete position of the camera in space, characteristics of the sensor and optics). This multiplexing of images and metadata is based on a NATO standard, STANAG 4609.
The availability of this metadata opens up the possibility of having them processed by an artificial intelligence engine in the same way as image data in order to improve its performance in detcetion range and its classification accuracy. In fact, knowing the position of the camera makes it possible to couple the image in real time with pre-existing cartographic data, on the basis of a digital terrain model. It is then possible to generate data planes parallel to the image pixel planes:
• Depth map of each pixel in the image, allowing the physical size of the observed object to be deduced
• Vector cartographic segmentation map to characterize each pixel of the image: land / sea, urban / non-urban area, infrastructure, public space / private space ....
The processing by the artificial intelligence engine of this additional data will make it possible to prevent it from inferring without reference the size of the objects detected as is the case in conventional architectures, giving rise to parasitic detections of 'objects of aberrant size. It will thus be possible to lower the inference threshold and improve the detection range and classification performance. The availability of vector cartographic data will make it possible for each pixel of the image to adapt the classes of objects to be detected and classified: boats, rocks, buoys on sea areas, buildings, vehicles, humans, infrastructures on land areas for example. . A single artificial intelligence engine will thus be able to process a heterogeneous scene image more efficiently.
INPIXAL, associated with its research partner IMT Atlantique, therefore proposes to explore the methods of processing heterogeneous data using an adapted neuron network, and as such falls under sub-theme 2 of the call for projects.