Earth Observation and Generative AI for Rare Event recognition – OGRE

Increasing availability of remote sensing data, such as the images acquired by the Sentinel program, offers numerous opportunities to automate various tasks in Earth Observation. However, most of the raw data acquired by aerial and satellite sensors represent usual phenomenon that do not required a deep analysis. On the opposite, events of interest can be hard to identify in the abundance of remote sensing data acquired everyday. It has become critical to be able to automatically identify observations that contain the most information. Artificial intelligence can play a role in filtering unusual observations that could benefit from a more thorough analysis.

The OGRE project (Earth Observation and Generative AI for Rare Event recognition) aims to develop new techniques based on deep generative models to identify extreme events in multimodal SAR/optical remote sensing stacks. We hypothesize that the most relevant observations are the image that exhibit unusual characteristics, deviating from the standard image distribution. This definition recoups natural disasters (floods, wildfires, storms, earthquakes...), environmental and meteorological events (ice melting, droughts) or unusual and illegal human activities (deforestation, contraband, unlawful buidling). To identify these events, the project mixes the state of the art in unsupervised out-of-distribution sample detection and deep generative models, such as diffusion models and masked autoencoders. Indeed, these model are able to learn the data distribution, without any human supervision. Criterion such as reconstruction error can then be used as proxies to estimate the likelihood of new observations, and therefore determining whether this new data belongs to the distribution, or outside of it. Such images are likely to be exceptional events that are scarcely observed in the usual data distribution.

The strength of the project is to build a framework dedicated to remote sensing imagery by developing generative models that are tailored to multispectral and SAR images, extending these models well outside their usual computer vision applications. By building upon these tools, the project moves forwards along three dimensions:
- conditioning generative models, and how these conditions can influence their ability to detect different types of extreme events. In particular, we aim to be able to detect events that are only rare in some context (a snowstorm is exceptional in Paris, but not in Alaska).
- localization of the event in the image at the pixel level, and not only at the image level,
- taking into account the dynamics of extreme events by extending the generative models to satellite image time series, allowing us to detect rare events that constitue a temporal discontinuity in between two images (such as changes in the urban tissue).

Merging deep generative models and out-of-distribution image detection is a new perspective for Earth Observation. The project builds a new framework with a strong explatory dimension. Nonethless, it is grounded in practical applications and real use cases, on which the newly developed techniques will be experimentally validated. These use cases range from detecting flood events to ice cap meltings, deep fake detection and monitoring urban changes.