Explaining Deep lEarning Models of Satellite Time Series for the Agritech domain. – EDEM

Deep Learning for Crop Mapping

Crop mapping from hyperspectral imagery (HSI) involves classifying land pixels into crop types based on their spectral signatures. Two dominant DL architectures are commonly applied:

- 2D-CNN (Convolutional Neural Network): Extracts local spatial and spectral features by applying convolutional filters across the image. It is well-suited for capturing texture and spatial patterns in HSI data.

- Vision Transformer (ViT): Uses self-attention mechanisms to capture long-range dependencies across the image, making it effective at modeling global context — particularly useful when crop fields span large, heterogeneous areas.

Both models are trained to map each pixel (or patch) to a crop class label, typically using labeled ground truth data.

The Hyperparameter Tuning Challenge

Adapting these models from airborne to spaceborne HSI is non-trivial, as spaceborne sensors introduce different noise levels, spatial resolutions, and spectral characteristics. Standard hyperparameter tuning relies on labeled validation data, which is scarce and expensive to collect in remote sensing contexts.

The xAI-Based Approach: Integrated Gradients

In our work we adapt Integrated Gradients (IG), an xAI method that attributes a model's prediction to its input features by computing the gradient of the output with respect to the input, integrated along a path from a baseline (e.g., a blank or zero image) to the actual input. This reveals which spectral bands or spatial regions most influenced a given prediction.

Problem 1:

To take the problem of computing explainability in terms of causal effect on the classification model, we have revisited patch-based hyperspectral satellite image (HSI) classification (crop mapping) from a causal perspective.

Using this paradigm, we fairly assume that the central pixel and its local neighborhood constitute the main source of the predicted label crop. By explicitly controlling spatial leakage through a disjoint data partitioning protocol, we showed that common evaluation practices not only inflate performance metrics but also alter the causal structure learned by state-of-the-art models.

Our causal sensitivity analysis reveals that, in the absence of leakage, models consistently rely on the Region of Interest to support their predictions, whereas under random sampling, this causal link becomes obscured by confounding spatial correlations.

These findings highlight that reliable assessment of HSI classifiers requires both leakage-free evaluation and causal scrutiny of learned representations. Beyond improving benchmarking practices, our work calls for a rethinking of patch-based HSI classification toward models and protocols that favor causal fidelity over shortcut-driven performance.

Problem 2:

To address problem 2, namely the automatic tuning of deep learning architectural elements to optimize crop mapping models.

We focus on two state-of-the-art DL models widely used for airborne hyperspectral imaging (HSI) — an attention-based Vision Transformer (ViT) and a standard 2D-CNN. To adapt these models to the distinct characteristics of spaceborne HSI, we assessed the consistency of explanations across multiple model instances generated using the Integrated Gradients xAI method.

Assuming that well-designed DL architectures yield more focused and interpretable outputs, our approach computes a consistency metric to automatically determine the optimal hyperparameter configuration without additional labeled validation data. We have validated the effectiveness of our method through experiments on spaceborne Hyperspectral Precursor and Application Mission (PRISMA) data applied to crop type mapping.

We have published our work in 2 conference papers :

1) Rethinking HyperSpectral Image Classification (HSIC) Benchmark with Explainability (xAI) under a Causal Estimation Perspective. (EDBT/ICDT Workshops - ceur-ws.org/Vol-4192/XAI4Science-paper9.pdf)

2) Optimizing Deep Learning for Satellite Hyperspectral Data: an xAI-Driven Approach to Hyperparameter Selection (IGARSS 2025-2025 IEEE International Geoscience and Remote Sensing Symposium - ieeexplore.ieee.org/abstract/document/11243437/)

We currently have 2 ongoing submissions:

1) A novel DL architecture for corp mapping on HSI time series (IEEE Transactions on Geoscience and Remote Sensing)

2) A tutorial of xAI applied to Remote Sensing and Earth observation at ADBIS (European Conference on Advances in Databases and Information Systems).

Publicly available land-cover products provide rich semantic information for building large-scale training datasets in satellite-based crop mapping. However, effectively exploiting these products requires reliable labeled samples from inherently noisy thematic maps.

As a future perspective, we plan to leverage Causa-based xAI to perform efficient weak-label extraction from large-scale thematic products.

Our goal is to develop a framework that identifies reliable labeled units in an unsupervised fashion through two complementary components: (1) a window-based stratified sampling strategy that mitigates spatial autocorrelation by controlling the spatial density and distribution of selected samples, and (2) a context-aware, perturbation-based filtering mechanism that detects spectrally or temporally inconsistent pixels — indicative of potential spectral mixing.

Deep Learning (DL) models are today the most widely and successful techniques adopted on Multi-Variate Time Series (MTS) of satellite data for agricultural monitoring and mapping. Although effective, DL models are often considered black boxes due to their complex and opaque decision-making processes. This lack of transparency is a showstopper in the adoption process of these technologies by end-users. Efforts have been made in recent years to produce Explainable Machine Learning techniques (xAI) to provide a stronger descriptive approach to Deep Learning (DL) algorithms, allowing end-users to increase their trust and confidence in the model output. The most recurrent xAI method is based on Relevance Attribution, where importance scores are assigned to each data feature over time according to their importance for the model prediction. Relevance Attribution methods cannot explain the causes that drive a model to assign a specific target to a given instance. Such causes can have a dual nature: (1) some data variables or instances can induce a DL prediction model to assign importance to specific features, and (2) the obtained results can also depend on the model's internal primitives. In our work, we want to propose a novel Explainability (xAI) framework that aims to produce relevant explanations for DL models applied to MTS satellite data, allowing the users to perform large-scale agricultural mapping and monitoring. This high-level problem will be at the core of our project, which we name EDEM: Explaining Deep Learning Models of Satellite Time Series for the Agritech domain. EDEM aims to define novel Explainability methods that capture causal relationships between model outcomes and important events inferred from the data and DL building blocks. Throughout this project, we will validate the utility of the proposed explanations in several real-world use cases from agricultural monitoring using publicly available benchmark datasets, including France-specific data.