Decoding and monitoring operators’ cognitive states – BLADE

To explore these questions, we conducted several neuroscience and experimental psychology experiments in the laboratory with adult volunteers. We used various techniques:

- Electroencephalography (EEG) to record brain activity;

- Eye-tracking to measure oculomotor behaviors;

- Psychophysics to analyze behavioral performances.

We then combined these data with machine learning methods to identify brain and ocular "signatures" capable of reflecting levels of control, attention, and confidence in operators.

These experiments allowed us to better understand how the brain and behavior reflect the sense of control, attention, and confidence.

Sense of Control

We first demonstrated that the sense of control can be operationalized through the capacity for prediction. Individuals feel a greater sense of control over events when they can predict them compared to when they cannot anticipate them. We also observed that predicted events are perceived more clearly and processed more efficiently by the brain than unanticipated events.

Using EEG, we identified specific brain signatures (e.g., N1, P2, and P3 components) associated with this predictive capacity. Finally, we found that predicted events are better decoded by machine learning algorithms applied to brain signals, confirming that they are processed more efficiently by the brain.

These results are particularly important for understanding interactions with automated and opaque systems, where operators, becoming passive supervisors, struggle to predict the actions and decisions of the machine, which can alter their sense of control and reduce their engagement.

Attention

We showed that the orientation of attention in space can be predicted from EEG signals, particularly through the activity of alpha waves and raw brain activity. Alpha waves are associated with the shifting of attention across space, while raw brain activity relates to the amplification of expected sensory information. Moreover, the speed of pupil dilation proved to be a good indicator of attentional state: it helps determine whether attention is focused on a single location or divided among several areas.

These results are crucial for interactions with automated systems, as poorly oriented or divided attention can delay the detection of anomalies and compromise safety.

Confidence

We demonstrated that the confidence a person places in their decisions partly depends on their motor actions, suggesting that individuals distinguish between good and bad decisions more easily when they are active rather than passive. We also showed that when individuals doubt their decisions, they are more likely to accept choices made by a machine or another partner, even when those choices may be incorrect. Finally, we successfully predicted participants' levels of confidence by analysing pupil dilation, using machine learning methods.

These results show that poorly calibrated confidence can lead either to excessive dependence on the machine or to an unjustified rejection of its decisions, thus compromising the safety of human-machine interactions.

This project has opened numerous perspectives:

- New doctoral theses have started to deepen the study of control, prediction and confidence in complex tasks;

- We are participating in a European project (ERC HORIZON-MSCA) focusing on confidence in decision-making;

- Several collaborations have been developed with laboratories in France and Europe to extend these research avenues;

- Finally, these results could find concrete applications in fields such as aviation, assisted driving, and human-robot interactions.

Barne, L. C.; Giordano, J.; Collins, T.; Desantis, A. Decoding Trans-Saccadic Prediction Error. Journal of Neuroscience. 2023, 43(11), 1933-1939.

Bonnet, E.; Masson, G. S.; Desantis, A. What over When in causal agency: Causal experience prioritizes outcome prediction over temporal priority. Consciousness and Cognition. 2022, 104, 103378.

Ficarella, S. C.; Desantis, A.; Zenon, A.; Burle, B. Preparing to React: A Behavioral Study on the Interplay between Proactive and Reactive Action Inhibition. Brain Sciences. 2021, 11, 680.

Desantis, A.; Chan-Hon-Tong, A.; Collins, T.; Hogendoorn, H.; Cavanagh, P. Decoding the Temporal Dynamics of Covert Spatial Attention Using Multivariate EEG Analysis: Contributions of Raw Amplitude and Alpha Power. Frontiers in Human Neuroscience. 2020, 14.

Nowadays, numerous human activities rely on the interaction with automated systems. The positive aspects associated with automation are undeniable: our lives are certainly easier and safer. However, automated systems do not simply replace humans; they create new challenges for the individuals in charge of operating them. Indeed, automated systems are not entirely independent, they still require human control and have to be coordinated with human control processes. A challenge in human-machine interactions is posed by the fact that people can experience a severe reduction in their sense of control over the outcomes of processes undertaken by automated systems. In turn, this loss of control drastically changes users’ perceptual and cognitive processing: their perceptual sensitivity decreases, they can exhibit over-trust in the control of the machine, and they lose situation awareness, leading to a considerable reduction of engagement in the activities of the system. Importantly, these are aspects that can escape operators’ awareness. The series of impairments associated to the decrease of control over the operations of the automated system is also known as the out-of-the-loop (OOL) performance problem. In case of failure of the automated system the OOL problem can lead to longer latency to determine what has failed, to decide if an intervention is necessary and to find the adequate course of action. The crash of the Air France flight 447 between Rio de Janeiro and Paris in 2009 or the nuclear incident in the Three Mile Island in 1979, are only a few examples showing the consequences of a loss of operators’ control. Given that these aspects can escape the operators’ awareness, it is crucial to be able to decode and monitor over time from (neuro)physiological data the operators’ cognitive state, and eventually allow the automated system to adjust its parameters online and help operators regain engagement and control. This is a challenge that the current research project addresses. More specifically, a loss of engagement and control during interaction with automated machine can be associated to changes in at least three processes: 1) a decrease in experience of control; 2) a loss of attentional focus; 3) over-trust toward the system. Accordingly, this project describes basic research in neuroscience aiming at identifying, decoding and monitoring over time, three cognitive processes: 1) experience of control (control vs no-control), 2) attentional state (divided vs focused attention), and 3) confidence level in one’s own decisions (low vs high confidence), using electroencephalography, functional near-infrared spectroscopy and eye-tracking combined with machine learning tools. The ability to monitor operators’ internal states using (neuro)physiological measures is crucial to improve safety during human-machine interactions and to design more adaptable and flexible automated systems. The project will be conducted at the Office Nationale des Etudes et Recherches Aerospatiales who has a large experience in the study of human-machine interactions.