Confidence, noradrenaline and adaptive learning – CONFI-LEARN

We use a mathematical model of the learning process. This model shows resemblance with the actual subjects' estimates, offering a useful tool to model quantitatively learning and confidence.

We will use functional MRI (fMRI) to measure indirectly brain activity in a spatially resolved manner. We will measure the activity of the locus coeruleus, the main nucleus that releases noradrenaline in the brain, in order to test whether its activity level is on par with confidence during the task.

We will also use, as a complementary measure, the pupil diameter. This diameter, under well controlled condition, reflect (among other things) the release of noradrenaline in the brain.

Magnetoencephalography (MEG) measures the minuscule changes in magnetic field, on the scalp, that arise due to the electrical activity of neurons in the brain. We will use MEG to measure the dynamics of cortical interactions and the state of brain network, in particular through the oscillation of brain activity in specific frequency bands.

We hope to test causally the implication of noradrenaline with pharmacological intervention, but this is difficult to do in practice. We plan to use a selective reuptake inhibitor of noradrenaline (atomoxetine) in order to measure how changes in noradrenaline release in the brain impacts adaptive learning.

Last, we will use encoding models for fMRI, which enable to test how different potential coding schemes of information in the brain account for brain activity. We will also use artificial, recurrent neural network, trained on the same task as participants, to test what are the coding scheme that can be used to solve this task.

We have made important progress on several aspects of the project.

The pupil diameter, in particular its tonic level (i.e. the most temporally stable one) change on par with the ideal confidence level during our learning task. This finding is in line with our hypothesis that the noradrenergic system tracks confidence during learning. We have also completed an MEG study. The result show that the power of beta-band oscillation specifically change as a function of the ideal confidence level during the task, and that their level predict the actual confidence level reported by subjects. Those brain oscillation and the pupil diameter, on top of ideal confidence, modulated well-known neural signature of surprise measure with MEG during our task. Those results provide on of the most direct piece of evidence to date, of confidence-weighted surprise responses in the brain, whose existence is prescribed by optimal models of adaptive learning, and the implication of specific brain oscillations and the noradrenergic system in this process.

Our theory works also demonstrates that low-dimensional (i.e. with few units) recurrent activity can approximate with almost no error the optimal solution in our learning task.

Those first results are in line with our hypothesis that confidence regulate the learning process. They also start to shape the mechanisms of such a regulation, supporting again our hypothesis that noradrenaline is key to this process. Our results need further confirmation with other approaches (fMRI of the locus coeruleus and pharmacology).

Heilbron & Meyniel (2019) Confidence resets reveals hierarchical adaptive learning in humans. Plos Computational Biology
Meyniel (2019) Brain dynamics for confidence-weighted learning. BioRxiv doi.org/10.1101/769315

In a changing world, learning must be adaptive. When a change point occurs, what has been learned previously suddenly becomes outdated and must quickly be re-learned on the basis of new data. By contrast, in stable periods, we should accumulate more data in order to stabilize and refine what has been learned previously. This ability to balance flexibility and stability appropriately is the core problem of adaptive learning.

In practice, this problem is very difficult because our world is also uncertain, such that it is not clear whether flexibility or stability should be favored. For instance, a heatwave in winter may denote an infrequent fluctuation in normal weather or a profound change in climate. Should I trust my current knowledge and stabilize it, or be flexible and revise it?

Bayesian inference is a mathematical tool that affords optimal solutions to arbitrate between flexibility and stability. Inspired by those Bayes-optimal solutions, I propose that human algorithms for adaptive learning are accurate because they rely on a sense of confidence to strike the balance between flexibility and stability. This project now aims to uncover the neuro-cheminal mechanisms of this confidence-weighted learning algorithm.

I propose that the shaping of brain-scale dynamic interactions by noradrenaline may implement a confidence-weighting of information during learning. To test this idea, I propose to use an integrated application of computational modeling, behavioral data, functional magnetic resonance imaging (fMRI), magnetoencephalography (MEG) and pharmacological interventions in human subjects. The project is organized into three work-packages (WPs) that are unified by the use of a single probabilistic learning task and an optimal Bayesian model that were previously validated.

In WP1, I will improve fMRI methods at high field (7T) to probe non-invasively the activity of the locus-coeruleus (LC, the main noradrenergic nucleus) in humans. Preliminary data in a 3T scanner are already promising. With this method, I will then relate on a trial-by-trial basis, the activity in the LC to behavior and to the hidden variables of Bayes-optimal inference during a learning task, and therefore test the hypothesis that LC activity correlates with confidence. Such correlations would be compatible with the hypothesis that LC activity mediates an effect of confidence on learning.

In WP2, I will test the causal role of noradrenaline in the confidence-weighting of learning, and its specificity with respect to another neuromodulator (acetylcholine) using a pharmacological manipulation. This manipulation will be combined with MEG so as to reveal the potential mediation of LC activity onto brain-dynamics in confidence-weighted computations. My hypothesis is that LC activity regulates learning by modulating information flow in the cortex, which can be assessed with frequency fingerprints in MEG signals.

In WP3, I will probe the neural codes of the probability distributions inferred during the task, and the format of the associated confidence information which is critical for adaptive learning. I combine artificial neural networks and computational neuroscience to propose several putative codes. I will develop encoding models, which allow to translate the neural details of those putative codes into testable predictions at the fMRI level. I will use machine learning techniques to test and compare those models.

Together, those results may advance neuroscience and learning theories, which is the core aim of this project. The project also offers speculative, but promising implications for technological and medical developments.