Learning under uncertainty: functional dynamics of information processing within frontostriatal circuits – LU2

We have combined expertise from behavior theory, brain functions, and experimental neuroscience. The consortium is unique in Europe. Our project aims at defining the computational principles that allow adaptation of behavior under uncertainty and investigating their neural substrate. The specific aims are:
(1) to determine the influence of uncertainty on learning strategies using behavioural experiments with two animal models (rat, monkey) and decoding learning principles in situations manipulating the degree of uncertainty.
(2) to analyse the respective contribution and the interactions between striatal regions and dopaminergic signals when coping with uncertainty, thanks to electrophysiological experiments in monkeys and pharmacological manipulations in rats.
(3) to define the role of dopaminergic signals in the prefrontal processing of uncertainty, thanks to simultaneous recordings in the anterior cingulate cortex and in the mesencephalon in monkeys, and the analysis of effect of pharmacological inactivations in rats.
(4) to synthesize the functional principles of neural networks contributing to learning under uncertainty, by developing theoretical modeling tools and neural simulations.

Our research led to the development of a behavioural test adaptable to different species (rat, monkey, human), which allow analysing the capacity and the dynamics of adaptation under uncertainty. Uncertainty is changed regularly in order to test the regulation of exploratory strategies. Preliminary data show how adaptations to negative and positive feedback obtained during exploration are modulated under various levels of uncertainty. The work by ISIR aimed first at studying which existing computational models of reinforcement learning best account for dopaminergic signals as observe din the literature, in order to make precise predictions for the experiments in the project. These models have also been used to analyse some behavioural data of the partners (monkey data from Inserm (Khamassi et al , in revision), rat data from INCIA.

A key issue in neuroscience concerns the neural substrates for decision making in varying environments. Reaching beyond the actual focus on error-based learning, this proposal is of general significance for understanding how uncertain evidence about the consequences of actions influences neural activity states and behavior. Understanding how the brain resolves uncertainty during choice behavior is also a fundamental issue in theories of economic decision making (Neuroeconomics), and may underlie individual differences in attitude toward risk and ambiguity.
Our proposal also has important implications for the clinical domain because it addresses the neural bases of mechanisms that are key to the behavioural flexibility and that are impaired in obsessive behaviours, in addiction, or in executive dysfunctions (Schizophrenia, Parkinson).
This project is situated at the interface between cognitive neuroscience and robotic research. Like all adaptive systems, artificial agents need to explore and master uncertainty. We intend to provide tools to incorporate uncertainty processing into reinforcement learning models to express truly adaptive behaviors. Thus, our results based on actual biological data have the potential to be integrated into the design and control architecture of autonomous robots.

Part of this work has already been published: Coutureau E, Esclassan F, Di Scala G, Marchand AR. The Role of the Rat Medial Prefrontal Cortex in Adapting to Changes in Instrumental Contingency, 2012, PLoS ONE 7(4): e33302. doi:10.1371/journal.pone.0033302. and an conference paper has obtained the best paper award at the SAB 2012 meeting (Bellot et al 2012). Other papers are submitted or in preparation. A new computational framework has been proposed to distinguish dopamine-based learning mechanisms that engage different cortico-striatal loops (Khamassi & Humphries, 2012).

The complexity of societies, pressures to act, and the need to adapt to continuous technological progress imply making decisions with uncertain consequences. Above all, learning and acting in uncertain environments require evaluating the uncertainty and the outcomes of every action. Several theoretical approaches (machine learning, neuroeconomics) have provided mathematical tools to study decision making under uncertainty. However, these tools remain limited and incapable to explain the astonishing adaptive capacity of animals, let alone of humans. Yet, understanding these mechanisms would give powerful solutions to problems requiring the construction of autonomous artifacts (robots), capable of dealing with ambiguous situations. This understanding would also offer clear analytical tools to evaluate the distinctiveness of mental diseases that are characterized by maladapted (Schizophrénia), compulsive (OCD), or self-destructive (addictions) behaviors.
Neuroscience shows that specific regions of the prefrontal cortex, striatum, and their dopaminergic afferences play a major role. According to current hypotheses, the interactions between these regions and their modulation allow behavioral flexibility, to cope with uncertainties generated by the environment. Preliminary data from the partners of the project show that subdivisions of the medial frontal cortex and subregions of the striatum represent several levels of uncertainty. These structures would integrate statistical information on action outcomes to optimize learning mechanisms. However, the synergy between the different regions, the differential role of dopamine on each node of the system, and the fundamental principles that govern these mechanisms are unknown.
We propose to resolve these issues using an interdisciplinary study of learning under uncertainty. To this end, we have regrouped expertise from behavioral and cognitive theory, and from experimental neurosciences, forming a consortium unique in Europe. Our project aims at defining the computational principles underlying behavioral adaptation under uncertainty, and describing the neural mechanisms at play. The specific aims are:
(1) To determine the influence of uncertainty on learning using experiments with two animal models (rats, monkeys) and by identifying the principles governing learning under various levels of uncertainty.
(2) To analyze the respective contribution and the nature of interactions between striatal and dopaminergic neural signals during learning under uncertainty, on the basis of neurophysiological recordings in behaving monkeys, and pharmacological manipulations in rats.
(3) To define the role of dopaminergic signals and transmission in the processing of uncertainty in the medial prefrontal cortex, by means of recordings in the anterior cingulate cortex and midbrain in monkeys performing learning under uncertainty, and by evaluating the effects of altering aminergic transmission in the rat prefrontal cortical subdivisions.
(4) To synthesize the functional principles of neural networks contributing to learning under uncertainty through the development of theoretical models and neural simulations driven by the neurobiological data.
These experiments will provide new fundamental knowledge on the regulation of behavior and provide tools applicable both to new emerging technologies and to human clinical evaluations.