Adaptive Stereovision Systems in Man, Monkey and Machine – 3D3M

For the modeling part, we used spiking neural networks equipped with a bio-inspired learning rule, the «spike-timing dependent plasticity« or «STDP«. In this approach, the modification of synaptic weights within the neural network depends on the latency of the input and output spikes of the system. Such a model allows the neural network to develop gradually and naturally (i.e. without any supervision) a selectivity to the most frequent visual properties within the input sequences. For example, a network trained with sequences presenting moving objects develops neurons selective to motion direction (Bichler et al., 2012). As part of the project, networks were trained from natural stereoscopic images captured by two cameras (see Hunter and Hibbard, 2015) and then processed from a retina model using center-surround spatial filters. Networks have also been trained from dynamic sequences simulated or captured by event cameras. The selectivity of neurons in the network after learning was compared to those measured in living organisms and in particular within the visual cortex of non-human primates (macaque model). We compared the receptive fields of the network to those measured from approaches in electrophysiology. We also tested whether network responses predicted perception during visual discrimination tasks.

The comparative approaches were based on neuroimaging recordings performed in humans and macaques using the same experimental protocol. We used static (disks defined from the gradient of binocular disparity) and dynamic (disks performing motions in depth) 3D stimuli as well as stimuli invariant to different types of spatial transformations (symmetric stimuli). These stimuli all contained spatial properties that are very common in the environment. For each of these stimuli, we defined control stimuli which shared the same local properties but which did not reflect the statistics of natural scenes. We recorded brain responses to stimuli as well as their respective controls. In particular, we identified the visual areas within which responses were significantly stronger for more natural stimuli. We also performed psychophysical measurements to determine whether the 3D regularities of the environment influence spatial perception and in particular whether objects located in the lower / upper parts of the visual field are more often perceived as closer / further, such as this is the case in natural scenes.

The work carried out in the modeling part of the project allowed us to show that the properties of artificial neural networks after unsupervised training by STDP are very close to what is observed in primates. In the case of a network trained with natural stereoscopic images, the responses of the network at the unitary level but also of the population of neurons are very similar to those which can be measured in electrophysiology in the visual area V1 of the macaque. In this case, the model can therefore help to better understand how the selectivity to 3D properties emerges with experience during development under normal or abnormal vision conditions (the model can notably explain how 3D selectivity can be affected in patients with amblyopia). Results very close to what is measured in electrophysiology have also been observed when the neural network is trained from dynamic stimuli. In this case, the artificial neurons have developed selectivity to the different components of the optical flow and their responses can notably help to reconstruct the 3D structure of the surrounding space.

The comparative studies of the second axis made it possible to characterize which cortical areas deal with different types of spatial regularities in primates. We have been able to demonstrate that in humans and in macaques, certain areas of the brain have significantly stronger responses when the visual stimuli respect the spatial regularities of the environment. These responses might reflect an effecient processing of more common properties within natural scenes. It has been observed with static (discs whose 3D orientation was defined from the gradient of the horizontal binocular disparity) and dynamic (discs performing deep movements) 3D stimuli as well as for stimuli invariant to different types of spatial transformations (symmetric stimuli). Behavioral (psychophysical) measurements have confirmed the hypothesis that objects located in the lower / upper parts of the visual field are more often perceived as closer / more distant, as it is the case in natural scenes.

The results of the project are in line with the initial hypotheses and support the idea that the visual system of human and non-human primates adapts to the spatial (and especially 3D) regularities of the environment. They pave the way for an even more exhaustive characterization of the observed mechanisms, for example from multi-unit recordings which could be made in monkeys. Beyond its fundamental aspect, the project also has implications at the clinical level because the approaches used could make it possible to better understand how the selectivity to certain visual properties evolves in patients suffering from pathologies such as amblyopia but also as a result of a retinal damage in patients with retinitis pigmentosa or macular degeneration. At the industrial level, they could be used in various applications where a rapid and robust extraction of the spatial properties of the environment using cameras is necessary, for example in bio-robotics or for navigation assistance within autonomous vehicles.

This project led to numerous publications (7) in international journals and presentations in international conferences (9). The modeling results were published in the journals Journal of Neuroscience (Chauhan et al., 2018) and Frontiers in Neuroscience (Chauhan et al., 2021). They have also been presented in several international (Chauhan et al., CNS 2017; ECVP 2017; Montlibert et al., Bernstein 2019; Fricker et al., Bernstein 2021; VISAPP 2022; Rancon et al., SNUFA 2021) and national conferences. (Chauhan et al.; SNF 2019). A review article has also been published in Vision Research (Chauhan et al., 2020). The results of the comparative studies have been published in the journals Brain Structure and Function (Bogdanova et al., 2019) and Cerebral Cortex (Hejja-Brichard et al., 2020; Audurier et al., 2021). They have also been presented in several international (Hejja-Brichard et al., ECVP 2017; SFN 2018; Audurier et al., SFN 2019) and national conferences (Bogdanova et al., SNF 2019).

Publications in peer-reviewed international journals:

1) Chauhan T, Masquelier T, Montlibert A & Cottereau BR (2018): Emergence of binocular disparity selectivity through Hebbian learning. Journal of Neuroscience, 38(44), 9563-9578.

2) Bogdanova O, Bogdanov V, Durand JB, Trotter Y, Cottereau BR (2019): Dynamics of the straight-ahead preference in human visual cortex. Brain Structure and Function, 225(1), 173-186.

3) Chauhan T, Hejja-Brichard Y, Cottereau BR (2020): Modelling binocular disparity processing from statistics in natural scenes. Vision Research 176, 27-39.

4) Hejja-Brichard Y, Rima S, Rapha E, Durand JB, Cottereau BR (2020): Stereomotion processing in the non-human primate brain. Cerebral Cortex, 1-16.

5) Chauhan T, Masquelier T & Cottereau BR (2021): Sub-optimality of the early visual system explained through biologically plausible plasticity. Frontiers in Neuroscience, 15.

6) Cottereau BR, Trotter Y, Durand JB (2021): An egocentric straight-ahead bias in primate’s vision. Brain structure and function, 1–13.

7) Audurier P, Héjja-Brichard Y, De Castro V, Kohler P, Norcia AM, Durand JB & Cottereau BR (2021): Symmetry processing in the macaque visual cortex. Cerebral Cortex, bhab358, doi.org/10.1093/cercor/bhab358

One of the major challenges faced by the visual system is to recover the 3D structure of the environment from the ambiguous 2D images that fall on the two retina. This project supports the hypothesis that the nervous system relies on regularities in natural scenes to solve this issue. More neural resources would be dedicated to the processing of 3D visual properties that are more often found in the environment. We propose a multidisciplinary approach based on studies in man, monkey and machine to characterize how this ‘efficient coding’ of 3D properties emerges in early visual cortex and possibly impacts cortical processing in higher-level visual areas and depth perception.

In a first research axis, we will use state-of-art acquisition devices to precisely characterize the images entering the two eyes of someone evolving within various natural environments (indoor, outdoor,…). The stereoscopic inputs will be acquired with 2 asynchronous event-related (AER) cameras (or spiking retina). Like the retina, these AER cameras do not process the entire image at a given frequency but rather transmit information about pixels where a significant luminance change was detected. They therefore permit to drastically reduce the amount of data to handle while keeping the most relevant information. The output of these cameras will be processed by filters close to those encountered in primate visual cortex before to feed an artificial neural network equipped with a bio-inspired learning rule, the spike-timing dependent plasticity (or STDP). This network will automatically (i.e. without supervision) develop selectivity to those image characteristics that occur most frequently within the input stereoscopic sequences and thereby constitutes a plausible model to explain how selectivity to 3D regularities in natural scenes emerges in primate early visual cortex.

In a second research axis, we will perform neuroimaging and behavioural measurements in both human and non-human primate. This multi-specie approach is very important in order to characterize the homologies but also the differences between human and macaques for these adaptive mechanisms. fMRI recording will permit to characterize the cortical areas whose responses reflect 3D scene statistics in both species. Psychophysical experiment will bring to light how depth perception is affected by 3D regularities in natural environment. Altogether, our results will reveal the cortical networks whose responses reflect 3D statistics and affect perception in both human and macaque. This approach will be used for 3D regularities that were already reported (like the prevalence of surfaces aligned with the ground and the higher probability for point in the near/far space to be bright/dark) but also for new biases that will be discovered with the studies of the first axis.

Our project has major applications for modelling and better understanding how sensory systems in primate adapt their responses to efficiently process the properties of the surrounding environment. On a clinical perspective, it could provide important insight regarding selectivity to different properties in patients suffering from visual pathologies (e.g. in amblyopic patients or patient suffering from macular degeneration). At the industrial level, it could be used in various applications where a fast a robust extraction of 3D properties of the environment is needed.