The space in between: where local meets global in human vision – WHOLESCENE

Vision has puzzled researchers for centuries and remains as enigmatic as ever. During the last seventy years, research has successfully pursued a programme structured around localized neural circuits. In behavioral and neural experiments, circuits are typically characterized using simple laboratory stimuli. At the heart of this classic framework is the assumption that these separate neural circuits operate largely independently of each other, and that their hierarchical feedforward integration eventually leads to complex object and scene recognition. However, this explanatory framework collapses altogether once stimuli are put into context, for example, in natural scenes. Local circuits do not operate independently. Although this fundamental limitation is largely acknowledged, the research community has devoted relatively little efforts towards understanding contextual vision in its full articulation (i.e., beyond summary and natural scene statistics). The few attempts to tackle contextual vision appear stuck between two worlds: globally embedded circuits on the one hand, and phenomenological Gestalt on the other. Here, we propose a new framework for contextual vision by marrying two conceptual/computational approaches spear-headed by the two proponents, based on their parallel work over the past 15 years.

Peter Neri has expanded low-level psychophysical techniques with simple laboratory stimuli to stimuli embedded within natural scenes. He has constructed circuit models based on canonical computations that are able to capture the local, target-related aspects in great detail. However, the global aspects of the natural scene are modeled more abstractly with no mechanistic connection to the local circuits, thus lacking computational insight. In parallel, Michael Herzog has demonstrated the critical importance of global long-range aspects of the stimulus configurations, but has neglected aspects connected with local circuitry. Put another way, Neri’s approach is primarily influenced by low-level psychophysics while Herzog’s approach is primarily influenced by the Gestalt tradition, both experimentally and computationally.

We will first merge the two modeling frameworks to create an umbrella model of vision. If the model is successful, we will make it publicly available. In accompanying psychophysical experiments, we will test and fine-tune the model. In particular, we will test to what extent long-range spatial interactions can be explained by chains of local interactions within a cortical layer, and/or whether top-down interactions are needed to take objecthood into account. To this end, we will conduct psychophysical, fMRI and EEG experiments. At the end of the project, we will organize a workshop with experts in the field.

This joint project represents a unique chance to merge two related research areas and develop a unified model that will guide future research into mid-level vision.