A computable account of human image reconstruction from natural scenes – ACCURATE

Our brain does not reconstruct images by simply putting together small elements to make bigger elements and ever more complex elements in a bottom-up fashion. Instead, our evolving interpretation of the image informs and constrains this reconstruction process in a top-down fashion, by imposing knowledge about the statistics of natural scenes. We know that this happens, but we do now know exactly how: we do not have a computational model that would allow us to incorporate top-down/bottom-up interactions into an artificial visual aid, for example. To achieve this goal, we must obtain a description of these processes and their interaction using the language of circuits and mathematical operations, so that they can be implemented on a prosthetic device. The goal of this proposal is to develop an account of this nature, constrained by detailed empirical data delivered by a novel set of experimental tools that overcome current limitations in our ability to characterize bottom-up processes embedded within natural scenes. Our pilot results demonstrate that we have already made substantial progress not only in terms of designing empirical tools that can effectively constrain candidate computational accounts, but also in terms of developing those computational accounts.

The scientific part of the proposal is divided into 5 main sections. Within each section, we describe how we will target different stages within the visual processing hierarchy and how we will use EEG measurements to further enrich our experimental dataset. Of these 5 sections, the first 3 correspond to different experimental conditions that gradually approximate natural vision: controlled, where rigorous established protocols are adapted to natural scenes; dynamic, where the requirements associated with controlled protocols are relaxed to allow for dynamic stimulus conditions; active, where vision is characterized while embedded in natural behaviour. We will cross-check every step of the way: paradigms/protocols will be developed, cross-validated and extended at different stages of the transition in a mutually informative fashion, so that the limitations and strengths of a given stage become complementary counterparts for the strengths and limitations of other stages.

The two additional sections relate to auditory processing and our computational plan. In the auditory section, we exploit experimental tools which we have developed over the past few years to design hybrid stimuli defined by natural sounds as opposed to scenes, and demonstrate with pilot data that we have already been successful in studying how auditory feature extractors are reshaped by semantic representations of speech signals. We then propose a set of novel manipulations and experiments to dovetail the visual programme, so that the two sensory domains may be merged into a unified computational account of bottom-up/top-down processing.

The computational account, detailed in the final section, will be developed around a collection of important modelling results which we have secured for the purpose of expanding them into a full-scale computational description of the perceptual process. These results combine physiologically inspired circuit models, such as fully specified variants of gain control, with large-scale neural networks from deep machine learning, in particular recurrent variants that support semantic segmentation of natural scenes. We will start from these concrete modelling foundations and build upon them as informed by experiments designed to generate datasets that will constrain the models effectively, to deliver an integrated computational account based not around vague terminology and nebulous concepts, but well-defined operations that can be translated into machine algorithms.