A Visual Memory Network for Scene Understanding – AVENUE

People visually interpret any new environment, interact with, and navigate it almost effortlessly. Despite significant progress, this level of visual intelligence has not been achieved by artificial systems. Project AVENUE aims to address this through a visual memory network for human-like interpretation of scenes. To this end, we address three scientific challenges. The first is to learn a network representation of image, video and text data collections, to leverage their inherent diverse cues. The second is to depart from supervised learning paradigms, without compromising on the performance. The third one is to perform inference with the learnt network, e.g., to estimate physical and functional properties of objects, or give cautionary advice for navigating a scene. Progress on these fronts will provide a new framework for interactions between people and artificial systems, e.g., reliable assistive technology to help the visually impaired, with potential impact on a global scale.