Do human and artificial intelligence share semantic representations? – AI-REPS

Recent developments in AI (artificial intelligence) and deep learning are revolutionizing technology and society. Cognitive neuroscience can draw inspiration from these deep neural networks (DNNs) to understand important brain functions like face recognition in vision, or word and sentence comprehension in natural language processing (NLP). Many DNNs develop latent representations that embed items in high-dimensional vectors with powerful compositional properties, including the ability to perform simple linear operations on complex constructs (e.g. QUEEN-WOMAN+MAN=KING). We hypothesize that certain human brain regions also offer vector compositional properties to manipulate high-level semantic concepts via simple linear operations. This multi-disciplinary project combines the fields of neuroscience, vision science, linguistics, and AI. We will use fMRI to compare semantic representations in humans vs. AI. We will systematically relate human multivariate fMRI patterns to three of these DNN latent vector spaces, respectively encoding (i) face images, (ii) words and (iii) sentences textual representations. In each case, we will use multivariate Representational Similarity Analysis (RSA) to reveal a potential isomorphism between human and AI representations. In addition, we will directly test the linear composition properties of the human multivariate encoding space using stimulus sets derived from the artificial DNN counterparts. The potential scientific impact is obvious in Neuroscience: the project could uncover the functional architecture of large regions of cortex involved in vision (particularly face representations) or language processing. The expected impact is also high for Social and Human Sciences, because our project could pave the way to a theoretical or even computational explanation of uniquely human abilities that are primordial for social cognition (language, face recognition). Finally, this project will not only allow us to understand how semantic information is organized in the brain, but could also help us validate and interpret current AI models—or suggest potential improvements to an already revolutionary technology.