Inductive Biases for Compositionality-capable Deep Learning Models of Natural Language – COMPO

In Natural language processing, large deep learning-based language models have substantially changed the field in the last few years. However, we observe that these models sometimes fail in cases where humans do not have particular problems.

The COMPO project proposes to design new models with better generalization capabilities, more parsimony and lower computational requirements. The project focuses on the task of compositional semantic interpretation of natural language, that is a task for which the weaknesses of large contemporary language models have been best identified. The project proposes to introduce inductive biases from the scientific tradition in linguistics and cognitive sciences into the models.

The inductive biases we consider are on the one hand the introduction of structure biases inspired by linguistics. There is a very strong tradition in linguistics that proposes to calculate meaning from a syntactic structure. On the other hand, inspired by the tradition in cognitive science, we think that semantic interpretation in humans is related to the properties of working memory: our working memory cannot easily recall sequences of unstructured information, on the contrary it must compose the information to give it a structure and a meaning.

The project is part of a current trend that seeks to inject theoretical aspects into artificial intelligence models that are mostly derived from successful applications. Viewed from the other side, the project also allows to test and scale up theoretical proposals that are traditionally evaluated on a smaller scale.