Synchronization of speech and gestures – SYNCOGEST

The SYNCOGEST project aims at modeling a part of the gestuality (mimics, postures and gestures) spontaneously deployed by speakers and their interlocutors during face-to-face communication, with a view to endowing embodied virtual conversational agents (ECAs) with a more natural and efficient attitude in interactional context. The aim is to propose a model for the automatic generation of appropriate gestures synchronized with speech in production (communicative gestures). This project is based on the articulation of complementary interdisciplinary approaches, bringing together specialists in artificial intelligence, language sciences and movement sciences.
If the automatic generation of ECA gestures is of increasing interest, the complex relationship between gestures and speech is still largely to be determined; indeed, it implies identifying the types of gestures spontaneously performed by speakers and interlocutors (gestures related to verbal content or to the regulation of the interaction) and understanding their synchronization modes with speech (accentual, intonative, propositional criteria...). We therefore propose first to collect and annotate a dyadic reference corpus (gestures and speech), and then to work on a generation model based on deep learning adapted to this type of problem.