Lexicon of Kinaesthetic Human Machine Interaction – LEXIKHUM

In terms of method, we wish to focus initially on the understanding of human-human interaction. We want to extract a linguistically interpretable and programmable understanding of the interactions to create virtual partners that will be comparable to humans. Finally, an application demonstration in a real helicopter piloting situation will be used as a proof of scale of the developed concepts.
We plan to develop the concept of a kinaesthetic sense unit that will be the basis for kinaesthetic communication.
To develop this project, we have organised the project into workpackages as follows:
- WP1: Management, communication and legal procedures
- WP2 : Tools and software for experiments (SEMAPHORO-2D, Software environment)
- WP3 : Study on the information needed to understand the intentions of others
- WP4 : Identification of kinaesthetic sense units in human-human interaction
- WP5: Development and evaluation of kinesthetic sense units
- WP6: Evaluation of kinesthetic sense unit-based augmented driving

At the moment we have developed the robot that will be used for the studies. Using an older version we have developed the software for interfacing the kinesthetic sense units with different technical systems (3 different robots and a flight simulator).
The development of software analysis tools for interaction situations has been implemented, and calculation servers are operational and used between partners.

This work will increase the understanding of human-robot interaction, it will provide a set of kinesthetic sense units that will be indexed according to the meaning they bring to the interaction. From an operational point of view, these sense units will help to improve the intelligibility of highly automated systems. In this sense, this project aims to propose a new information channel that will reduce the opacity of systems and support cooperation between humans and machines. This improvement will both optimise the efficiency of this interaction (and therefore the safety of operations), but also improve the acceptability of artificial systems. Through these various communications, the consortium will seek to promote the work carried out in different scientific communities (robotics, psychology, neuroscience, linguistics).
In this project, the robot and the lexicon are intended to be published under open-source licences (CC-nc-by) so that each scientist can use these tools and enrich the lexicon with their kinaesthetic meaning units. Thus, a European project could naturally follow from this project. It would involve European experimental psychologists who could be concerned by this technology. Contacts have already been made with : G. Pezzulo (Rome), E. Burdet (London), E. Pacherie (Paris). We also plan to contact N. Sebanz (Budapest) and P. Haggard (London).
The lexicon developed will be a useful tool for any user (industrial, researcher, user...) who wishes to implement kinaesthetic sentences in a robot. The documentation of the performance of each kinaesthetic sense unit will give him an idea of the effectiveness of the desired message. If, for example, an industrialist wishes to convey the instruction to go left in his system, by means of our lexicon, he will have the assurance that this kinaesthetic sense unit will deliver the message in an understandable way and with a given precision and measured by scientific procedures.
The collaboration of this project with ONERA allows to bring a proof of the operationality of this concept in an aeronautical context directly in demand. Detecting obstacles and helping to navigate a helicopter in a cluttered environment is therefore an important function for improving flight safety. Many manufacturers see force feedback sticks as new ways of interacting with the pilot and some aircraft are already equipped with them without knowing exactly how to specify the feedback. This work will also find direct application in the research of the partner teams, in a general way, in cobotics and more particularly, in assistance robotics and surgical robotics.

- ACM transaction on Human-Robot Interaction, Lucas Roche, Ludovic Saint-Bauzel, Study of Kinesthetic Negotiation Ability in Lightweight Comanipulative Decision-making Tasks: Design and Study of a Virtual Partner based on Human-human Interaction Observation, p.1-23, 2022
- Conférence invité ISVR, Ludovic Saint-Bauzel : Physical Patient-Robot Interaction : A small journey from patients assistance to fundamental studies 15/06/22
- Conférence invité ISVR,Ouriel Grynszpan : The sense of agency in collaborative action
- F. Catteau, A. Morgenstern et C.S. Bianchini. “From gestures to kinesthetic modality: how to express epistemicity in a haptic device for human-machine interaction”. 9th Conf of the Intl. Soc. for Gesture Studies (ISGS) “Gesture: From Description to Application”. Chicago (USA), 13-15/7/22.
- C.S. Bianchini, C. Danet, L. Chevrefils, F. Catteau, C. Thomas, M. Rébulard, A. Contesse et P. Doan. “Transcribing and comparing sign languages and co-speech gestures: the contribution of the Typannot typographic transcription system to the study of epistemicity”. 2nd Intl. Workshop on Cognitive and Functional Explorations in Sign Language Linguistics (SignCafe2). Ragusa (IT), 13-15/10/22.

- C.S. Bianchini, F. Catteau. “Le geste épistémique: identification et comparaison des traits articulatoires sous-paramétriques en gestualité coverbale du français et en langue des signes française (LSF). Faits de Langue “Discours et Multimodalité”, numéro spécial dirigé par A. Morgenstern. Publication prévue: fin 2023.

The aim of this project is to study and develop solutions for human-robot interaction through kinesthetic communication. This work aims at improving human-robotic interaction, it will provide a set of kinaesthesic semantic units that will be stored regarding the meaning they bring to the interaction. In an increasing automation context, this work will support the enhancement of the understandability of smart systems. In this context, bringing common understanding in human-machine interaction is a key feature. This project's focus is to bring a new communicative channel that reduces opacity of systems and support the cooperation between human and machine. This improvement will optimize efficiency (therefore the safety) and acceptability of artificial systems. This consortium will be able to reach multiple community with these new concepts ( robotics, psychology, neurosciences and linguistics).