Deriving Performance-Optimized Gestural Interaction – GESTURE

The project combines interaction design, empirical studies and computational modelling

Technical results
- A gestural vocabulary on an interactive watch published at the national HMI'17 conference.
- A smartphone-based robotic system that explores other functions of gesture (e.g. social communication) published at UIST'18.
- A new interactive skin technology offering a rich gesture vocabulary (UIST'19)
- A new presentation of gesture shortcuts. Article in progress for CHI'22.
Methodological results
- New methods to study the transition from menus to gesture shortcuts (but also keyboard shortcuts) published at IHM'18.
Theoretical results
- Introduction and use of the concept of «retroactive transfer« from psychology for gesture interaction published at ACM CHI'20
- Elaboration and validation of a model of the transition from menus to gesture shortcuts (and keyboard shortcuts) published at ACM ToCHI (2022).

Translated with www.DeepL.com/Translator (free version)

best paper award à ihm'18

Marc Teyssier, Gilles Bailly, Catherine Pelachaud, Eric Lecolinet. 2020. Conveying Emotions Through Device-Initiated Touch. IEEE Transactions on Affective Computing. doi: 10.1109/TAFFC.2020.3008693.

Gilles Bailly, Mehdi Khamassi, Benoit Girard. Computational Model of the Transition from Menus to Shortcuts. ACM Transactions of Computer Human Interaction (ToCHI). Sous Press

Marc Teyssier, Gilles Bailly, Catherine Pelachaud, Eric Lecolinet.. MobiLimb: Augmenting Mobile Devices with a Robotic Limb. The 31st Annual ACM Symposium, Oct 2018, Berlin, Germany. ACM Press,

Marc Teyssier, Gilles Bailly, Catherine Pelachaud, Eric Lecolinet, Andrew Conn, and Anne Roudaut. 2019. Skin-On Interfaces: A Bio-Driven Approach for Artificial Skin Design to Cover Interactive Devices. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology(UIST '19). Association for Computing Machinery, New York, NY, USA, 307–322. DOI:https://doi.org/10.1145/3332165.3347943

Reyhaneh Raissi, Evanthia Dimara, Jacquelyn H. Berry, Wayne D. Gray, and Gilles Bailly. 2020. Retroactive Transfer Phenomena in Alternating User Interfaces. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). Association for Computing Machinery, New York, NY, USA, 1–14. DOI:https://doi.org/10.1145/3313831.3376538

Kashyap Todi, Gilles Bailly, Luis Leiva, and Antti Oulasvirta. 2021. Adapting User Interfaces with Model-based Reinforcement Learning. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, Article 573, 1–13. DOI:https://doi.org/10.1145/3411764.3445497

Gilles Bailly, Emmanouil Giannisakis, Marion Morel, Catherine Achard.Caractériser la transition des menus vers les raccourcis claviers. 30eme conférence francophone sur l’interaction homme-machine, Oct 2018, Brest, France. IHM-2018, pp.30-41, 2018. (Meilleur papier)

Emeline Brulé, Gilles Bailly, Marcos Serrano, Marc Teyssier, Samuel Huron. Explorer le potentiel des interactions tangibles rotatives pour les Smart Watches. AFIHM. 29ème conférence franco- phone sur l’Interaction Homme-Machine, Aug 2017, Poitiers, France. ACM, IHM-2017, 8 p., 2017,

Robotic Finger for Mobile Device (Doigt robotise´ sur dispositif portable). FR 1858553. 20/09/2018.
Marc Teyssier, Gilles Bailly, Catherine Pelachaud, Eric Lecolinet.

This project in Human-Computer Interaction (HCI) aims at developing user expertise in interactive systems so that users can quickly and efficiently select a large set of commands. It focuses on gestural interaction, a promising input modality for selecting commands. Indeed, gestural interaction can drastically shift the learning curve of users on a wide variety of platforms (e.g. wearables) and thus deeply modify how users will interact with novel technologies. However, it is first necessary to establish a fundamental understanding of gestural interaction that influences usability and performance of information systems.

This project addresses this objective and provides methods and tools to help designers to create gestural interaction. It primary focuses on stroke gestures (2D gestures), which are widespread on devices equipped with tactile surface (e.g. touchscreen, trackpad, etc.). The difficulty is that we consider both novice users and expert users. The gesture vocabularies should be easy to learn, efficient and large enough to let users access a large number of gesture shortcuts.

The three key innovative aspects of this project are:
- Formulation. We consider the problem of gestural interaction design as an optimization problem. We thus build a predictive model of performance and develop interactive optimization method to guarantee optimal interfaces and reduce the needs of experimental evaluations, which are expensive and time consuming;
- Approach. We primarily focus on command-gesture mappings rather than motor-control. In particular, we investigate the semantic relations between commands and how to represent them in the design space of gestures.
- Method. We transpose principles and methods from cognitive science and linguistics, “Elaborative encoding and “Simplicity Theory”, to optimize gestural interaction.

The outcomes of this project are:
- Scientific, with a fundamental understanding of command-gesture mappings;
- Technical, with the elaboration of a design tool to help designers to create efficient gestural interaction;
- Economical, by using our results to develop OuiHop, a startup in the area of transportation that will require advanced gestural interaction for Desktop and mobile devices.