adaPtive Artificial iNtelligence fOR humAn coMputer interAction – PANORAMA

The key concept of PANORAMA is “user adaptive AI in the context of human-computer interaction”. First, we will conduct research on user adaptivity of Artificial Intelligence embodied as a conversational agent. When people talk to other people, they change their verbal and nonverbal communication behaviors continuously according to those of the partner. Therefore, user adaptivity is an essential issue in improving human-agent interaction. Communication style is also different depending on the culture, and adapting the agent behaviors to a target culture is required in system localization. PANORAMA will tackle these problems with Machine Learning. However, a bottleneck of this approach is that annotating users’ nonverbal behaviors to create training data is time consuming. We will solve this problem by exploiting Explainable Artificial Intelligence (XAI) technique, through which labels predicted by the system are adapted based on the interaction with the user as an annotator. Thus, user adaptive AI enables to support users in creating multimodal corpus as well as improve human-agent interaction. Moreover, user adaptivity is considered in PANORAMA via psychological theories, in which user motivation will be investigated in one relevant use case (personalised motivational coaching for physical activity). Therefore, PANORAMA envisions a new research methodology for Machine-Learning-based conversational agents by focusing on the concept of user adaptivity.

PANORAMA aims to accomplish the following five research goals. (1) propose a user adaptive multimodal annotation tool based on XAI techniques, (2) exploit this tool to collect annotated multimodal corpora in three countries (France, Germany, and Japan), (3) propose models and methods for developing conversational agents with multi-level adaptation functionality, where nonverbal signals of the agent as well as the content of the dialogue are adapted to the user, (4) provide multitask learning and transfer learning techniques to learn models using the multi-cultural corpus obtained in (2) and adapt the conversational agent to each culture, and (5) propose the design basis of adaptive AI systems grounded in psychological theories and evaluation studies.

This project provides technological and psychological foundations for user adaptive and culture-sensitive AI systems applied to the design of advanced Embodied Conversational Agents (ECA).

To reach these 5 research goals, we will focus on the following scientific objectives:
- provide the foundations for cross-cultural corpora creation and learning
- provide methods for creating personal and cultural adaptivity in ECAs
- provide methods for adapting ECA’s behavior to the interaction context and partners
- provide the psychological theories of interindividual differences as a basis for designing and evaluating user adaptive agents (consistent consideration of personality and culture)

The technical working objectives are:
- reduce efforts required for data collection and annotation by leveraging cooperative machine learning, data augmentation and multimodal transfer learning;
- learn models for continous user adaptive interaction

PANORAMA will contribute to research on developing autonomous systems that allow adaptive high-level interactions with users by focusing on multimodal processing including facial expressions, gestures, postures, gaze, and head nods in addition to speech and language information.
It will contribute to human-centered AI with a user-in-the-loop approach for designing a semi-automated annotation tool and developing cultural and personal adaptive conversational agents.

PANORAMA’s use case will help understanding how to interface AI with users to effectively motivate them to adopt a healthy lifestyle in the long term. Cross cultural studies as the ones conducted in PANORAMA will help understand the differences between cultures in terms of approach to healthy behaviors and interactions with AI systems.