Seconds that matter: Managing First Impressions for a more Engaging Virtual Agent – IMPRESSIONS

From the user’s perspective, the interaction will consist of a natural face-to-face communication with the agent, i.e. by using speech and gestures for example. By observing the agent’s behavior, the user might (or might not) form impressions of warmth and competence of it as well as other impressions that we did not take into account.
The assessment of the impressions will be done via subjective questionnaires administered after the interaction, as well as with the real-time detection of physiological states and multimodal behavior exhibited by the user during the interaction. The correlation among these various sources of user’s response will provide a rich set of information that goes in input to the agent to create a continuous interaction loop.

The agent manages impressions of warmth and competence on the user by exhibiting nonverbal multimodal behavior and adapting this behavior to the detected user’s behavior and physiological state during the interaction.

Interactive human-agent platform where the agent manages user's fist impressions.

B. Biancardi, A. Cafaro, C. Pelachaud, «Could a Virtual Agent be Warm and Competent? Investigating User's Impressions of Agent's Non-Verbal Behaviours«, Workshop «Investigating Social Interactions with Artificial Agents«, Glasgow, November 2017.
B. Biancardi, «Towards a Computational Model for First Impressions Management«, Doctoral Consortium of 19th ACM International Conference on Multimodal Interaction, Glasgow, November 2017.
B. Biancardi, A. Cafaro, C. Pelachaud, «Analyzing First Impressions of Warmth and Competence from Observable Nonverbal Cues in Expert-Novice Interactions«, 19th ACM International Conference on Multimodal Interaction, Glasgow, November 2017.
B. Biancardi, A. Cafaro, C. Pelachaud, «Investigating the role of gestures, arms rest poses and smiling in first impressions of competence.«, at Virtual Social Interaction Workshop, Bielefeld, German, July 2017.
B. Biancardi, A. Cafaro and C. Pelachaud «Gérer les premières impressions de compétence et de chaleur à travers des indices non verbaux.'', In Quatorzièmes Rencontres des Jeunes Chercheurs en Intelligence Artificielle (RJCIA 2017), July 2017, Caen, France.
B. Biancardi, A. Cafaro, C. Pelachaud, «Investigating User's First Impressions of a Virtual Agent's Warmth and Competence Traits.'', in Workshop Affect, Compagnon Artificiel, Interaction (WACAI), June 2016

In any encounter the first moments are critical and the impressions that we form of others matter. These impressions tend to last and affect the interaction experience.

The goal of this project is to build an anthropomorphic virtual character (ECA - Embodied Conversational Agent) able to make the best possible first impression on a user, thus effectively engaging him or her in an interaction. This goal will be realized by building an affective loop which ties the behavior of the ECA to the actual emotional reactions of the user facing it in real-time. This will give rise to an ECA capable of managing its first impressions on users. We focus on the identification and modeling of the nonverbal behavior, towards exhibiting, managing and maintaining impressions of two important socio-cognitive dimensions in the first minutes of interaction with a user. These dimensions are warmth (i.e. being friendly, agreeable, engaging and approachable) and competence (i.e. appear skilled, knowledgeable on a given topic).

The IMPRESSIONS project has humanistic and computational components. The analysis and modeling of nonverbal communicative behavior is drawn from existing literature in sociology and psychology, as well as from new data gathered from controlled user studies. The computational component implements those models in a virtual agent, resulting in believable and effective social behaviors.

Furthermore, user’s behaviors and physiological signals will help the agent managing the desired impressions. The analysis of multimodal signals is valuable to assess the user’s affective states, to determine the quality of the interaction, and, ultimately, for assessing the impressions that the user has formed of the agent. Results undergo a thorough user evaluation.

The use case scenario for this research has immediate impact in both the research community and a wide public audience. We envisioned a museum as context for our scenario. The agent will be installed in the museum’s hall with the purpose of exchanging a dialogue with visitors about the museum’s exhibit. Therefore the agent must be seen as warm and competent in order to obtain the visitor’s interest, positive judgments, and to enhance their experience of the museum.

The project builds on results from the PIA “Avatar 1:1 (A1:1)" project , it includes a partnership between the Swiss Center for Affective Computing and the LTCI-CNRS, and it envisions the agent deployment in the Natural History Museum of Neuchatel in Switzerland.