Chess Expertise from Eye Gaze and Emotion – CEEGE

We will observe the visual attention, physiological responses and mental states of subject with different levels of expertise solving classic chess problems, and participating in chess matches. We will observe chess players using eye-tracking, sustained and instantaneous face-expressions (micro-expressions), skin conductivity, blood flow (BVP), respiration, posture and other information extracted from audio-visual recordings and sensor readings of players. We will use the recorded information to estimate the mental constructs with which the players understand the game situation. Information from visual attention as well as physiological reactions will be used to determine and model the degree to which a player understands the game situation in terms of abstract configurations of chess pieces. This will provided a structured environment that we will use for experimental evaluation of current theories of mental modeling and emotional response during problem solving and social interaction.

We expect to obtain a better understanding of how to model the mental processes of subjects engaged in solving problems in critical situations.
Possible applications include construction of systems that can monitor the cognitive abilities and emotional reactions of users of interactive systems to provide assistance that is appropriate but not excessive, companion systems that can aid with active healthy aging, and tutoring systems that can assist users in developing skills in a variety of domains including chess.

Project that opens a new approach to human computer interaction in critical situations.

T. Guntz, R. Balzarini, C. Vaufreydaz,J.L. Crowley, Multimodal Observation and Classification of People Engaged in Problem Solving: Application to Chess Players, Multimodal Technologies Interact. 2018, 2(2), 11; doi.org/10.3390/mti2020011.

T. Guntz, C. Vaufreydaz, R. Balzarini, J.L. Crowley, Multimodal Observation and Interpretation of Subjects Engaged in Problem Solving, Workshop on Behavior, Emotion and Representation at the conference on Human-Agent Interaction, HAI-2017, Bielefeld Oct. 2017

CEEGE is a multidisciplinary scientific research project. The aim of this project is to experimentally evaluate and compare current theories for mental modeling for problem solving and attention, as well as to refine and evaluate techniques for observing the physiological reactions of humans to situations that inspire pleasure, anguish, arousal, dominance and fear.

We will observe the visual attention, physiological responses and mental states of subject with different levels of expertise solving classic chess problems, and participating in chess matches. We will observe chess players using eye-tracking, sustained and instantaneous face-expressions (micro-expressions), skin conductivity, blood flow (BVP), respiration, posture and other information extracted from audio-visual recordings and sensor readings of players. We will use the recorded information to estimate the mental constructs with which the players understand the game situation. Information from visual attention as well as physiological reactions will be used to determine and model the degree to which a player understands the game situation in terms of abstract configurations of chess pieces. This will provide a structured environment that we will use for experimental evaluation of current theories of mental modeling and emotional response during problem solving and social interaction.

The project is organized in three phases. During the first phase, we will observe individual players of different levels of chess expertise solving known classic chess problems. We will correlate scan-path from eye tracking and other information about visual attention to established configurations of pieces and known solutions to chess problems. This will allow us to construct a labeled corpus of chess play that can be used to evaluate competing techniques for estimating mental models and physiological responses. In a second phase, we will observe the attention and face expressions of pairs of players of different levels of chess ability during game play. In particular, we will seek to segment and annotate recordings with respect to the difficulty of the game situation as well as situations which elicit particularly strong physiological reactions. In the final phase, we will use these recordings to evaluate the effectiveness of competing techniques for mental modeling and observation of emotions in terms of their abilities to predict the chess abilities of players, game outcomes and individual moves and player self reports.

Results of our work will be published in scientific conferences and journals concerned with cognitive science and cognitive neuroscience as well as computer vision, multi-modal interaction, affective computing and pervasive computing. Possible applications include construction of systems that can monitor the cognitive abilities and emotional reactions of users of interactive systems to provide assistance that is appropriate but not excessive, companion systems that can aid with active healthy aging, and tutoring systems that can assist users in developing skills in a variety of domains including chess.