Virtual Reality, Medical Assistance and Rescue for Spationauts – VR-MARS

Within the Lab-STICC, as part of the doctoral work, an thorough state of the art on the behavior of a medical coordinator and Situational Leadership was conducted.
In addition, we are developing the virtual agent by generating its non-verbal behavior following the SAIBA framework. In particular, work on the real-time generation of gestures is underway. These will be guided by invisible objects on which hands will come to rest. The shape and movement of these objects in the ECA relative space will define the movement of the arms, the shape of the hands and the movements of the fingers.
The architecture enabling collaboration between the augmented reality situation on the remote site and virtual reality in the control room is also under development. This architecture is based on a server which routes messages between the two situations while managing the latency simulation. The architecture is also based on a formalism of messages. This formalism is based on the MASCARET metamodel allowing the situation to be represented in the form of a model of the world and above all a model of the procedures to be carried out. The messages passing between the two situations (RV and RA) then take the form of property values, of action performed or of modification of procedure.
Within the LP3C (Université Bretagne Sud), 3 aspects of the VR-MARS project were tackled: the consolidation of the heuristic Cognitive Work Analysis (hCWA) method necessary for the analysis of patient management tasks, the participation in the definition of the experimental conditions for evaluation of the first prototype, and finally the interfacing between this method and the programming formalism of MASCARET virtual environments. The hCWA method aims to analyze critical adaptive systems, such as the treatment of a medical emergency, in order to specify the modalities of assistance.

The study carried out by the PhD student made it possible to propose a new taxonomy of non-technical skills, communicative intentions and the behavior of a virtual agent acting as a medical coordinator. Situational Leadership Theory is applied to allow the virtual agent to lead a group of people. Low-level behaviors (non-verbal cues) belonging to two different leadership styles, directive and supportive, are identified in the literature and are included in the taxonomy. High-level taxonomy information consists of the non-technical skills necessary for a medical coordinator, such as situational awareness, decision-making, task management and teamwork. The taxonomy makes it possible to link these non-technical skills to the communicative functions of the virtual coordinator and these latter to the verbal and non-verbal behavior that the agent must show.
In addition, a case study simulating abdominal pain on a patient was also carried out. The procedure and its context were formalized with the help of experts from Lorient hospital and a scene simulating a Martian habitat was carried out allowing collaboration between an RA caregiver (Hololens) and an RV expert (CAVE). The actions in progress in this task relate to the creation of an interface (in VR or in 2D on interactive table) allowing the expert to navigate, temporally, in the procedure and to modify it according to his diagnosis.
Within L3PC, the hCWA method has been applied to the management of a cardiac emergency. Currently, we have started to apply the method to the abdominal pain scenario which will serve as a test for the first model of the VR-MARS device. This scenario was pre-tested on board the TARA boat. Further mathematical formalization work, in the form of Markov chains, was tackled in collaboration with the LMBA.

As part of the thesis work, the next step will be to assess the proposed taxonomy. To this end, an first evaluation is being carried out to verify that the non-verbal behaviors linked to the agent's communicative intentions express a style of leadership that is recognized by the users.
The abdominal pain scenario, pre-tested on the TARA boat, will be the subject of a parabolic flight test next October. We are currently working on the interface between hCWA and the MASCARET language. The objective is to provide MASCARET with activity indicators to guide the activation of assistance methods according to the evolution of the treatment of the patient by the caregivers.

The work of the PhD student has been published in two papers. A first, submitted at the very beginning of the thesis, made it possible to present the work during the doctoral consortium of the conference on virtual intelligent agents, IVA 2019. More recently, the taxonomy was published in an article submitted to the international conference on human behavior, ICHBSA 2020 (Internation Conference on Human Behavior and Scientific Analysis).
The work carried out at the LP3C enabled the submission of two papers, one accepted, the other being proofread.

The VR-MARS project represents a support system for urgent healthcare delivery in isolated environments, based on virtual reality and embodied conversational agents (ECA). We hypothesize that these two technologies enable better situational awareness and care coordination between 3 parties: a care provider in an isolated location, a critically ill patient and the control centre on Earth. VR-MARS explore the scientific fields of emergency medicine, human factors and virtual reality.

The use case of VR-MARS will be related to space medicine, in particular emergency care during a manned spaceflight to Mars. During these missions, temporal isolation will add to physical isolation, because of delays in communication between the care provider (on Mars) and ground control (on Earth), which will preclude real-time telemedical support.

VR-MARS will be built around two simultaneous decision loops which will allow task assignment and synchronisation between the care provider, the ECA and ground control. The ECA will interact with the care provider via augmented reality. Upon request, it will deliver step-by-step guidance on medical protocols, using reassuring verbal tone and cues in order to mitigate the stress of the care providers. As soon as it is available, ground control on Earth will be made aware of the situation on Mars and of the procedures being undertaken by the care provider. This will improve situational awareness on the ground and enable the most optimal decision making in the mid- to long-term. In return, ground control will deliver its recommendation to the care provider via the ECA. Therefore, the ECA will represent the central hub of communication between the two sites.

VR-MARS will be tested on two medical scenarios involving a critically ill patient represented by a high-fidelity simulator. Technical and non-technical skills of the care provider will be assessed at two levels: immediate interactions between the care provider and the ECA (for urgent, life-saving decisions) and delayed interactions between the care provider and ground control (for mid- and long-term decisions).

With regards to research output and spinoffs, we anticipate that VR-MARS will improve medical care in remote environments, such as humanitarian missions, the combat environment, medical evacuations, expedition medicine, etc.