Virtual Maintenance Companion – VIMACO

The proposed approach is to develop a virtual maintenance companion able to analyze the operator's behavior to acquire knowledge on the maintenance operation and on his level of expertise to provide her/him with the necessary guidance in augmented reality. The principles of the proposed approach are: i) the analysis of movements allows to build the maintenance procedure, ii) the analysis of data from the observation of the operator allows the analysis of the level of expertise and the motor singularities of the operator, iii) the adaptive multimodal assistance in augmented reality allows to assist the operator in her/his task according to her/his own needs. The project is organized in 5 scientific and technical workpackages (WP). The first 3 WP correspond to the 3 principles stated above. WP 4 is dedicated to a proof of concept on an industrial use case and WP5 is dedicated to the technological integration of the bricks developed in WP1 and WP2 within an existing industrial software tool. The project will last 48 months. The necessary means are made up of human resources, equipment and operating costs. The expected results of the project are: i) scientifically, a contribution on the research of gesture sequences in a movement, a contribution of artificial intelligence techniques for the analysis of human behavior in maintenance operations, a contribution of multisensoriality in augmented reality to guide a technical gesture, ii) industrially, to better capitalize on the know-how of the maintenance operations, to promote a human-centered approach in industry, to improve the quality of products, to acquire advanced knowledge in the service of augmented reality. The VIMACO project has the originality to combine the disciplines of virtual reality, artificial intelligence, movement neuroscience, advanced technologies of augmented reality and industry to meet the challenges of assisted maintenance for the industry of the future.

The VIMACO project focuses on the operator's behavior during their maintenance task rather than on the behavior of the systems they operate on (tools, parts). It will also make industrial jobs more attractive thanks to its human-centered approach. The expected results of the project are augmented reality, AI, and behavior analysis methods and tools that allow adaptive and interactive assistance to the maintenance operator, taking into account their level of expertise. The expected scientific results are related to a better understanding of the human interaction process to propose methods for breaking down a movement into key phases in the context of maintenance, an analysis of the added value of AI techniques for analyzing human movement, and the added value of multisensory feedback in augmented reality to guide a maintenance gesture. The expected industrial results are time savings in design, assembly, and maintenance, facilitation of collaborative decision-making, improved operator learning, a refocusing of the industry on humans, a significant improvement in maintenance quality, and the provision of new advanced services associated with the products. Technological building blocks will eventually have to be integrated into existing software tools for assisting operators using augmented reality.

/

Public presentation during the Science Festival, October 2023. Demonstration showing the concept of measuring operator movement and displaying parameters that can guide an estimation of the level of expertise.

Presentation of the VIMACO project during the Reality Days at Safran's Molsheim site (March 2024). Given as an example within the SAFRAN group as an added value for operator assistance during maintenance.

Presentation during the XR Days organized by La Ruche Industrielle in Lyon in October 2022. Demonstrator shown and presentation during a roundtable discussion.

The VIMACO project aims to develop methods and tools for personalized augmented reality for industrial maintenance assistance. It is a collaborative research project (PRCE) involving the LiSPEN (engineering sciences in virtual immersion and artificial intelligence) and CAPS (cognitive sciences) laboratories and the companies DIOTA (SME in augmented reality technologies for industry) and SAFRAN (a major group in the aeronautics industry). Industrial maintenance is a field where human skills are precious and operations are complex. The scientific literature in the field of maintenance assistance by augmented reality reports works on the added value of augmented reality, the capture and analysis of human gesture as well as the analysis of maintenance procedure based on the system. Few works focus on the operator, his level of expertise and the information adapted to his task.
The VIMACO project aims to provide personalized assistance to the maintenance operator. The scientific issues identified are related to the analysis of the behavior in order to extract a skeleton of the maintenance procedure, the knowledge of the level of expertise and the motor singularities of the operator and the interactive guidance techniques in augmented reality. Three research questions will be addressed: how to extract a maintenance procedure by analyzing the movements of an expert operator? How to extract the level of expertise and the motor singularities of the operator in order to better guide her/him? What are the sensory feedbacks to bring to the operator according to the missing information just necessary?
The proposed approach is to develop a virtual maintenance companion able to analyze the operator's behavior to acquire knowledge on the maintenance operation and on his level of expertise to provide her/him with the necessary guidance in augmented reality. The principles of the proposed approach are: i) the analysis of movements allows to build the maintenance procedure, ii) the analysis of data from the observation of the operator allows the analysis of the level of expertise and the motor singularities of the operator, iii) the adaptive multimodal assistance in augmented reality allows to assist the operator in her/his task according to her/his own needs.
The project is organized in 5 scientific and technical workpackages (WP). The first 3 WP correspond to the 3 principles stated above. WP 4 is dedicated to a proof of concept on an industrial use case and WP5 is dedicated to the technological integration of the bricks developed in WP1 and WP2 within an existing industrial software tool. The project will last 48 months. The necessary means are made up of human resources, equipment and operating costs.
The expected results of the project are: i) scientifically, a contribution on the research of gesture sequences in a movement, a contribution of artificial intelligence techniques for the analysis of human behavior in maintenance operations, a contribution of multisensoriality in augmented reality to guide a technical gesture, ii) industrially, to better capitalize on the know-how of the maintenance operations, to promote a human-centered approach in industry, to improve the quality of products, to acquire advanced knowledge in the service of augmented reality.
The VIMACO project has the originality to combine the disciplines of virtual reality, artificial intelligence, movement neuroscience, advanced technologies of augmented reality and industry to meet the challenges of assisted maintenance for the industry of the future.