Robust deep learning in surgical robotics – RODEO

The RODEO project focuses on robotic-assisted spine surgery. The system perceives its environment through various sensors, RGB-D (depth) images and intraoperative CT scans, and it moves its articulated arms to perform the surgery. RODEO aims to significantly increase the automation of the process, particularly to relieve the surgeon's cognitive load from manipulation tasks and allow them to concentrate on the surgical procedure. To achieve this, the project proposes new generations of Artificial Intelligence (AI) that are more robust for surgical assistance. We introduce novel planning methods leveraging the latest advancements in language models (e.g., GPT) and multimodal models (e.g., CLIP). These methods are designed to be compatible with real-time constraints (computation, memory, annotations) and capable of grounding instructions in the robot's internal representations. We also propose the development of hybrid robotic controllers that utilize partial physical knowledge and learn the unknown residual terms. We will conduct a detailed study of dynamics decomposition, the ability to incorporate partial observations, and the design of controllers that are both accurate and efficient. To enhance decision reliability and the acceptability of AI by surgeons and patients, we propose uncertainty quantification (UQ) methods that unify various factors from the state of the art. These methods can address structured problems such as semantic image segmentation. We will employ these UQ methods to resolve ambiguities in human-robot interactions and make exploration more efficient in reinforcement learning. A medical demonstrator will be implemented to integrate these methodological advances in AI, allowing us to evaluate co-manipulation performance and assess the system's added value for the surgical team.