Shared-Control Algorithms for Human/Multi-Robot Cooperation – MULTISHARED

One goal of MULTISHARED will be to study how to apply modern reactive trajectory planning approaches to the problems of decentralized formation control and localization for a multi-robot group, with the long-term aim of increasing the group autonomy and decision-making possibilities (e.g., better handling environmental constraints such as obstacles of multi-robot collisions, limited sensing/communication constraints, limited energy/actuation, and finally optimality w.r.t. any criterion of interest). Quantifying localization uncertainty with confidence (integrity) is also a key point for safe navigation. It enables to define a bounding volume of the actual robot position. In a multi-robot context, the relative localization uncertainty bounds can be leveraged to enforce a guaranteed minimum separation distance between the UAVs. When operating in an open environment, additional regulatory or mission-related flight rules may apply (no-fly zones, charted obstacles, terrain). They require global positioning of the UAV fleet in order to be enforced. Bounded-error approaches based on interval analysis and constraint propagation have successfully been used for the localization of marine, aerial and ground vehicles. We plan to study the use of interval-based decentralised approaches for localization in a fleet of UAVs. A important part of this research will be to provide robustness to erroneous measurements and to keep the required communication bandwidth low. Bounded-error Simultaneous Localization And Mapping (SLAM) approaches will also be studied in order to improve the global positioning accuracy.
Another goal of MULTISHARED will be to investigate the problem of intuitively teleoperating multiple robots of different nature, by finding novel theoretical approaches and scientific solutions that advance the state-of-the-art of shared-control of semi-autonomous teleoperation systems through machine learning. We will start by tackling innovative supervised learning approaches coming from our long experience in the robotics field, aiming at maximizing the similarity of action between the human body and the robotic team with respect to the chosen task. Another important objective will be to investigate the problem of conveying multiple pieces of feedback information in a comfortable and unobtrusive way, by advancing the state-of-the-art of multi-type and multi-point haptic rendering techniques through machine learning. We will start by studying the perceptual effect of providing distributed multiple haptic stimuli (e.g., skin stretch, vibrotactile, pressure) to learn the best way of providing the target sensations. Then, we will investigate supervised learning approaches able to map the many different information registered by the robot team to the human user, trying to match the actions of the slave robots and human hand with respect to the environment.

Robot trajectory optimization for persistent environmental monitoring (single-robot and multi-robot): we presented a control-theoretic approach for trajectory optimization of mobile robots suitable for environmental monitoring. Possible applications involve the localization of sources emitting a substance (e.g., a gas) or the estimation of chemicals, pollution, pesticides, etc., in the environment. The proposed method is based on the optimization of the Constructability Gramian to maximize the information collected while traversing a state trajectory. The maximization of the collected information is combined with energy constraints to define an optimization-based controller that achieves persistent environmental monitoring. This work can be the basis of several follow-ups: for example a human operator can also be part of the team (for example in search and rescue missions) and the robots need to continuously scan the environment for localizing sources of dangerous substances
Shared control of a heterogeneous human-robot team: we presented a decentralized connectivity-maintenance control framework for an heterogeneous human-robot team. The algorithm is able to manage a team composed of an arbitrary number of mobile robots (drones and ground robots in this case) and humans, for collaboratively achieving exploration and patrolling tasks. While the human explores the environment, the robots move so as to keep the team connected via a connectivity-maintenance algorithm; at the same time, each robot can also be assigned with a specific target to visit for exploring the unknown environment. We considered tasks of patrolling, exploration, and search-and-rescue. The human user was also provided with haptic feedback (also the in form of vibro-tactile cues) for improving her/his situational awareness and informing about the status of the robot group.
Perception-aware multi-robot path planning: we presented a decentralized and online optimal perception-aware strategy for multi-robot systems. The aim is to maximize the information collected along the planned trajectory about the relative configurations of the robots and, hence, to minimize the localization uncertainty. Thanks to a proper change of coordinates, we computed the Constructability Gramian, which can quantify the information about the future state of a nonlinear system, in a decentralized way with only minor approximations. This allows for formulating an online and decentralized trajectory generation problem for optimal localization.

In the next period we plan to explore more the VR component of the project and to organize and run comprehensive experiments with a human in control of a remote multi-robot group. In particular we will consider the implementation of the proposed control, planning, learning and haptic cueing machinery by exploiting a real team of UAVs flying in a real (e.g., cluttered) environment. In this scenario, we will address tasks of multi-robot surveillance, mapping, and exploration of dangerous and/or remote environments, combining decentralized topological motion control with our innovative MULTISHARED approach. We will run extensive human-subjects studies, followed by principled statistical analysis, to compare the proposed MULTISHARED approach with more classic solutions, such as sensory substitution via audio and visual stimuli. We will consider both simulated scenarios and real environments (e.g., UAVs flying in our drone room).

G. Notomista, C. Pacchierotti, P. Robuffo Giordano. Online Robot Trajectory Optimization for Persistent Environmental Monitoring. IEEE Control Systems Letters, 6, 1472-1477, 2021
M. Aggravi, G. Sirignano, P. Robuffo Giordano, and C. Pacchierotti. Decentralized Control of a Heterogeneous Human-Robot Team for Exploration and Patrolling. IEEE Transactions on Automation Science and Engineering, 2021
M. Aggravi, A. Elsherif, P. Robuffo Giordano, and C. Pacchierotti. Haptic-Enabled Decentralized Control of a Heterogeneous Human-Robot Team for Search and Rescue in Partially-known Environments. IEEE Robotics and Automation Letters, 6(3):4843–4850, 2021
G. Notomista, C. Pacchierotti, and P. Robuffo Giordano. Multi-Robot Persistent Environmental Monitoring Based on Constraint-Driven Execution of Learned Robot Tasks. In 2022 IEEE Int. Conf. on Robotics and Automation (ICRA 2022), 2022
N. De Carli, P. Salaris, and P. Robuffo Giordano. Online Decentralized Perception-Aware Path Planning for Multi-Robot Systems. In 2021 IEEE International Symposium on Multi-Robot and Multi-Agent Systems (MRS 2021), 2021

Research on multi-robot systems has flourished over the last decades with a number of theoretical and experimental results, based on the idea that proper coordination of many simple robots can lead to the fulfillment of arbitrarily complex tasks in a robust (to single robot failures) and highly flexible way. Autonomous search and rescue, firefighting, exploration and intervention in dangerous or inaccessible areas are some of the most promising multi-robot applications. An active research direction is that of decentralized formation control of multiple mobile robots based on only local (onboard) sensing and communication, with the aim of deploying highly autonomous robot teams in `non-trivial' environments (e.g., inside buildings, underwater, underground, or even in deep space) where centralized measuring/communication facilities (such as GPS) are not available. These effort are, therefore, aimed at increasing the autonomy and decision-making of a robot group for accomplishing missions in complex situations (e.g., outdoor in presence of obstacles, disturbances, limited sensing/communication, and so forth). At the same time, more and more attention is focused on the topic of human/multi-robot interfacing, i.e., how to interface a human operator with a team of multiple robot for sharing the load of autonomous decision-making and mission control. Nevertheless, it is truly challenging to design effective multi-robot teleoperation systems. First, the human operator should be able to single-handedly control the action of the robotic team in a natural and intuitive way. Second, the robotic team should be able to effectively and exhaustively provide the human with the large amount of feedback information coming from the remote environment. This topic is a very promising direction since human assistance is most often required for a successful completion of a mission for several reasons: (i) technological ones, as robot autonomy is still quite limited when needing to deal with uncertain and unstructured environments, and the human superior cognitive abilities are crucial for taking the right decisions assessing the situation, (ii) task-related ones, since in some missions the human operator needs to be interfaced with the robot team for taking part to the task itself (i.e., organizing a mapping/exploration mission for selecting areas of interest), (iii) safety ones, since in most cases the current legislation requires presence of a human operator for supervising a mission and taking responsibility of any unexpected outcome.


Building upon the consolidated experience of the applicant and his team in the topics of multi-robot coordination and decision-making, AI for robotics, shared control and human-robot interaction, the goal of MULTISHARED is to significantly advance the state-of-the-art in multi-robot autonomy and human-multi-robot interaction for allowing a human operator to intuitively control the coordinated motion of multi-UAV group navigating in remote environments, with a strong emphasis on the division of roles between multi-robot autonomy (in controlling its motion/configuration and online decision-making) and human intervention/guidance for providing high-level commands to the group while being most aware of the group status via VR and haptics technology.