See, Touch and Manipulate: Robot Learning for Dexterous Robot Bimanual Manipulation through Vision and Touch – Aistotle

Dexterous manipulation has been a long-standing challenge in AI and robotics. Already Aristotle noted that the hand is the “tool of tools”, and Anaxagoras held that “man is the most intelligent of the animals because he has hands”. Thus, intelligence has long been understood to come together with dexterity, and the lack thereof in artificial systems explains why current robots are mostly limited to pre-programmed tasks in structured environments. In this project named after Aristotle, we aim to develop a new generation of dexterous robotic manipulation systems endowed with a pair of three-fingered hands, human-like vision and tactile sensing, and more importantly, the capacity for never-ending learning and inference to deal with ever more complex manipulation tasks. To achieve this goal, the Aristotle project requires the following components of increasing level of complexity: representation learning for tactile-driven finger control for in-hand manipulation, visuotactile hand-eye calibration fusing tactile and visual sensing modalities, dual-arm coordination that combines representations from the left and right hand, and finally, never-ending lifelong learning to capitalize on the previously learned manipulation skills. As such, the Aristotle project challenges the current 2nd-wave AI paradigm centered on learning from static human-curated datasets, and asserts itself as a 3rd-wave AI initiative by targeting human-level robotic dexterity. The Aristotle project stands out from traditional research proposals thanks to a deep and harmonious 40-year-long partnership between Ecole Centrale de Lyon (ECL) and Technische Universität Darmstadt (TUDa), conducive to forming a joint French-German AI research lab between two world-class AI research teams with complementary expertise in computer vision and robotics, thereby establishing an open platform for collaboration between other AI faculty at both institutions and further European partners. For this purpose, the Aristotle project proposes two co-supervised PhD theses and an annual 4-day open workshop, fostering scientific exchange and Europe-wide collaboration among AI researchers.