Virtual humanoids learning motion skills – HUSKI

Instead of following a benchmark approach that would consider humanoid motion as just another example of problem that can be tackled by reinforcement learning, we propose to concentrate our work on the particularities of humanoid motion, and on how specialized learning algorithms can take advantage of them. The project focuses in particular on three aspects of reinforcement learning: exploration, representation learning and modularity.

At the end of the project, the developed algorithms will be made available in an open-source tool.

The main perspective of the scientific results of the project is to advance the fields of reinforcement learning, humanoid robotics and virtual character animation.

1. Thomas Pierrot, Guillaume Ligner, Scott Reed, Olivier Sigaud, Nicolas Perrin, Alexandre Laterre, David Kas, Karim Beguir, Nando De Freitas, Learning Compositional Neural Programs with Recursive Tree Search and Planning, NeurIPS 2019.
2. Astrid Merckling, Alexandre Coninx, Loic Cressot, Stephane Doncieux, Nicolas Perrin, State Representation Learning from Demonstration, LOD 2020.
3. Matheron et al., Understanding Failures of Deterministic Actor-Critic with Continuous Action Spaces and Sparse Rewards, ICANN 2020.
4. Guillaume Matheron, Nicolas Perrin, Olivier Sigaud, PBCS: Efficient Exploration and Exploitation Using a Synergy Between Reinforcement Learning and Motion Planning, ICANN 2020.
5. Alexandre Chenu, Nicolas Perrin-Gilbert, Stéphane Doncieux, Olivier Sigaud, Selection-Expansion: A Unifying Framework for Motion-Planning and Diversity Search Algorithms, ICANN 2021.
6. Thomas Pierrot, Nicolas Perrin-Gilbert, Olivier Sigaud, First-Order and Second-Order Variants of the Gradient Descent in a Unified Framework, ICANN 2021.

Human motion has several particularities that make it difficult to reproduce with algorithms. It relies on numerous heuristics that combine well and are mobilized and adjusted with great flexibility. For the control of humanoid robots, powerful techniques exist, mainly based on links between classical mechanics, kinematics, linear algebra and optimization. Yet, these approaches (which we refer to as "model-based") do not permit to generate complex autonomous behaviors, because many components of the movements must be described by precise tasks. Recently, machine learning has lead to interesting results without exploiting model-based methods. The objective of this project is to go further by developing specific machine learning techniques that share some aspects of the model-based methods and permit to automatically construct repertoires of motion skills with virtual humanoid robots.