Human4D: Acquisition, Analysis and Synthesis of Human Body Shape in Motion – Human4D

Human4D begins with building new datasets of captured shape of people under motion. In order to capture a large variety of moving shapes, we aim at acquiring hundreds of human body shapes under motion with diverse physical characteristics. Our dynamic shape capture sessions will be based on the Kinovis platform, a unique acquisition platform of 10m×8m area surrounded by 68 color cameras and 20 mocap cameras operating at INRIA Grenoble.
A first step toward efficient 4D modeling is to investigate proper mathematical representations for dynamic shapes. There are at least three possible approaches that we will investigate to push the limit of current representations that model shapes but seldom their dynamics, i.e. the evolution along time: One consists of 3D graphs that enable local dynamic properties to be captured and learned, following our recent success. Another is nonlinear manifold representation. The advantage of representing dynamic data on a non-linear manifold is a compact encoding of the constraint and distance measures that are in general superior to ones from Euclidean space. Furthermore, statistical modeling on manifold are in general better than in Euclidean space. Finally, spectral shape analysis, which relies on the study of the eigenvectors of specifically defined mesh operators, needs adaptation since the representation is domain-dependent.
Finally, we will develop prediction methods and generative models that, based on the above dataset and 4D human representations, are able to analyze, recover, and synthesize person- and motion-specific shape changes in the spatiotemporal domain, from incomplete/partial data. This will be implemented by deploying deep neural network architectures that are appropriate for learning time varying data, such as recurrent neural network.

The expected outcome of Human4D can be summarized as follows:
* A database of 4D human body shapes from multi-view sequences, composed of diverse individuals of different physical characteristics and exhibiting different actions.
* A full pipeline for the reconstruction and representation of highly precise 4D models of human.
* A compact, scalable, and readily available 4D human atlas based on the above database of 4D human models.
* An extension of Deep Learning techniques to 3D and 4D data.
* Representative applications demonstrating the efficiency of the above 4D human atlas with the recovery, the synthesis and the analysis of 4D human models.

The project is expected to make significant advances on both theoretical and applied aspects on shape representation, statistical 4D modeling, and learning. The construction of 4D atlas itself can have a great impact in several domains that deals with human motion observations, media contents and medical diagnosis, for instance. One expected outcome of the project, the ability to synthesize or control 4D human shapes under constraints, is very challenging as today, animating a real human model in situations different from the data acquisition still requires a lot of manual interactive works.
Numerous applications using the shape capture technology have already significantly impacted our society and have profoundly affected our handling of human shape data. For example, 3D range scans of several thousand individuals collected in the Civilian American and European Surface Anthropometry Resource project, combined with follow-up technologies developed for handling the collected data, has opened new types of retail business models such as Virtual try-on. The same evolution is likely to happen to 4D human shapes: in the future we will be able to make digital copies of moving persons using a handy imaging device, send them over the network, make customized compositions of digital 4D human shapes, etc. Human4D will participate to this evolution with objectives that can profoundly improve the creation, transmission, and reuse of digital human data. This will have direct impact in the recently growing Virtual- and Augmented Reality applications in health, sport, and edutainment sectors.

esults will be published by all partners in major conferences in both computer graphics and vision domains (e.g. Siggraph, Eurographics, CVPR, ICCV, 3DV) and main journals (e.g. ACM Transactions on Computer Graphics, IEEE PAMI, etc.). The project will follow an Open Science approach to dissemination and communication. Project-related publications will be uploaded to the French National repository HAL (UL, INT) as allowed by the French Digital Republic Law. Two international workshops will be organized to present scientific achievements to Human4D to scholars and industries, and to establish possible collaboration opportunities. Co-located workshops in cooperation with major conferences are considered, such as CGI, ICPR, ICCV, CVPR.

We also plan to allow free access to some of our dataset and solutions through project websites, which will certainly contribute to the increased visibility of the project among scientific communities. Notably, we will be vigilant so as to comply with the EU General Data Protection Regulation.

While most of the results will be made freely available, some of the outcomes will be registered as patents or licensed softwares. In each partner’s University, a structure is in charge of protecting the interests of the University and its researchers for what concerns intellectual property: consortium agreements, patents, copyrights, software, etc. We will discuss with our structure on how to distribute the code, and what kind of licence (CeCiLL, GPL, etc) to choose for it. We shall also examine the possibility to make a deposit of our sources to the APP. LIRIS partner has previous experience on integration of research results, particularly with the Arskan Start-Up, in partnership with the SATT Lyon St-Étienne and 1kubator (French incubators network).