Human4D: Acquisition, Analysis and Synthesis of Human Body Shape in Motion – Human4D
Human4D begins with building new datasets of captured shape of people under motion. In order to capture a large variety of moving shapes, we aim at acquiring hundreds of human body shapes under motion with diverse physical characteristics. Our dynamic shape capture sessions will be based on the Kinovis platform, a unique acquisition platform of 10m×8m area surrounded by 68 color cameras and 20 mocap cameras operating at INRIA Grenoble.
A first step toward efficient 4D modeling is to investigate proper mathematical representations for dynamic shapes. There are at least three possible approaches that we will investigate to push the limit of current representations that model shapes but seldom their dynamics, i.e. the evolution along time: One consists of 3D graphs that enable local dynamic properties to be captured and learned, following our recent success. Another is nonlinear manifold representation. The advantage of representing dynamic data on a non-linear manifold is a compact encoding of the constraint and distance measures that are in general superior to ones from Euclidean space. Furthermore, statistical modeling on manifold are in general better than in Euclidean space. Finally, spectral shape analysis, which relies on the study of the eigenvectors of specifically defined mesh operators, needs adaptation since the representation is domain-dependent.
Finally, we will develop prediction methods and generative models that, based on the above dataset and 4D human representations, are able to analyze, recover, and synthesize person- and motion-specific shape changes in the spatiotemporal domain, from incomplete/partial data. This will be implemented by deploying deep neural network architectures that are appropriate for learning time varying data, such as recurrent neural network.
The expected outcome of Human4D can be summarized as follows:
* A database of 4D human body shapes from multi-view sequences, composed of diverse individuals of different physical characteristics and exhibiting different actions.
* A full pipeline for the reconstruction and representation of highly precise 4D models of human.
* A compact, scalable, and readily available 4D human atlas based on the above database of 4D human models.
* An extension of Deep Learning techniques to 3D and 4D data.
* Representative applications demonstrating the efficiency of the above 4D human atlas with the recovery, the synthesis and the analysis of 4D human models.
The project is expected to make significant advances on both theoretical and applied aspects on shape representation, statistical 4D modeling, and learning. The construction of 4D atlas itself can have a great impact in several domains that deals with human motion observations, media contents and medical diagnosis, for instance. One expected outcome of the project, the ability to synthesize or control 4D human shapes under constraints, is very challenging as today, animating a real human model in situations different from the data acquisition still requires a lot of manual interactive works.
Numerous applications using the shape capture technology have already significantly impacted our society and have profoundly affected our handling of human shape data. For example, 3D range scans of several thousand individuals collected in the Civilian American and European Surface Anthropometry Resource project, combined with follow-up technologies developed for handling the collected data, has opened new types of retail business models such as Virtual try-on. The same evolution is likely to happen to 4D human shapes: in the future we will be able to make digital copies of moving persons using a handy imaging device, send them over the network, make customized compositions of digital 4D human shapes, etc. Human4D will participate to this evolution with objectives that can profoundly improve the creation, transmission, and reuse of digital human data. This will have direct impact in the recently growing Virtual- and Augmented Reality applications in health, sport, and edutainment sectors.
esults will be published by all partners in major conferences in both computer graphics and vision domains (e.g. Siggraph, Eurographics, CVPR, ICCV, 3DV) and main journals (e.g. ACM Transactions on Computer Graphics, IEEE PAMI, etc.). The project will follow an Open Science approach to dissemination and communication. Project-related publications will be uploaded to the French National repository HAL (UL, INT) as allowed by the French Digital Republic Law. Two international workshops will be organized to present scientific achievements to Human4D to scholars and industries, and to establish possible collaboration opportunities. Co-located workshops in cooperation with major conferences are considered, such as CGI, ICPR, ICCV, CVPR.
We also plan to allow free access to some of our dataset and solutions through project websites, which will certainly contribute to the increased visibility of the project among scientific communities. Notably, we will be vigilant so as to comply with the EU General Data Protection Regulation.
While most of the results will be made freely available, some of the outcomes will be registered as patents or licensed softwares. In each partner’s University, a structure is in charge of protecting the interests of the University and its researchers for what concerns intellectual property: consortium agreements, patents, copyrights, software, etc. We will discuss with our structure on how to distribute the code, and what kind of licence (CeCiLL, GPL, etc) to choose for it. We shall also examine the possibility to make a deposit of our sources to the APP. LIRIS partner has previous experience on integration of research results, particularly with the Arskan Start-Up, in partnership with the SATT Lyon St-Étienne and 1kubator (French incubators network).
Reconstructing, characterizing, and understanding the shape and motion of individuals or groups of people have many important applications, such as ergonomic design of products, rapid reconstruction of realistic human models for virtual worlds, and an early detection of abnormality in predictive clinical analysis. Naturally, the capture and analysis of people’s shape and motion have a long tradition in disciplines such as computer vision, computer graphics, and virtual reality. This is evidenced by the large amount of research done on shape reconstruction from images and 3D scans, on subspace construction with multiple shapes, on motion capture and action recognition from video inputs. However, most current techniques treat shape and motion independently, with devoted techniques for either shape or motion in isolation. This is largely due to the difficulty of acquiring proper observations on full moving shapes: Traditional systems have been devoted to capture either static shapes, e.g. 3D scanners, or motion only, e.g. motion capture.
Recent evolutions in the technology for capturing moving shapes have changed this paradigm with new multi-view acquisition systems that enable now full 4D models of human shapes including geometry, motion and appearance, as in Microsoft, Inria, MPI, or more recently with commercial platforms deployed by Intel, 8i or Microsoft, among others. Such data open new possibilities and challenges for the analysis and the synthesis of human shapes in motion that are yet largely unexplored but would be of benefit to a wide range of applications in virtual and augmented reality, or in the sport and medical domains, among others. This is especially true with the rapidly growing VR/AR immersive applications based on head mounted displays, which require realistic and detailed models to improve the immersive experience. Magic leap, Microsoft Hololense, Facebook Oculus Rift, Sony PS4 HMD and the HTC Vive, among others, are clear examples of this recent and rapid evolution and the associated need to produce adapted realistic contents. In the future we will be able to make digital copies of moving persons using a handy imaging device, send them over the network, make customized compositions of digital 4D human shapes, etc. Human4D will participate to this evolution with objectives that can profoundly improve the reconstruction, transmission, and reuse of digital human data, by unlocking the limited use of deep learning techniques in human shape modeling.
The Human4D project aims at investigating some of these issues with a particular emphasis on how to represent a collection of 4D data in an efficient and compact way, as required by advanced applications such as statistical analysis or motion synthesis. Our ambition is to go beyond existing shape space representations that mostly focus on static shape poses, and seldom consider the continuous dynamic of shapes. Although all these ideas exist already and are not novel per say, no existing work has achieved a similar goal. Indeed, the complexity and persistent change in geometry and topology of such time-varying data make most traditional shape analysis algorithms unsuitable. This project is timely, as shown by important ongoing research activities in this field in both the academia and the industry (as mentioned previously). It gathers French research teams with long-standing expertise in the field, associated with one of a few acquisition platforms available worldwide that can produce the required data on moving human bodies.
Madame Hyewon SEO (Laboratoire des sciences de l'Ingénieur, de l'Informatique et de l'Imagerie (UMR 7357))
The author of this summary is the project coordinator, who is responsible for the content of this summary. The ANR declines any responsibility as for its contents.
ICube Laboratoire des sciences de l'Ingénieur, de l'Informatique et de l'Imagerie (UMR 7357)
Inria GRA Centre de Recherche Inria Grenoble - Rhône-Alpes
CRIStAL Centre de Recherche en Informatique, Signal et Automatique de Lille
LIRIS - CNRS UMR 5205 - LABORATOIRE D'INFORMATIQUE EN IMAGE ET SYSTEMES D'INFORMATION
Help of the ANR 694,768 euros
Beginning and duration of the scientific project: September 2019 - 48 Months