Analyzing Large Scale Geometric Data Collections – AIGRETTE

We propose to structure our project around the following four main axes:

1. Aggregation of different existing geometric datasets and design of a new dataset containing heterogeneous data. This will include 3D shapes represented as meshes and point clouds but also volumetric data, graphs representing geometric relations and potentially point clouds in high-dimensions.
2. Design of novel learning techniques that can seamlessly integrate different geometric data representations and also creating novel representations that are best adapted for modern machine learning approaches.
3. Design of methods for injecting geometric information into machine learning techniques, including both standard geometric features (normals, curvature, topological invariants, etc.) but also geometric consistency measures, which promote consistency of predicted measures across individual objects, to help improve robustness and alleviate potential scarcity of labeled data.
4. Developing novel analysis mechanisms, capable of providing visual feedback for both explaining the results obtained by the learning methods and also for better leveraging human input.


Project expertise To complete these tasks, we propose to exploit the expertise of the PI in geometry processing, and visual data and shape analysis, as well as in machine learning for shape processing and synthesis. Indeed, the combination of geometric insights with machine learning methods will be a key distinguishing characteristic of this chair. Our main contention is that this combination will be essential in helping to overcome many of the current challenges, such as the limited amount of training data, its heterogeneity and the lack of visual feedback mechanisms. Therefore, we propose to structure our chair around resolving the challenges associated with each of these axes as outlined below.

1.The work undertaken by the PI, Maks Ovsjanikov, with different colleagues, within the context of the project ANR AI Chair AIGRETTE has resulted in 19 publications in top-level venues, including CVPR, NeurIPS, NeurIPS, ECCV, ICCV, Transactions on Graphics, Eurographics, Symposium on Geometry Processing, 3DV, and Computer Graphics Forum, among others

2. At least 10 of these research papers were done in collaboration with international colleagues.

3. Our work has received the Best Paper Award at the International Conference on 3D Vision 2021 (3DV), as well as the Best Student Paper Award at 3DV 2020.

4. Our work has been covered extensively in both national scientific popular science journals, such as an article in the CNRS News Art and archaeology are the new frontiers for AI, as well as in international publcitations, such as Forbes Science, New Scientist, an article on Medium, among others.

5. Thanks to the funding of the project, we were able to hire 2 PhD students and 2 Postdoctoral researchers. Remarkably, all 4 of these are international, thus significantly strengthening the outreach and diversity of our group. Furthermore, we also had 3 international Master’s-level research interns, one of which continued to do a PhD in our group.

6. The PI has given 11 invited talks both online, e.g., TU Munich AI Seminar 2021, Banff Workshop on Geometry & Learning from Data 2021, London Geometry and Machine Learning Summer School 2021, etc., and in-person, e.g., in the 2022 Curves and Surfaces conference.

The project has already led to several breakthrough publications, some of which have been covered in popular media and several of which have had strong scientific impact.

We are currently working on extending these proposed methods to handle different data modalities such as noisy point clouds and non-isometric shape pairs, including man-made rather than organic shapes. We are also investigating more accurate methods for partial shape correspondences, and localizing complex objects inside large scenes instead of computing correspondences between two given objects. More broadly, we are studying ways of injecting semantic information that might exist within large 3D data collections.

All of the publications of the PI, associated with this project (which started in September 2020) are listed on the publications part of the PI’s website, and can be found at the following URL:

www.lix.polytechnique.fr/~maks/publications.html

We propose to develop efficient algorithms and mathematical tools for analyzing large scale collections of 3D shapes. Our project is motivated by the fact that large annotated collections of 3D models have recently become available, and that machine learning algorithms applied to such collections have shown promising initial results, for both shape analysis and synthesis. We believe that these existing techniques can be significantly extended by building on recent advances in optimization and learning, and adapting them to handle geometric 3D data directly. Moreover we propose to develop novel powerful visual feedback mechanisms that could help harness user input, and also provide better analysis and exploration tools of variability in the data.

We propose to concentrate on the following fundamental shape analysis tasks: shape segmentation, shape labeling, computing correspondences between shapes, estimating geometric quantities such as curvature, and shape space exploration. Our goal is to build on recent advances in machine learning and we describe a research plan based on using deep learning to address these tasks. To exploit the full potential of the power of deep learning we will both build on shape data sets that have recently become available and we describe a plan to build our own 3D shape data set. A key aspect in our work will be to develop novel representations of 3D data that will be better suited for machine learning, than the commonly used representations such as triangle soups or point clouds.

Since the complexity of most analysis tasks is high, a combination of deep learning with different techniques will be required. This is both because the scale of 3D data sets is significantly smaller than that of natural images, and because they place additional requirements such as invariance to rigid motions for example. Therefore, we describe several ideas to alleviate these problems, first via information transfer with functional maps, and then by ensuring consistency of computed segmentations, and using joint learning with images and 3D shapes. We also propose interaction techniques to generate new training data, inspect the output of automatic learning algorithms, and to explore shape collections via shape space exploration.