Mathematics of Stochastic and Deterministic Optimization for Deep Learning – MaSDOL

The project will be structured as follows:
• Part I - Fundamentals of Machine Learning
Task 1 – Acceleration methods, beyond convex problems. Coordinator: C. Dossal
Rates of inertial algorithms, Robustness, Link with dynamical systems, Non-asymptotic analysis, Infinite dimensional situation

Task 2 - Stochastic algorithms and sampling with errors. Coordinator: G. Fort
Untractable target function and stochastic settings.Multimodal problems. Non-convexity. Applications in statistics. Sampling.

Task 3 – Optimal transport, optimization and statistics. Coordinator: J. Bigot
Geometry and Optimal Transport, Regularization, Stochastic methods

• Part II - Theory of Deep Learning
Task 4 - Optimization methods for deep learning. Coordinator: E. Pauwels
New dedicated algorithms, Adaptivity of deep network training algorithms, Sensitivity and generalization,
Task 5 - GAN: game theory and OT. Coordinator: J. Renault
Development and use of robust optimization methods with complex geometry, GANs, Deep Learning, and computational optimal transport, Nonlinear dynamics for training GANs, GANs and game theory algorithms
Task 6 - Stability and generalization for interacting systems and deep N.N. Coordinator: S. Gadat
Stability of AI interacting systems, Stability of GANs, sensitivity and non-uniform sampling for training deep networks,

• Part III - Numerical experiments and applications
Task 7 - Numerical experiments and applications. Coordinator: M. Serrurier.
Image segmentation with GAN, Data augmentation with GAN for supervised learning, Bio-informatics, Data processing on graphs

The consortium brings together a unique mix of researchers in statistics, optimization, machine learning and game theory. This unique mix of these areas of mathematics is expected to achieve significant breakthroughs in the theory of deep learning. Members of MaSDOL are of the highest academic quality (members of IUF, senior CNRS position, board of top international journals in their fields) and are already collaborating with each other showing the existing important synergy between TSE & IMT, TSE & IMB and IMT & IMB.
These 3 PhD theses will pool the competences between the 3 research labs and will make the cooperation concrete: each research center will globally benefits from 1 Phd grant all along the project. We will organize two major events all along the duration of the project: an intermediate workshop in Bordeaux during the 2nd/3rd year of the project and a final international one in Toulouse at the end of the project. These workshops will not be only focused on the contributions brought by the project members, but they will also emphasize on new contributing researchers in the field.

If the project aims to open ambitious breakthroughs in machine learning, it is
also grounded in past and current collaborations between its members. Among them, we emphasize on the existing works:
• Aujol, Dossal and Rondepierre are currently working together on accelerated optimization algorithms and their link with dynamical systems.
• Bigot, Bercu and Gadat have shared strong interactions in the past years and they are working on stochastic algorithms for optimal transport applications.
• Bolte, Fort, and Gadat started discussions around stochastic algorithms with KL inequalities.
• Fort and Dossal are also currently working on related topics.
• Bolte and Gadat are working on an infinite dimensional version of acceleration of gradient dynamics and they aim to apply then to sampling in the Wasserstein space.
• Pauwels has a long-standing research collaboration with Bolte around optimization issues and he is now also working with Serrurier on extra-gradient dynamics for training GANs (supervision of a PhD student).

Publication lists
2020
• Renault, J. & Ziliotto, B. Limit equilibrium payoffs in stochastic games, Mathematics of Operation Research – 45(3), 2020.
• Bolte, J. & Pauwels, E. A mathematical model for automatic differentiation in machine learning, Advances in Neural Information Processing Systems , 2020, Spotlight Presentation.
• Chen, T. & Lasserre, J.B. & Magron, V. & Pauwels, E. Semialgebraic optimization for Lipschitz constants of ReLU networks. Advances in Neural Information Processing Systems (2020).
• Pauwels, E. & Putinar, M. & Lasserre, J.B. Data analysis from empirical moments and the Christoffel function. Foundations of Computational Mathematics (2020).
• Buckdhan, R. & Renault, J. & Quincampoix, M. Representation formulas for limit values of long run stochastic optimal controls. Siam Journal on Control and Optimization – 58(4), 2020.
• Laraki, R. & Renault, J. Acyclic gambling games, Mathematics of Operation Research – 45(4), 2020.
• Renault, J. & Ziliotto, B. Hidden stochastic games and limit equilibrium, Games and Economics Behavior - (124), 2020.
• De Castro, Y. & Gadat, S. & Marteau, C. & Maugis, C. SuperMix: Sparse regularization for mixture, Annals of Statistics, 2020.
• Sebbouh, O. & Dossal, C. & Rondepierre, A. Convergence rates of damped inertial dynamics under geometric conditions, SIAM Journal on Optimization - 30(3), 2020.
• Bolte, J. & Pauwels, E. Conservative set valued fields, automatic differentiation, stochastic gradient method and deep learning. Mathematical Programming, 2020.

Machine learning and artificial intelligence are rising themes of research because they have been considered as one way to produce new methods for solving striking challenges in language understanding, advice finding, signal processing, fraud detection.
The explosion of datascientist jobs is certainly an evidence of the societal, economic and scientific impact of artificial intelligence. The cornerstone of machine learning methods is the use of applied mathematics and in particular, statistics and optimization.

These two fields of research allow to handle both the randomness of the data and its high dimensional features. Machine learning then involves the
computation of hidden parameters for a system designed to make decisions for yet unseen data.

The MaSDOL project aims at developping new deterministic and stochastic
methods for solving optimization tasks from data with a possibly complex geometry. We will address accelerated optimization algorithms, stochastic algorithms and sampling with errors, optimization linked to game theory and GANs and optimal transport for deep learning.

A particular emphasis is made on the analysis of deep learning approaches for generative models (GANs) and adversarial learning. In this setting, we aim to develop novel numerical methods for machine learning and to study their mathematical properties. Another expected output of the project is to implement these algorithms on challenging issues in various domains such as image analysis, bio-informatics or data processing on graphs.