Tractable quasI-Newton non-smooth Optimization Methods for large scale Data Science – TRINOM-DS

The overall objective of the TRINOM-DS project was to overcome these scientific obstacles and develop theoretical, numerical, and algorithmic advances to exploit the great potential of quasi-Newton schemes for large-scale non-smooth optimization problems, which are ubiquitous in data science. We combined advanced tools from optimization, variational and nonsmooth analysis, stochastic calculus, differential geometry, and dynamical systems. We have systematically developed an approach where we first studied continuous-time dynamics and analyzed their properties, before developing the corresponding algorithms seen as appropriate discretization them. We have developed extensive convergence guarantees for each developed algorithm including when quasi-Newton metrics are used.

We have developed quasi-Newton algorithms in several difficult settings, including non-convex, non-smooth, stochastic, and non-Hilbertian settings. We were thus able to prove many difficult convergence results for these algorithms, thereby solving open problems/conjectures. We have also developed a new approach on stochastic second-order in time and space dynamics for convex optimization from the SDE (stochastic differential equation) perspective, offering new and unprecedented insights.
Our work not only had theoretical results but has had also impact on several applications such as machine learning (deep learning, optimal transport), signal/image processing, and computer vision. This project offered a unique opportunity to develop a tangible, scalable, and digital embodiment of higher-order non-smooth optimization methods.

The execution schedule of the project has been adjusted twice because of the pandemic situation at the time it was accepted. Despite this, we were able to recruit excellent students and to achieve all our scientific objectives. The TRINOM-DS project has been extremely productive, with articles published in leading journals and conference proceedings in the fields of optimization and machine learning. Our algorithms have been implemented in the form of publicly downloadable libraries, in line with the philosophy of reproducible research. This project also served as a springboard for an ERC starting grant application (rated A but not funded).

PARUES
1. Rodrigo Maulen-Soto, Jalal Fadili, Hedy Attouch, Peter Ochs. An SDE Perspective on Stochastic Inertial Gradient Dynamics with Time-Dependent Viscosity and Hessian-Driven Damping. Optimization, 2025. ?hal-04637159v2?.
2. Michael Sucker, Jalal Fadili, Peter Ochs, Learning-to-Optimize with PAC-Bayesian Guarantees: Theoretical Considerations and Practical Implementation, Journal of Machine Learning Research, 2024. arXiv:2404.03290.
3. Hamza Ennaji, Jalal M. Fadili, Hedy Attouch. Stochastic Monotone Inclusion with Closed Loop Distributions. Evolution Equations and Control Theory, 2025. ?hal-04974921?.
4. Rodrigo Maulen-Soto, Jalal Fadili, Hedy Attouch, Peter Ochs. Stochastic Inertial Dynamics Via Time Scaling and Averaging. Stochastic Systems, 2025. ?hal-04520106v2?.
5. Shida Wang, Jalal M. Fadili, Peter Ochs. Quasi-Newton Methods for Monotone Inclusions: Efficient Resolvent Calculus and Primal-Dual Algorithms. SIAM Journal on Imaging Sciences, 2024. ?hal-04506477?.
6. Rodrigo Maulen-Soto, Jalal M. Fadili, Hedy Attouch. Tikhonov Regularization for Stochastic Non-Smooth Convex Optimization in Hilbert Spaces. Open Journal of Mathematical Optimization, 2025. ?hal-04506500v3?.
7. Rodrigo Maulén, Jalal M. Fadili, Hedy Attouch. An SDE Perspective on Stochastic Convex Optimization. Mathematics of Operations Research, 2024. ?hal-04789071?.
8. Tejas Natu, Camille Castera, Jalal Fadili, Peter Ochs. Accelerated Gradient Dynamics on Riemannian Manifolds: Faster Rate and Trajectory Convergence. 26th International Symposium on
Mathematical Theory of Networks and Systems (MNTS), 2024. ?hal-04337117?.
9. Camille Castera, Hedy Attouch, Jalal Fadili, Peter Ochs. Continuous Newton-like Methods featuring Inertia and Variable Mass. SIAM J. on Optimization. 2023. ?hal-03951620?.
10. Shida Wang, Jalal Fadili, Peter Ochs. A Quasi-Newton Primal-Dual Algorithm with Line Search. International Conference on Scale Space and Variational Methods in Computer Vision (SSVM). Lecture Notes in Computer Science, Springer, 2023.
11. Rodrigo Maulen-Soto, Jalal Fadili, Hedy Attouch. Stochastic convex optimization from a continuous dynamical system perspective, GRETSI, 2022.
12. H. Attouch, J. Fadili, From the Ravine method to the Nesterov method and vice versa: a dynamical system perspective. SIAM J. on Optimization, 32(3):2074-2101, 2022.
SOUMISES
13. S. Maier, C. Castera, P. Ochs, Near-optimal Closed-loop Method via Lyapunov Damping for Convex Optimization. arXiv:2311.10053, 2023.
14. Shida Wang, Jalal Fadili, Peter Ochs, Convergence rates of regularized quasi-Newton methods without strong convexity. 2025. arXiv:2506.00521.
15. Shida Wang, Jalal Fadili, Peter Ochs, Global non-asymptotic super-linear convergence rates of regularized proximal quasi-Newton methods on non-smooth composite problems. 2024. arXiv:2410.11676.
16. Rodrigo Maulen-Soto, Jalal Fadili, Peter Ochs. Inertial Methods with Viscous and Hessian driven Damping for Non-Convex Optimization. 2025. ?hal-04651420v4?.

Structured composite non-smooth optimization has proved to be extremely useful in data science, where a common trend is the deluge of large-scale multidimensional data, which drives the need for more efficient optimization schemes. While, so far, primarily, first-order schemes have been widely used to solve such optimization problems, they are quickly approaching their natural (and provable) limitations. In contrast, higher-order methods, in particular quasi-Newton ones, are hardly used due to their lack of scalability, the complexity in their mathematical analysis and the deployment in the non-smooth case. TRINOM-DS will unlock these bottlenecks and develop theoretical, numerical and algorithmic advances to exploit the great potential of quasi-Newton-type schemes for non-smooth large-scale optimization problems, which are ubiquitous in data science. These algorithms will be developed in a variety of challenging settings, and are expected to have far reaching applications in data science, e.g., machine learning (optimal transport, deep learning, etc), imaging and computer vision. They will be implemented as fast optimization codes, possibly on dedicated architectures, that will be made publicly available following the philosophy of reproducible research. TRINOM-DS’s members are very active players in the European school of optimization and data science, which makes this project a unique opportunity to give a scalable, computational and practical embodiment to higher-order non-smooth optimization methods.