Mathematics of Automatic Differentiation – MAD

Automatic Differentiation (AD) is pivotal in efficiently and accurately calculating derivatives of functions, crucial for optimizing mathematical models in gradient-based optimization tasks. AD operates by breaking down functions, expressed as computer programs, into basic operations and applying the chain rule for derivative computation. This stands in contrast to symbolic differentiation and numerical differentiation. However, from a mathematical standpoint, AD faces challenges with nonsmooth functions due to its foundational reliance on the chain rule. Convergence issues may emerge with iterative methods involved in the function being differentiated. Moreover, when AD interacts with parametric integral, difficulties, particularly in approximation, can arise. Project MAS aims to reconcile modern AD applications in machine learning pipelines with mathematical assurances for its correctness. The project is structured around three workpackages: WP1 delves into proving guarantees for AD application to iterative algorithms (unrolling), WP2 explores AD in bilevel optimization, and WP3 examines the interplay between Monte Carlo methods and AD usage.