Analysis, Improvement and Evaluation of Numerical Black-Box Optimizers – NumBBO

Numerical optimization problems are at the core of many present industrial design or development tasks and inherent to some of the decisive issues of our society. Numerical blackbox optimization methods, interpreting a problem as a blackbox where the only available information is the obtained function value for some query points, are the methods of choice when models are non-differentiable, non convex, multi-modal, noisy, or too complex to be mathematically tractable. Simulation is a cornerstone for the analysis, design and evaluation of those blackbox optimizers as practical algorithms are too intricate to be easily analyzed theoretically or models where theory is possible do not cover the range of problems where we need to understand and evaluate algorithms.

A nowadays standard method for addressing numerical blackbox optimization in practice is the CMA-ES algorithm developed by the INRIA-TAO partner. It is a parameter-free adaptive stochastic search method that addresses properly the questions of solving non-separable, ill-conditioned problems, converging linearly, and being robust to noise and multi-modality. In parallel to the development of CMA-ES, various other methods were introduced to address difficult numerical blackbox optimization problems. From a practitioner point of view, it is therefore difficult to know which method to apply. Benchmarking, by allowing to assess performances of different algorithms in the same conditions, offers a way to gain understanding and guide the choice among this variety of optimizers. The framework COmparing Continuous Optimizers (COCO) developed by partners of this project (TAO and TU Dortmund) has been designed to automatize the tedious benchmarking tasks of collecting, postprocessing and visualizing data. It proposes a carefully justified choice of test functions and meaningful quantitative performances measures. It allows to benchmark but is also a natural environment to design better algorithms faster. So far, more than 60 algorithms were benchmarked with COCO by researchers around the world. This data collection opens the way for intensive statistical analysis to perform Exploratory Landscape Analysis whose final goal is to select the best available algorithm for a given problem.

Algorithms benchmarked with COCO are restricted to single-objective unconstrained optimization and concern methods that are not tailored to expensive problems, where only a few hundred function evaluations can be afforded and one has to resort to surrogates of the original problem (expertise of the EMSE partner). Constrained optimization is another important aspect of many real-world problems and is challenging for adaptive methods like CMA-ES. Also, many numerical optimization problems involve the simultaneous optimization of several conflicting objectives where no single optimal solution exist but where Evolutionary Multiobjective Optimizers (EMO) (expertise of the INRIA-DOLPHIN team), can find good sets of solutions showing the trade-offs among the objectives. Like for the single-objective case, benchmarking of EMO is important to understand and assess performance of algorithms.

This project builds on the expertise of the different partners for analyzing, improving, and evaluating numerical blackbox optimizers in the context of single-objective (constrained, large-scale), multiobjective, and expensive optimization with a focus on CMA-ES. One cornerstone of the project is the COCO framework that will be used to gain knowledge about existing methods but also as an environment to design better algorithms faster. We aim here at extending COCO to constrained and multiobjective optimization, exploiting the similarities between the topics and develop further visualization tools. Exploratory Landscape Analysis in single and multiobjective optimization (goal of a submitted DFG proposal by TU Dortmund) will take place using the outcomes of benchmarked algorithms collected with COCO.