Reinforcement learning as optimal control for fluid flows – REASON

Environmental needs are invigorating research interest in many engineering fields. A compelling example is provided by carbon-dioxide emissions, widely considered one of the main causes of global warming. This urgency extends to numerous applications including aeronautics, where it is recognized that the optimization of aerodynamic flows may have a deep impact on the reduction of pollutant emissions, mitigation of acoustic noise or control of highly complex conditions such as separation. In principle, flow control strategies allow to optimize the flow in real time, leveraging sensor measurements and physical models; in practice, in realistic cases, this technique is used only in limited numerical and experimental cases.
In this proposal, we aim at verifying to what extent Reinforcement Learning (RL) is a viable, alternative strategy for the control of fluids in realistic conditions. RL algorithms use past data to enhance the future manipulation of a dynamical system by discovering optimal control policy, determined from the exploration of the state-action space. This step replaces physical models; thus these algorithms are fully data-driven and solely rely on the measurements of the system and the way it reacts to prescribed actions. This allows to circumvent some drawbacks of model-based control, as approximate reduced-order models of the physical system can critically lose accuracy when control is applied, resulting in poor performance and lack of robustness.
The main goal of the project is to provide major breakthroughs in flow control by bridging it with Reinforcement Learning in order to tackle the challenges which have limited the success of standard control tools in nonlinear and complex flows. In that sense, we aim at developing robust strategies by integrating in this framework efficient techniques of learning/estimation, physics constraints and tools from control theory. We will test our strategies in models of fluid-mechanics interest of increasing complexity. The system governed by the Kuramoto-Sivashinsky and the linearized boundary layer in 2D will serve as simplified benchmarks for comparing the optimal control solutions and RL. Final demonstrators will include numerical simulations of transitional shear flows at moderate Reynolds number, and the control of the bistable dynamics of an experimental 3D bluff body wake-flows.