Algebraic and geometric domain decomposition for subsurface /groundwater flows – DEDALES

Project DEDALES is structured along three scientific tasks.

Extension of domain decomposition methods to two phase flow. One has to solve a nonlinear system of partial differential equations, that couple pressure and saturation for each phase. Three research directions are: the interplay of Newton's method with the DD algorithm, which can be «Newton inside, DD outside« or «Newton outside, DD inside«, determining optimized transmission conditions to improve the convergence rate of the DD algorithm and optimizing the solution of the subdomain problems in vector languages. The validation will use the open-source code MRST (Matlab Reservoir Simulation Toolbox).

Achieving optimal performance on hierarchical computers by providing an efficient hybrid solver on distributed architectures of heterogeneous nodes. The MaPHyS solver, developed by the Inria Team HiePACS, will be the development platform for this task. The two objectives are to combine an outer iterative solver based on MPI with an inner direct solver implemented on top of a runtime system such as StarPU, and to study alternative coarse grid preconditioners in the context of strongly heterogeneous problems such as coming from subsurface flow models. The third part of this task will be the validation of this hybrid solver on data from the targeted geosciences applications.

Create powerful solvers using domain decomposition so as to take advantage of two levels of parallelism: (space-time) DD will provide an outer level of parallelism, while a parallel direct solver gives the inner level. The outer level will run on MPI using message passing, coarse grain parallelism, while the outer level will be based on the StarPU runtime, using fine grain scheduling techniques, as explained in Task 2. Two open source codes will be used for the implementation: TRACES, used by Andra for flow and transport simulations, and DuMuX, a framework for two phase flow simulations.

Efficient algorithms for finite element assembly in vector languages (published);

Extension of the optimized Schwarz method with Robin transmission conditions to a nonlinear diffusion model ;

Development of stopping criteria for the Schwarz method based on a posteriori estimates ;

Integration into TRACES of a version of MaPHyS for heterogeneous compute nodes ;

Development of a two-level preconditioner for MaPHyS.

The main advances are expected in the following directions:

The project will advance research in domain decomposition for two-phase flow problems, and for nonlinear problems in general.

Linear solvers based on a hybrid iterative (based on an algebraic view of domain decomposition) and direct methods will be validated on large scale heterogeneous problems.

Hybrid codes mixing MPI with programming models for many-core also lead to mostly open issues.

In general, the project aims at taking a step towards co-design of new solvers with many-core programming in mind. This fits with recommendations of the European Exascale Software Initiative about numerical solvers. In addition to methodological advances both in the theory and use of domain decomposition for porous media flow, tangible outcomes of this project will be a demonstration code together with examples showing the viability of the approach.

Publication in journal : F. Cuvelier, C. Japhet and G. Scarella, An efficient way to assemble finite element matrices in vector languages, BIT Numerical Mathematics (2015-12-10): 1-32, 2015.

Conferences:

Groupe de travail « méthodes numériques » UPMC (Paris, mars 2015), F. Cuvelier

Séminaire d'analyse numérique, LAGA, Paris 13 (mars 2015), F. Cuvelier

Conférence Mamern (Pau, juin 2015), C. Japhet

SMAI 2015 (Les Karellis, juin2015), G. Scarella

SMAI 2015 (Les Karellys, juin 2015), Louis Poirel, Emmanuel Agullo, Mathieu Faverge, Luc Giraud

Journées MoMas multiphasique (Nice, octobre 2015), M. Kern

SPPEXA Symposium (Münich, Janvier 2016), M. Kern

The recent progress in computer architecture lead to increased demand for simulations including more, and more realistic; physics, and the geosciences are no exception: nuclear waste simulations involve coupling between thermal, hydraulic, mechanical and chemical phenomena. The same is true for CO2 storage in deep saline aquifers, which requires full coupling between two-phase flow and chemical reactions. The goal of the DEDALES project is to contribute to the mathematical methods, and the simulation tools, that will be needed to enable simulation of these complex phenomena on the next generation Exascale computers. A characteristic of these architectures will certainly be the hierarchical nature of their computing elements.

For applications requiring the solution of (systems of) partial differential equations, domain decomposition methods (in space and / or in time) have shown to lend themselves well to this architecture: in a DD method, an outer algorithm iterates over subdomain solves, and these can be arranged to be independent, and solved in parallel (this is a coarse grain type of parallelism). In turn, each of the subdomain solves is itself the solution of a problem of the same type (or simpler), and can be parallelized using techniques for the parallel solution of PDEs. Such an approach is moreover well suited to local time stepping in the subdomains.

Linear solvers have always been, and will continue to be, at the center of simulation codes. However, parallelizing implicit methods on unstructured meshes, such as are required to accurately represent the fine geological details of the heterogeneous media considered, is notoriously difficult. It has also been suggested that time level parallelism could be a useful avenue to provide an extra degree of parallelism.

Project DEDALES will show that space-time DD methods can provide this extra level, and can usefully be combined with parallel linear solvers at the subdomain level.

To improve two-phase flow simulation in such context, we propose a two level approach with the outer level using physically based domain decomposition, parallelized with a distributed memory approach, while the inner level for the sub domain solver will exploit both MPI and thread level parallelism. This approach matches well with the hierarchical nature both of physics and of the hardware architecture. The resulting hybrid parallel solver will be integrated into an Open Source flow simulator, and validated against several large scale examples.

This proposal will
- Study how space time domain decomposition methods with local time stepping, which have been successfully used in saturated flow situations, can be extended to the two phase flow case;
- Implement a hybrid linear solver by combining an outer iterative method (using MPI) with an inner direct solver on top of a runtime system like StarPU, and study how this solver can be adapted to deal with the specific issues arising from hydrogeological models, such as the very high degree of medium heterogeneity;
- Implement the previous ideas to show that they provide a natural and efficient way to exploit two- or three-level parallelism, and validate the implementation on several challenging examples.