Improving performance of spectral-element based geoscience simulations by combining low-level optimizations into high-level abstractions – INSEPTION

While the rise of high performance computing is tremendous, a majority of codes is still not fully adapted to the wide diversity of current and upcoming architectures, thus pushing computer scientists and physicists to modify and optimize them to better exploit supercomputers. In this line, the scope of INSEPTION is to develop a cross-platform open-source highly optimized multi-memory levels library to outperform the Spectral Element Method (SEM) state of the art implementations by designing and programming low-level optimizations of the SEM kernel on current CPUs and GPUs architectures and to prepare them for Exascale computing. An examination of the SEM codes showed that the computation of the internal forces “F” equal to the product of the stiffness matrix “K” with the displacements vector “U” (most time-consuming part common to any SEM codes) could be significantly improved by an explicit vectorization of the KU product using C++ Single Instruction Multiple Data (SIMD) intrinsics. The achieved single core speedup ranges between 2 and 5 depending on the SIMD architecture. Although this speedup is significant, it is not as high as expected since 512-bit wide SIMD units (single precision) may theoretically bring a speedup of 16. This lack of performance is due to memory bottlenecks specific to the SEM KU computation that we will investigate by studying multiple levels cache misses to improve both the use of SIMD units and caches, as well as the intra-node scalability that would be impacted. To hide the complexities of these optimizations to non-computer scientists and easily integrate future optimizations, we propose to isolate them in low-level software layers, combine them using a modular software architecture, and access them transparently through a domain specific language. This software architecture will improve SEM kernel portability across CPUs and GPUs with very different characteristics (size of SIMD units, memories hierarchy) and different programming paradigms, libraries and tools (SIMD intrinsics, OpenMP, CUDA, HIP, OneAPI) ; thus easing code modernization for Exascale computing. INSEPTION library will be interfaced with two SEM codes involved in multiple science fields whose applications range from laboratory ultrasound studies to planetary-scale seismology: SPECFEM3D and EFISPEC3D. Although INSEPTION project will push simulations’ size above the conventional practice’s size, grand challenge simulations are not foreseen. The expected gain will allow for more realistic earthquake ground motion predictions applied in France, more accurate full waveform seismic imaging, more exhaustive uncertainty quantification analyses performed on terabytes of data, and more rigorous seismo-acoustic simulations of anthropogenic ocean noise to reach spatial and spectral ranges not computable with actual codes. The algorithmic and I/O improvement of our end-to-end workflows (unstructured mesh preprocessing/partitioning, results high-performance post-processing, runtime filtering and decimation, etc.) will be under close scrutiny as well. They typically orchestrate multiple Earth’s models, multiple meshes or multiple sources configurations to compute millions of synthetic seismograms required by geophysical applications, like full waveform inversions to image Earth’s internal structure; global sensitivity analyses to quantify means, variances, marginal/joint probability distributions, intervals probability and sensitivity indices; or high-resolution seismo-acoustic simulations in shallow water to asses the noise generated by the disposal of historical explosive devices which is both a source of disturbance for marine life and a source of damage to onshore infrastructures. From our different experiences as developers or users of HPC codes and computing frameworks, we strongly believe that INSEPTION is the way to improve performance, productivity, maintainability and portability across current and future HPC architectures.