High-Performance Computing Center Stuttgart
By the late 2010s, scientists across Europe had developed and validated new software for simulating systems inside the Earth. For these models to achieve the scale and resolution necessary to answer important scientific questions, however, they realized that they needed more computing power. In parallel, high-performance computers like those operated by the Gauss Centre for Supercomputing were gaining unprecedented performance capabilities. Geoscientists recognized the opportunities such supercomputers could offer, but they faced a problem: the codes they used did not run well on large, highly parallelized systems. To achieve the precision at a speed that would make it feasible to respond to natural disasters in real time, a focus on performance optimization became essential.
To address this challenge, an international community of scientists came together in 2018 to establish ChEESE, the Centre of Excellence for Exascale in Solid Earth. The effort was renewed in a second project phase called ChEESE-2P, which was funded by the EuroHPC Joint Undertaking and concludes at the end of 2026. Coordinated by the Barcelona Supercomputing Centre (ChEESE) and the Consejo Superior de Investigaciones Científicas (ChEESE-2P), the projects have brought together Earth scientists and experts in high-performance computing to improve scientific codes for solid Earth simulations in ways that would enable them to take maximal advantage of the opportunities that European HPC systems could offer.
To date, the ChEESE projects have focused on the scaling and optimization of 11 flagship scientific codes for solid earth simulations. This has included identifying performance bottlenecks, repeatedly monitoring code performance to track progress in scaling and efficiency, and using this information to achieve higher performance. ChEESE and ChEESE-2P have supported the publication of 32 scientific papers and — even more importantly — have helped in the development of computing workflows that could be deployed to more effectively predict and manage natural disasters.
At the same time that HLRS has been a partner in ChEESE, it has also been involved in a European consortium called POP, the EuroHPC Performance Optimization and Productivity Center of Excellence. Working together with other leading high-performance computing centers, HPC experts at HLRS have helped to develop analysis tools for evaluating software performance. They also regularly use them to improve code performance on HPC systems.
POP is a free service for academic and industrial HPC users. Once inserted into scientific codes, its performance analysis tools monitor key performance indicators including parallel efficiency, load balance, communication efficiency, scaling efficiency, serialization efficiency, and data transfer efficiency. HPC experts within POP use the results to identify bottlenecks and make recommendations that would improve the code's efficiency and scalability to larger computing systems.
As the the EuroHPC Joint Undertaking has rolled out multiple large-scale high-performance computing systems across Europe, it now recommends that all users of these systems submit their codes for POP analysis. This step not only ensures that users gain more scientific benefit from their computing grants, but also that EuroHPC systems are as productive as possible.
Within the ChEESE projects, performance was not merely a technical matter. One unique feature of the projects has been that they have included an advisory board made up of public officials from communities under threat from potential natural disasters. Representatives of civil protection agencies in Norway, Iceland, and Italy, for example, made specific recommendations concerning the kinds of prediction tools that would be most useful to them.
"Civil Protection Italy, for example, might come to ChEESE to describe what kind of maps they need and the resolution they require," says Dr. Jose Gracia, head of the HLRS Department of Scalable Programming Models and Tools. "The cooperation baked into ChEESE means that there is real interest from real users of our results."
In its most recent project phase, ChEESE-2P extended its focus beyond performance optimization of individual codes to include the workflows into which HPC simulations must be integrated in crisis management situations. To respond to a potential tsunami, for example, seismic sensors might trigger an alert and take initial measurements. The data they collect must be automatically fed into a model, which then quickly analyzes the simulation to predict the extent of danger. In severe cases, reliable, easily interpretable results must then immediately be transmitted to public agencies responsible for alerting the public and protecting their safety. Executing such a workflow efficiently means designing a computing workflow that rapidly moves data through multiple stages of analysis and processing.
One ChEESE simulation case was led by Mattia de' Michieli Vitturi and Esposti Ongaro at the National Institute of Geophysics and Volcanology (INGV) of Italy. Focusing on the Campi Flegrei Volcano in Naples, Italy, their team focuses on methods for simulating multiphase flows of phreatic volcanic eruptions, which occur when hot magma heats ground water to high pressure, leading to an explosion. The goal is to develop an early warning tool that could be used when a danger of a phreatic eruption is identified to predict likely eruption scenarios.
Vitturi and Ongaro use the ChEESE-2P flagship code OpenPDAC for their simulations, and have been focused on optimizing it for the EuroHPC system LUMI for urgent computing applications. Their approach involves running an ensemble of parallel simulations to search the parameter space and find the likeliest scenario for an eruption. The workflow also involves pre-processing steps such as parameter setting and generation, and the processing of mesh grids. During post-processing of the ensemble simulation, the simulation results are visualized into figures and movies. Job scheduling and delegation of computing resources are also integral parts of the workflow. Although pre- and post-processing steps account for a very low percentage of the overall compute time, they are pivotal to the correctness and performance of the simulations, and the interpretation of simulation results.
The performance optimization team at HLRS worked with colleagues from INGV to improve the Python and Bash scripts used in pre- and post-processing. After using tools such as Pylint and Shellcheck to do an initial correctness check, the researchers conducted test runs to identify the most time consuming section. Employing the Numba module to convert functions that process mesh grids to machine codes, they achieved a speed up of approximately 27% in the pre-processing stage. In the post-processing, HLRS also suggested moving video generation from CPU to GPU and sped up this process up to a factor of thousands.
These kinds of performance improvements will help to make it more feasible to use OpenPDAC within urgent computing settings, and demonstrate the important role that HPC expertise can play in translating scientific innovation into applications that protect property and life.
— Christopher Williams
A member of the Gauss Centre for Supercomputing, HLRS is one of three German national centers for high-performance computing.
