Code Porting to Hunter Accelerates Simulations for Meteorology Research

To simulate atmospheric turbulence, scientists rely on the Monin–Obukhov similarity theory. This theory holds that if you know wind velocity between two points in the boundary layer, it is possible to predict how wind speed changes at higher levels within the surface layer. For this approach to be mathematically practical, however, Monin–Obukhov assumes that the Earth's surface is flat. In the real world, topographic features such as mountains and valleys affect how turbulence develops in the atmospheric boundary layer, and so Deshpande and his colleagues have been investigating methods for better incorporating surface roughness into atmospheric turbulence models.

Deshpande uses an open source code called tLab for this research. Initially developed by Juan Pedro Mellado, now a professor at the University of Hamburg, together with Cedrick Ansorge, Deshpande's advisor at the FU Berlin, tLab is a code for direct numerical simulation (DNS). DNS methods use fundamental principles of fluid mechanics to resolve turbulence at extremely high precision — a computationally demanding task that can only be performed using high-performance computing. 

DNS also has its limitations, though. The enormous number of mathematical operations it requires, together with the data it produces, make it computationally impossible to simulate the atmospheric turbulent boundary layer at real-world scales, even using modern supercomputers. To manage this challenge, DNS codes such as tLab solve a non-dimensional version of the Navier-Stokes equations, allowing the results to be scaled up to real-world applications using the principles of the Buckingham π-theorem. 

In each experiment, Deshpande and other users of tLab typically run 20,000 to 30,000 iterations of the same system. By calculating a statistical average of all of these results, the scientists can make accurate predictions of atmospheric turbulence in nature. Performing so many iterations further amplifies the computing power required, and so in recent years the FU Berlin's working group "Turbulence in the Boundary Layer" has turned to the High-Performance Computing Center Stuttgart (HLRS) for access to its supercomputers. 

Code porting and optimization workshop leads to improved performance of tLab

The working group's most recent allotment of computing time at HLRS presented a challenge, as it coincided with the replacement of HLRS's Hawk supercomputer with its current system, Hunter. tLab was originally written for CPU-based system architectures like Hawk. Because Hunter uses AMD accelerated processing units (APU) that combine CPUs and GPU accelerators, however, the code would need significant adaptation to run. Working intermittently over the course of eight months, Deshpande took over the task of porting tLab to GPUs.

Recognizing that many researchers in HLRS's scientific user community would face similar difficulties, HLRS's user support team has been working closely with them to help navigate this transition successfully. This has included offering multiple instances of the workshop "Porting and Optimization for Hunter." Here, researchers work one-on-one with HLRS user support staff as well as representatives of HPE and AMD, who bring expert knowledge of Hunter's hardware and can deduce strategies for optimizing code performance on the GPU-accelerated system.

In 2025 and 2026, Deshpande attended two code porting workshops. In the first, HLRS user support staff ran a software profiler that identified subroutines within tLab that were running especially slow. This highlighted opportunities for improvement in the code's Poisson solver, an algorithm for solving second order linear differential equations representing pressure within turbulent systems. It proved to be relatively straightforward to restructure several subroutines within the solver for GPUs using OpenMP target offloading, and to use the shared CPU-GPU memory so that no data had to be copied back and forth.

In a second code porting workshop, Deshpande focused on tLab's implementation of the Thomas algorithm, a serial algorithm within the Poisson solver that solves a system of equations row by row. GPUs are not designed to handle this kind of procedure efficiently, and so a different approach was needed. As Deshpande explained, "Using GPU offloading, I changed the structure of our code so that multiple Thomas algorithms run in parallel, an approach that takes advantage of the massively parallel capabilities of GPU processors. Solving many Thomas algorithms at a time gave us an additional speed up." 

Porting to GPUs increases scientific productivity, reduces energy consumption

The successful porting of tLab to Hunter has enabled Deshpande to achieve performance gains that have enhanced his research capabilities. Previously, Jonathan  Kostelecky, a former PhD student and member of the FU Berlin working group, had used tLab on Hawk, publishing two research articles using data generated at HLRS. Using 64 CPU nodes on Hawk to simulate a case involving 9 billion data points, it could take a month before he received the results. Since porting the software to Hunter, Deshpande says that a similar simulation can now be handled by a single AMD APU node and finish the same work within a week. In a scaling test result, he demonstrated that a typical simulation task that involved running tLab on CPUs over a single Hunter node took 1,332 seconds, while the same task using Hunter's APU processors could be completed in 24.6 seconds, approximately 54 times faster. 

"Not only do we save time, but we can also test more simulations. This means that we can do more science with the allotment of compute hours that we receive," Deshpande explained. 

Running the code on Hunter also has environmental benefits: "From the perspective of energy we also save a lot moving from 64 CPU nodes down to just a single APU node," he added. HPE estimates that Hunter uses approximately 10% of the energy previously consumed by Hawk to do the same amount of work.

Additional performance improvements in tLab could also be possible in the future. Deshpande adjusted tLab such that memory from one APU can be written directly to another APU within the same node using OpenMP, making data transfer extremely smooth. When moving data between different nodes, however, tLab still relies on the MPI parallel programming standard, which on APUs introduces time-consuming bottlenecks due to hardware incompatibility and compiler compliance issues. Overcoming this limitation could make it possible to run even larger simulations. Deshpande has some ideas of how to address this, and plans to write a future paper about it.

Even now, however, the benefits of porting tLab to GPUs are apparent. "It's extremely fast," Deshpande says. "I can run four or five simulations simultaneously, and can test multiple cases in a single week. On Hawk, we needed between 48 and 64 nodes to simulate one case. On Hunter, one node is enough. With 48 Hunter nodes we could potentially perform pure ensemble simulations that include 48 independent realizations, perturbing the initial conditions and recording the changes in the results. Such a test could be a true ensemble simulation, which was difficult to do before." 

Porting tLab to APUs demanded major technical effort, but led to remarkable performance gains. Beyond increasing speed, this transition now enables scientists to perform high-resolution simulations over complex surface interactions that in the past were computationally out of reach. This ability could pave the way for gaining a better understanding of real-world atmospheric turbulence.

— Christopher Williams