High-Performance Computing Center Stuttgart
Bernuzzi and his team, together with international collaborators specializing in computational relativity, use high-performance computing (HPC) resources at the High-Performance Computing Center Stuttgart (HLRS) and Leibniz Supercomputing Centre (LRZ) to interpret the observations, running highly accurate simulations of BNS mergers. Using Hawk at HLRS and SuperMUC-NG at LRZ, as well as resources at other HPC centers, the team recently completed the development of GR-Athena++, one of the first numerical-relativity codes with pre-exascale capabilities. Bernuzzi also worked with postdoctoral researcher William Cook to develop a neural network to improve modeling of magnetic flows and turbulence in BNS simulations.
For years, computational physicists have used the Athena++ code, developed at Princeton University, to model astrophysical magnetohydrodynamics (MHD). Bernuzzi and his collaborators expanded on this public code by adding Einstein’s equations of general relativity. The expanded code, called GR-Athena++, enables researchers to study complex astrophysical phenomena under conditions of extreme gravity, such as black holes, binary neutron star mergers, gravitational collapse, and other phenomena characterized by dynamic spacetimes.
Among other advancements the researchers integrated into GR-Athena++, the work included a radiation transport module to model neutrinos – small, neutrally charged subatomic particles that are produced in a BNS merger in large numbers. Neutrino-matter interactions – also called weak interactions – change the composition of matter by transforming neutrons into protons and electrons, which, crucially, determine the conditions for nucleosynthesis of heavy elements, like gold.
“The original code was already one of the leading codes for computational astrophysics, having advanced methods for solving magnetohydrodynamics equations,” Bernuzzi said. “We included a spacetime solver, which is a significant addition, as well as neutrino transport and state-of-art technology for extracting gravitational waves and dealing with dynamic black holes and neutron star spacetimes.”
With these improvements, astrophysicists can now use GR-Athena++ to make precision calculations to study gravitational waves from black hole mergers and magnetic field instabilities in isolated neutron stars. The code also incorporates enhanced microphysics that permit new BNS simulations for studying elements’ formation. The team published its results in Physical Review D. They have also contributed to another code, called AthenaK. Primarily developed by US colleagues, the code uses modern heterogenous computing architectures with the programming ecosystem Kokkos for exascale computing.
More recently, Bernuzzi and his collaborators have used GCS resources to address one of the other major complications when modeling complex astrophysical phenomena on supercomputers – turbulence.
Neutron stars host some of the most extreme magnetic fields in the Universe, up to 100 million times stronger than the most extreme magnetic fields generated on Earth. The strength of the magnetic fields in plasma can be further amplified in a binary collision, as the pair of stars pull toward one another. As a result, the plasma surrounding these celestial bodies starts to move in a turbulent way, making the accurate simulation of how it behaves difficult, even using a supercomputer.
In fluid dynamics research – whether simulating astrophysical plasmas or fuel being injected in a combustion engine – computational scientists and engineers often use something called large-eddy simulations (LES) to balance accuracy and efficiency. Essentially, LES allows researchers to solve equations that accurately represent large-scale turbulent motions, or eddies. LES also include so-called “sub-grid” models that describe the impact of small-scale dynamics not resolved in their simulations. This approach allows scientists to simulate more realistic scenarios but requires them to carefully calibrate sub-grid models on reference data from extraordinary high-resolution direct numerical simulations or local simulations.
Bernuzzi and postdoctoral researcher William Cook want to apply this approach to BNS merger simulations, hoping that LES could improve the accuracy of plasma dynamics. While other teams have previously used LES for this purpose, Cook and Bernuzzi took a novel approach by using machine learning methods to train neural networks.
As their name implies, neural networks are machine learning models that work in a similar way to a human brain as it makes decisions. Researchers feed in large amounts of training data from other turbulence experiments and simulations to the neural network and give feedback to the model concerning how well it represents known behavior. Over time, neural networks start to find patterns in large, complex amounts of data, including connections that are not accessible in global 3D simulations.
The researchers trained the model and ran preliminary tests to ensure that their neural network faithfully reproduces the Kelvin-Helmholtz instability, one of the main mechanisms for magnetic field amplification in BNS. The researchers used the neural network to run a relatively low-resolution simulation and demonstrated that the model performed well in driving turbulence over time.
The team published its results in a second article appearing in Physical Review D and is applying for more computing time through the Gauss Centre for Supercomputing and other international high-performance computing (HPC) facilities. Its goal is to further refine sub-grid models that can comprehensively and accurately represent the complex physics happening at the smallest turbulent scales.
Bernuzzi indicated that this work gives the team a powerful new tool that will enable it to continue improving its models of plasma dynamics. At the same time, it speaks to how the availability of more powerful HPC resources has advanced what is possible in the field. “I’m very impressed with the developments in our numerical relativity community in the last 10 years,” he said. “Many of these methods have become very mature, and in the medium to long-term future, I can see us applying these methods to other astrophysical systems that involve strong gravity.”
-Eric Gedenk
Funding for HLRS's Hawk and Hunter supercomputers was provided by the Baden-Württemberg Ministry for Science, Research, and the Arts and the German Federal Ministry of Research, Technology and Space through the Gauss Centre for Supercomputing (GCS).
Daszuta B, Bernuzzi S, Jacobi M, et al. 2026. Binary neutron star merger simulations with neutrino transport using GR-athena++. Phys Rev D. 114: 044062.
Cook W, Bernuzzi S. 2026. Subgrid modeling for relativistic magnetohydrodynamics with machine learning. Physl Rev D. 114: 063010.
A member of the Gauss Centre for Supercomputing, HLRS is one of three German national centers for high-performance computing.
