Hazel Hen's Millionth Compute Job

27 July 2017

The HLRS supercomputer recently reached a milestone, crossing into seven digits in the number of compute jobs it has executed. The millionth job is an example of the essential role that high-performance computing is playing in many research fields, including fluid dynamics.

Traditional laboratory experimentation has been and continues to be indispensible to the advance of scientific knowledge. Across a wide range of fields, however, researchers are being compelled to ask questions about phenomena that are so complex, so remote, or that are found at such small or large scales that direct observation just isn’t practical anymore. In these cases, simulation, modeling, data visualization, and other newer computational approaches offer a path forward. The millionth job on Hazel Hen was one such case.

Leading the research behind the millionth job was Professor Bernhard Weigand, Director of the Institute of Aerospace Thermodynamics at the University of Stuttgart. His laboratory studies multiphase flows, a common phenomenon across nature in which materials in different states or phases (gases, liquids, and solids) are simultaneously present and physically interact. In meteorology, for instance, raindrops, dew, and fog constitute multiphase flows, as does the exchange of gases between the oceans and the atmosphere. Such phenomena also occur in our daily lives, such as when water bounces off our skin in the shower or when we inhale nasal sprays to control the symptoms of a cold.

The FS3D team in the Weigand Lab executed the
research behind Hazel Hen's millionth job.
(Photo courtesy of the Weigand Lab, ITLR.)

In engineering, multiphase flows can also be extremely important. Perhaps their most familiar application is in the design of fuel injection systems in automobiles, gas turbines, and rockets. Other examples include the spreading of fertilizers for farming or the use of spray drying in the production of pharmaceuticals and foods.

In all of these cases, understanding how multiphase flows behave in detail could both enhance our ability to study the natural world and improve the design of more effective and more efficient products. But because of the enormous numbers of droplets that are involved in multiphase flows and the extremely small scale at which they interact, our ability to gain precise knowledge about them purely through observation has been limited.

For this reason, Weigand turned to HLRS and its Hazel Hen high-performance computer to simulate multiphase flows computationally. His work and that of his colleagues has led to a variety of insights with wide-ranging practical relevance.

Supercomputing simulates droplet dynamics

Professor Weigand and his group are primarily interested in basic multiphase flows involving droplets, such as those that fall as rain from the sky. In the past Weigand and his group investigated topics related to the dynamics of cloud formation, for example, gaining insights into what happens when droplets in the atmosphere collide; these findings were subsequently used by other scientists to develop better weather forecast models. Weigand is also speaker of the Collaborative Research Council SFB-TRR 75 (a research project funded by the Deutsche Forschungsgemeinschaft (DFG) that also includes investigators at TU Darmstadt and DLR Lampoldshausen). In this capacity his team has been investigating the fluid dynamics of super-cooled water droplets in extreme situations, such as when ice crystals develop in clouds. This problem is important for precipitation forecasting (for example, hail) and also in air travel, as ice formation on airplane wings can negatively affect flight stability and decrease fuel efficiency.

To study the dynamics of droplets' physical behavior, Weigand and his group a mathematical approach called direct numerical simulation (DNS). Over many years he and members of his lab have been building DNS methods into an in-house software program called FS3D (Free Surface 3D), which they use to model droplet dynamics. FS3D can, for example, precisely simulate what happens when a water droplet falls onto a liquid film and forms a "crown," taking a new shape and breaking apart into smaller droplets.

A comparison of high-speed photographs of droplet crown formation (left side of each
column) with FS3D simulations. (Image courtesy of the Weigand Lab, ITLR.)

High-performance computing (HPC) is absolutely essential to the success of FS3D because the software requires an extremely high "gate resolution." Like the frame rate in a video or movie camera, the program must represent the complex collisions, adhesions, and breaking apart of droplets and molecules at extremely small scales of space and time. FS3D can simulate such interactions in 2 billion "cells" at once, each of which represents a volume of less than 7 cubic micrometers, tracking how the composition of every cell changes over time.

Achieving such a high resolution generates massively large datasets, and it is only by using a supercomputer as powerful as HLRS's Hazel Hen that these simulations can be run quickly enough to be of any practical use. Moreover, during simulations, HPC architectures can rapidly and reliably save enormous collections of data that are output from one round of calculations and efficiently convert them into inputs for the next. In this way, simulation becomes an iterative process, leading to better and better models of complex phenomena, such as the multiphase flows the Weigand Lab is investigating.

Having so much power at your disposal presents some unique challenges, though. In order to take full advantage of the opportunities that supercomputers offer, software behind algorithms like FS3D must be written specifically for the parallel computing architecture of high-performance computing systems. Programming in this way requires special expertise, and as FS3D has developed, staff members at HLRS and at Cray, the company that built Hazel Hen, have helped the Weigand Lab to optimize it for HPC.

"It's not really practical for us to have HPC experts in our lab, and so staff at HLRS and Cray have been very supportive in helping us to run FS3D effectively on Hazel Hen," says Dr. Weigand. "Their knowledge and advice have been very important to the success of our recent studies."

The millionth job: visualizing how non-Newtonian fluids break apart in jets

The millionth job on Hazel Hen was not focused on atmospheric water, but instead on multiphase flows in non-Newtonian fluids. Such fluids — which include materials like paint, toothpaste, or blood — do not behave in ways that Newton's laws of viscosity would predict; instead, their fluid dynamic properties follow other rules that are not as thoroughly understood.

More specifically, Weigand’s team wanted to use computational simulations to gain a better understanding of how non-Newtonian jets break up when injected into a gaseous atmosphere. This question is important because droplet sizes and the increase in a fluid's surface area as it becomes atomized can be important factors in optimizing the efficiency of a process — such as in the application of aerosolized paint to a car body.

3D visualization of the data set investigated in Hazel Hen's millionth job.
The CAVE at HLRS makes it possible to explore a fluid jet in fine detail.
(Photo: HLRS)

The researchers simulated the injection of aqueous solutions of the polymers Praestol2500® and Praestol2540® through different pressure nozzles into air. When used in water treatment, the viscosity of these polymers decreases due to shear strain. The fluid properties for this case were approximated by flow curves obtained from experiments by colleagues at the University of Graz.

Running FS3D on Hazel Hen, the Weigand team performed a variety of "virtual" experiments on the supercomputer to investigate specific features of these flows, gaining a much more precise picture of how the solutions disperse. For example, they modeled jet breakup after injection and how factors such as flow velocity and the shape of the nozzle changed the fluids' viscous properties. (This work was undertaken under the auspices of DFG-funded priority program SPP 1423-Process Spray. Speaker: Prof. Udo Fritsching, University of Bremen.)

The millionth job run on Hazel Hen was one of several post-processing visualizations the team undertook in cooperation with VISUS (University of Stuttgart Visualisation Research Centre) to investigate the development of a liquid mass over time. In this series of studies, they generated extremely fine-grained visualizations of changes in the shape of the flow passing through the jet, identified differences in the loss of flow cohesion under different conditions, and discovered changes in surface area as the flow becomes atomized, among other characteristics. This led to insights about similarities and differences between Newtonian and non-Newtonian flows, and about how nozzle shape affects flow properties.

In the future, such information could enable engineers to improve the efficiency of their nozzle designs. In this sense, the millionth compute job on Hazel Hen was just one page in a long and continuing scientific story. Nevertheless, it embodies the unique kinds of research that HLRS makes possible everyday.

Christopher Williams