Node-Level Performance Engineering

May 14: This course will be provided as ONLINE course (using Zoom).

This course covers performance engineering approaches on the compute node level. Even application developers who are fluent in OpenMP and MPI often lack a good grasp of how much performance could at best be achieved by their code. This is because parallelism takes us only half the way to good performance. Even worse, slow serial code tends to scale very well, hiding the fact that resources are wasted. This course conveys the required knowledge to develop a thorough understanding of the interactions between software and hardware. This process must start at the core, socket, and node level, where the code gets executed that does the actual computational work. We introduce the basic architectural features and bottlenecks of modern processors and compute nodes. Pipelining, SIMD, superscalarity, caches, memory interfaces, ccNUMA, etc., are covered. A cornerstone of node-level performance analysis is the Roofline model, which is introduced in due detail and applied to various examples from computational science. We also show how simple software tools can be used to acquire knowledge about the system, run code in a reproducible way, and validate hypotheses about resource consumption. Finally, once the architectural requirements of a code are understood and correlated with performance measurements, the potential benefit of code changes can often be predicted, replacing hope-for-the-best optimizations by a scientific process.

This course provides - via lectures, demos, and hands-on labs - scientific training in Computational Science, and in addition, the scientific exchange of the participants among themselves.


Online course
Organizer: HLRS, University of Stuttgart, Germany

Start date

Jun 29, 2020

End date

Jul 01, 2020



Entry level


Course subject areas

Parallel Programming


Code Optimization

Back to list


Day 1

08:45        Enter the Zoom meeting
09:00        Welcome – Intro
09:30        Computer architecture for software developers part 1
10:00        Coffee break
10:15        Computer architecture for software developers part 1 cont.
10:45        Hands-on: Warmup (login etc.)
11:00        Computer architecture for software developers part 2
11:45        Hands-on: Divide benchmark
12:30        Lunch
13:30        Tools: Topology and affinity, frequency
14:00        Hands-on: likwid-topology and likwid-pin
14:15        Roofline Model: Basics
15:15        Hands-on: Microarchitectural exploration
16:00-        Open end

Day 2

08:45        Enter the Zoom meeting
09:00        Tools: Performance counters
09:30        Hands-on: performance counters and memory bandwidth
10:00        Optimal use of parallel resources: SIMD
11:00        Coffee break
11:15        Hands-on: SIMD in MiniMD
12:15        Optimal use of parallel resources:  ccNUMA
13:00        Lunch
14:00        Performance Engineering: Basic skills
14:45        Coffee break
15:00        Hands-on: Dense Matrix Vector Multiplication
16:00-        Open end

Day 3

08:45        Enter the Zoom meeting
09:00        Roofline case study: Jacobi smoother
10:00        Coffee break
10:15        Hands-on: Matrix free CG solver part 1
11:15        Coffee break
11:30        Roofline case study: SpMVM
12:30        Lunch
13:30        ECM Performance model
14:15        Coffee break
14:30        Hands-on: Matrix free CG solver part 2 (optimizatons)
16:00        Farewell & feedback

Detailed Program


  • Our approach to performance engineering
  • Basic architecture of multicore systems: threads, cores, caches, sockets, memory
  • The important role of system topology

Tools topology & affinity in multicore environments

  • Overview
  • likwid-topology and likwid-pin

Microbenchmarking for architectural exploration

  • Properties of data paths in the memory hierarchy
  • Bottlenecks
  • OpenMP barrier overhead

Roofline model: basics

  • Model assumptions and construction
  • Simple examples
  • Limitations of the Roofline model

Tools: hardware performance counters

  • Why hardware performance counters?
  • likwid-perfctr
  • Validating performance models

Roofline case studies

  • Dense matrix-vector multiplication
  • Sparse matrix-vector multiplication
  • Jacobi (stencil) smoother

Optimal use of parallel resources

  • Single Instruction Multiple Data (SIMD)
  • Cache-coherent Non-Uniform Memory Architecture (ccNUMA)
  • Simultaneous Multi-Threading (SMT)

Extending Roofline: The ECM performance model

Optional: Pattern-based performance engineering 

Prerequisites and content levels


Participants should be able to work on the Linux command line. They must have basic knowledge in programming with Fortran or C, and basic OpenMP.

Content levels
  • Intermediate: 9 hours
  • Advanced: 9 hours

Learn more about course curricula and content levels.


The course language is English.


Dr. habil. Georg Hager and Dr.-Ing. Jan Eitzinger (formerly Treibig) (RRZE/HPC, Uni. Erlangen)


Before the course, the course material and an updated agenda will be available here.
An older version of this course with most of the material (including the audio information) can also be viewed in the ONLINE Parallel Programming Workshop.


Registration is now closed.


for registration is June 14, 2020 (extended deadline).

Late registrations after the deadline are still possible but not later than June 21, 2020 and  maybe with reduced quality of the service.


Members of German universities and public research institutes: none.
Members of universities and public research institutes within Europe or PRACE-member countries: none.
Members of other universities and public research institutes: 240 EUR.
Others: 600 EUR.
(includes coffee breaks)


Travel Information and Accommodation

see our How to find us page.


HLRS is part of the Gauss Centre for Supercomputing (GCS), which is one of the six PRACE Advanced Training Centres (PATCs) that started in Feb. 2012. The mandate for the PATCs is as follows: "The PRACE Advanced Training Centres will serve as European hubs of advanced, world-class training for researchers working in the computational sciences." (see D3.2.3)
This course is a PATC course, see also the PRACE Training Portal and Events. For participants from public research institutions in PRACE countries, the course fee is sponsored through the PRACE PATC program.

HLRS is also member of the Baden-Württemberg initiative bwHPC-C5.
This course is also provided within the framework of the bwHPC-C5 user Support.

Local Organizer

Rolf Rabenseifner phone 0711 685 65530, rabenseifner(at)
Lucienne Dettki phone 0711 685 63894, dettki(at)

Further Courses and

See also the Scaling and Optimization Workshop 2020/HPE2. One of the NLP trainers will stay for the first day of this scaling workshop.

Related training

All training

July 15 - 19, 2024

Hybrid Event - Stuttgart, Germany

October 23 - 25, 2024


December 09 - 13, 2024