2021 [ to top ]

1.
Zhou, N., Zhong, L., Hoppe, D., Pejak, B., Marko, O., Cardona, J., Czerkawski, M., Andonovic, I., Michie, C., Tachtatzis, C., Alexakis, E., Mavrepis, P., Kyriazis, D., Pospieszny, M.: "CYBELE: A Hybrid Architecture of HPC and Big Data for AI Applications in Agriculture". HPC, Big Data, AI Convergence Toward Exascale: Challenge and Vision (book chapter). CRC Press (2021).
[ Abstract ] [ URL ]
 
Containers are widely utilized in Cloud to encapsulate applications and their dependencies in isolated environments, thus providing compatibility and portability to application execution environments. Artificial Intelligence (AI) applications often incorporate a complex stack of software packages which can benefit from containerization immensely. Big data analytics push the development of AI applications, making them more computation-intensive or data-intensive. There is a growing interest in executing AI applications in HPC clusters that are conventionally applied for large-scale engineering, scientific and financial simulations. This chapter presents a hybrid architecture consisting of a Cloud cluster and an HPC cluster, which has been proposed in the EU-funded research project CYBELE. A login node bridges the two clusters and provides a unified interface for job submission. More specifically, via the login node, long-running service programs are scheduled to be hosted on the Cloud cluster. Per contra, AI applications are scheduled to run on the HPC cluster where they obtain significant performance improvement. Furthermore, the methods of parallelism and deployment of containerized AI applications on HPC systems are described.

2.
Zhou, N., Georgiou, Y., Pospieszny, M., Zhong, L., Zhou, H., Niethammer, C., Pejak, B., Marko, O., Hoppe, D.: Container Orchestration on HPC Systems through Kubernetes. journal of Cloud Computing: Advances, Systems and Applications. (2021).
[ Abstract ] [ URL ]
 
Containerisation demonstrates its efficiency in application deployment in Cloud Computing. Containers can encapsulate complex programs with their dependencies in isolated environments making applications more portable, hence are being adopted in High Performance Computing (HPC) clusters. Singularity, initially designed for HPC systems, has become their de facto standard container runtime. Nevertheless, conventional HPC workload managers lack micro-service support and deeply-integrated container management, as opposed to container orchestrators. We introduce a Torque-Operator which serves as a bridge between HPC workload manager (TORQUE) and container orchestrator (Kubernetes). We propose a hybrid architecture that integrates HPC and Cloud clusters seamlessly with little interference to HPC systems where its container orchestration is performed on two levels.

2020 [ to top ]

1.
Zhong, L.: A Method for Stream Data Analysis. Sustained Simulation Performance 2019 and 2020: Proceedings of the Joint Workshop on Sustained Simulation Performance, University of Stuttgart (HLRS) and Tohoku University, 2019 and 2020. 111 (2020).
 

2.
Zhou, N., Georgiou, Y., Zhong, L., Zhou, H., Pospieszny, M.: Container Orchestration on HPC Systems. 2020 IEEE International Conference on Cloud Computing (CLOUD) (2020).
[ Abstract ] [ URL ]
 
Containerization demonstrates its efficiency in application deployment in cloud computing. Containers can encapsulate complex programs with their dependencies in isolated environments, hence are being adopted in HPC clusters. HPC workload managers lack micro-services support and deeply-integrated container management, as opposed to container orchestrators (e.g. Kubernetes). We introduce Torque-Operator (a plugin) which serves as a bridge between HPC workload managers and container Orchestrators.

3.
Georgiou, Y., Zhou, N., Zhong, L., Hoppe, D., Pospieszny, M., Papadopoulou, N., Nikas, K., Nikolos, O.L., Kranas, P., Karagiorgou, S., Pascolo, E., Mercier, M., Velho, P.: Converging HPC, Big Data and Cloud technologies for precision agriculture data analytics on supercomputers. Springer LNCS (2020).
 

4.
Hoppe, D., Zhong, L., Andersson, S., Moise, D.: On the Detection and Interpretation of Performance Variations of HPC Applications. Sustained Simulation Performance 2018 and 2019. S. 41–56. Springer (2020).
 

2019 [ to top ]

1.
Gale, L., Valentin, B., Boulahya, H., Vingione, G., Scarpino, G., Marzell, L., Pettengell, T., Gialampoukidis, I., Hoppe, D., Zhong, L.: H2020 EOPEN EASING COPERNICUS DATA & SERVICES EXPLOITATION. (2019).