EXCELLERAT successfully closes its first chapter
The first funding phase of EXCELLERAT – the European Centre of Excellence for Engineering Applications in the field of high-performance computing (HPC) – has come to an end on 31st May 2022. Over the past three and a half years, the Centre’s consortium consisting of 13 European partners provided expertise on how data management, data analytics, visualisation, simulation-driven design and co-design could benefit engineering, in particular in the aerospace, automotive, energy and manufacturing sectors. Overall, EXCELLERAT’s work strongly focused on improving computational efficiency, dynamic mesh adaptation, load balancing, scalable data handling, usability (visualisation and workflow tools), as well as investigating novel architectures and opportunities for co-design and developing more efficient numerical methods.
“The EXCELLERAT Centre of Excellence made progress towards enabling Exascale technologies by working on six reference applications: Nek5000, Alya, AVBP, TPLS, FEniCS, and CODA”, said Dr. Bastian Koller, Managing Director of HLRS and Coordinator of EXCELLERAT, “We are proud of our achievements, best practices, and the strong collaboration network we could build and continuously extend throughout the project runtime. This has collectively allowed us to show how engineering in Europe will be able to benefit from Exascale capabilities.”
For the open-source high-order spectral element computational fluid dynamics (CFD) code Nek5000, the team incorporated Adaptive Mesh Refinement (AMR), which enabled previously unaffordable simulations – with the goal of making the mesh much finer around the regions of interest and do this during the runtime of the simulation in response to the wider physics it is developing. The uncertainty quantification toolkit UQit could be integrated to further extend the accuracy of simulations. The group of researchers also improved the parallel performance of the AMR solver as tested on the Dardel and Lumi supercomputers showing good strong scaling. The visualisation tools Catalyst and Vistle were also integrated, which helps to combine in situ post processing with virtual reality capabilities. In addition, full-scale high fidelity runs of use cases collecting turbulence statistics and analysing complex flow dynamics were achieved.
Additionally, the consortium made advances on Alya – an HPC multiphysics code that uses the Finite Element Method for large scale problems with complex geometries. They developed a fully parallel workflow removing computational bottlenecks to ensure that every part of the engineering workflow benefits from parallel computing. A lot of work was also done on load balancing at node and system level, redistributing the work at runtime to enable much better scaling. Developing and incorporating adaptive load balancing was a major part of the project. Moreover, porting to GPU and emerging technologies was enabled, which is an important step for future Exascale architectures.
Regarding the finite element/volume 3D compressible flow solver AVBP, EXCELLERAT’s researchers focused on adaptive mesh refinement and adapting the code for GPUs. A collaboration with the EPEEC project produced an AMR tool called “treeadapt”, which was incorporated into AVBP. This greatly improved the efficiency of the computation by a factor of two speedup compared to just having the static mesh.
The consortium further developed the open-source solver TPLS for the incompressible Navier-Stokes equations for two phase fluid flows. TPLS did not perform well on GPUs before, since the PETSc libraries were not optimised for GPUs yet. Hence, the EXCELLERAT team developed a native GPU version of TPLS. They used HDF5 to improve I/O scalability by removing bottlenecks. A new method for identifying the interface between fluids was developed, which enabled other physical phenomena to be modelled. Temperature and proportion of vapour were added in the gaseous phase as degrees of freedom. Finally, a good parallel efficiency of up to 30,000 cores was demonstrated.
EXCELLERAT also made advancements in the mesh generation and refinement of FEniCS, an open-source high-level problem-solving environment. Testing was done using the DrivAER standard geometry. A problem of contact between wheels and ground within the mesh could be solved to make sure it was numerically stable.
A group of EXCELLERAT’s researchers integrated and further optimised the Sparse Linear Systems Solver (Spliss) in the CFD solver for aircraft design, CODA. Numerical stability was then tested on GPUs (NVIDIA V100 and A100) and the code was evaluated on the Juwels Booster System at Jülich Supercomputing Center with over 70% parallel efficiency.
Together with selected new potential partners, the major part of the EXCELLERAT consortium submitted a proposal for the EuroHPC Joint Undertaking’s Call “Centres of Excellence for HPC Applications (Horizon-Europe-JU-2021-CoE-01)”, Topic “Centres of Excellence for supporting supercomputing applications for Science and Innovation”, with the aim to acquire funding for a second project phase, which would start in January 2023 if funding gets approved.