Success story # Highlights:
- Simulation optimisation
- mesh quality
- Industry sector: Global challenges, environmental applications
- Key codes used: Urban Air Pollution Pilot
Organisations & Codes Involved:
The HiDALGO Centre of Excellence – HPC and Big Data Technologies for Global Systems – is a European project funded by the Horizon 2020 Framework Programme of the European Union, carried out by 13 institutions from seven countries. HiDALGO addresses major global challenges by enabling highly accurate simulations, data analytics, and data visualisation that predict the impact of global decisions with their dependencies, and also provides knowledge on how to integrate the various workflows as well as the corresponding data.
The Urban Air Pollution (UAP) application is a pilot application within HiDALGO that addresses the assessment of urban air quality with regard to the 2008/50/EC directive of the European Commission. It can be coupled with a traffic camera network and pollution monitoring devices, as implemented in the city of Győr. The central part of the UAP application is an HPC-framework for simulating the air flow in cities by taking into account real 3D geographical information of the city, applying highly accurate computational fluid dynamics (CFD) simulation on a highly resolved mesh (1-2 m resolution at street level), and using weather forecasts and reanalysis data as boundary conditions. Emission is computed from the weakly coupled traffic simulations and general emission data of other sources. For the demonstration area, the city of Győr, Hungary, a traffic monitoring sensor network with a plate recognition camera system will be developed. The monitoring system will be completed with affordable air quality sensors as well.
Topic of collaboration:
HiDALGO developed the Urban Air Pollution Pilot (UAP) and asked the EXCELLERAT team for support during the pilot’s application improvement phase. The primary aim of the collaboration was to improve UAP’s performance on HPC systems, that uses OpenFOAM in its CFD module for pollution dispersion simulation. This included three components: On the first level, the team aimed to improve the mesh quality with snappyHexMesh. The second part of the collaboration dealt with the optimisation of the steady state simulation using simpleFoam and then the third, improving transient simulation with pimpleFoam.
Results of collaboration
The team used snappy HexMesh for meshing instead of octree mesher. Advanced pre-processing enabled finer surface and edge fit. Also, higher quality mesh comes with lower generation time for same resolution.
For the steady state simulation with simpleFoam wide range of numerical schemes were investigated for stability, runtime, and parallel performance. Unnecessary IO was turned off, and mesh cell index renumbering and multilevel decomposition was issued and incorporated into the automatic workflow. Improvements were most significant at lower mesh sizes.
For investigations of the transient simulation with pimpleFoam, software runtime was more thoroughly examined by using code instrumentation. Focusing on the most demanding equations, various solvers were tested. We could achieve higher speedups with higher optimal core count and even lower runtime where the optimum was previously.
What did we manage to do together that we could not have done separately
EXCELLERAT proved to be an invaluable partner in helping and consulting HiDALGO for parallel performance optimisation. The issued optimisations proved to have a factor 5 improvement of the speedup on our cluster, from 18 to 102 for a 1 million cell mesh. Larger meshes of almost 10 million cells also had a speedup improvement from 49 to 77. This greatly improved parallel efficiency, from around 8% to more than 47%. This success was efficiently achieved within a few months thanks to the excellent exchange of knowledge and cooperation. Previously we had several performance issues, as development was focused on feature integration and automatisation.
Future collaboration on the case study is already underway, progress is discussed in a weekly schedule. Deep performance analysis is done with code instrumentation, with focus of hot point identification for performance bottlenecks. We also plan on testing alternative compilers and alternative job assignment methods to inhibit penalties from high process count and a very low cell count per process. Later, we plan to investigate feasibility of UAP on an Exascale system, for which we want to incorporate results from the exaFoam project.
UNIQUE VALUE OF EACH SERVICE:
- OpenFOAM is open source, so can be used in a HPC environment without scalability limitation.
- CFD open-source software OpenFOAM®. with an open Field Operation And Manipulation toolbox/library
- Training about CFD software OpenFOAM®
Refrence: Zoltán Horváth, Andreas Ruopp, Ákos Kovács, and Bence Liszkai. 2021. Multi-scale Modelling of Urban Air Pollution with Coupled Weather Forecast and Traffic Simulation on HPC Architecture. In The International Conference on High Performance Computing in Asia-Pacific Region Companion (HPC Asia 2021 Companion), January 20–22, 2021, Virtual Event, Republic of Korea. ACM, New York, NY, USA, 2 pages. https://doi.org/10.1145/3440722.3440917