The use of in-situ analysis tools by EXCELLERAT
Paraview-Catalyst is widely used to perform in-situ visualisation, for example the OpenFOAM and Code-Saturne HPC codes have successfully used this software. In order to use Catalyst, a given code should be prepared. First, an appropriate version of Paraview-Catalyst must be available on the computer system on which the code will be executed. Then the code must be instrumented to transfer its data towards Catalyst. Finally, in most cases, a Python script automatically produced by Paraview must be prepared. This script contains the instructions which will be executed during the run time of the simulation.
Usually the main objective of these instructions is to manipulate and display scientific data through the use of the Python VTK API. One of the main advantages of this approach is related to the fact that, once the code has been instrumented, numbers of different analyses can be performed by simply changing the script used. This great advantage is overshadowed by the complexity of the numerous details to be considered during the compilation of Catalyst, ie OpenGL, Mesa, and Python (along with their respective dependencies) must already be available before the compilation.
In order to facilitate parts of the procedure described above, version 5.6 of Paraview has been modified to reduce the need for third-party software components, so filters must be implemented by using C++ instead of the scripts created by Paraview. As a consequence of these modifications, scalability and portability are encouraged, while rendering is obviated. This tuned version of Paraview, named Decaf Paraview, is an HPC in-situ data analysis tool that is currently being developed. The main goal of this tool is related to its capability in handling data arising from the C++ VTK API. As a result, data can be analysed in real time by using external algorithms originally unavailable in VTK.
The in-situ tool is used to analyse two relevant cases. The first one is related to the development of a parallel algorithm for the time tracking of coherent structures in turbulent flows. This algorithm has been tested on KTH’s supercomputer Beskow using up to 4,096 processors. In the second case, two codes are run by using the Multiple-Program, Multiple-Data (MPMD) programming model. Here an interpolation procedure is performed. A given source mesh is used to perform a turbulent simulation, while a principal component analysis (PCA) will be carried out through a destination mesh.
Decaf Paraview is currently used in three main collaborations related to EXCELLERAT’s core codes
- BSC-HLRS. Sensei in Alya. The main goal is the instrumentation of Alya in order to use Sensei.
- Cineca-KTH. Catalyst in Nek5000. The objective is to improve the in-situ visualisation that is already available in Nek5000.
- Fraunhofer-KTH. Data analytics and Catalyst in Nek5000. The main goal is to combine machine learning algorithms with the in-situ analysis already available in Nek5000
In the first case, previous to introducing Sensei in Alya, the instrumentation with Catalyst should be prepared. In this sense, a first version of this instrumentation has already been sent. This instrumentation will be tested on BSC’s supercomputer MareNostrum with a combustion case. The modified version of Paraview 5.6 (Decaf) has also been sent, along with the corresponding compilation instructions. In the second and third cases, a version of Nek5000 with a first Catalyst instrumentation has already been sent. A two-dimensional computation flow around a circular cylinder was included as an example of the in-situ visualisation. In the third case, in addition to the above mentioned, an example with the capability of performing an interpolation between two independent codes has been prepared. This example corresponds to the execution of a Multiple-Program Multiple-Data (MPMD) parallel model.
- Miguel Zavala, KTH
This work has been supported by the EXCELLERAT project which has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No 823691.
Many thanks to Jérôme Lecomte who kindly provided the community with the nice circlify package, which enables the circular packing to be computed, and codemetrics, which gives a lot of insight into the code we wrote.