Converting mesh to APF fails for large mesh about pumgen

[davschneller]

Hi, thanks; I've updated the branch so that it includes cstddef in some more positions; the error should (hopefully) not occur anymore.

[davschneller]

You could try the branch https://github.com/SeisSol/PUMGen/tree/davschneller/bypass-apf . I've tried to get around using PUMI/apf (which in total seems to be a bit of a relic from the Netcdf mesh era?); however, the code may not completely be optimized for memory consumption yet. But maybe it works now already; on small meshes it does.

[Thomas-Ulrich]

Thank you. I cannot compile with gcc/12.2

     72    /import/exception-dump/ulrich/spack/lib/spack/env/gcc/g++ -DH5_BUILT_AS_DYNAMIC_LIB -DLOG_LEVEL=2 -DPARALLEL -DUSE_HDF -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE
            -D_POSIX_C_SOURCE=200809L -I/tmp/ulrich/spack-stage/spack-stage-pumgen-bypass-apf-66in2kth545lc272udok4sxexxp4laib/spack-src/src -I/tmp/ulrich/spack-stage/spack-stage-pumgen-bypass-apf-66in2kth545
           lc272udok4sxexxp4laib/spack-src/submodules -I/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/pumi-2.2.8-mrdynb2pr2irnayun3j7g53itoxsdrjj/include -isystem /import/
           exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/hdf5-1.12.2-xwxzdqakdlyy5sknts5wilegc5dlrboi/include -isystem /import/exception-dump/ulrich/myLibs/spack-packages/linux-de
           bian11-zen2/gcc-12.2.0/openmpi-4.1.5-ab5rikqfa63jmfrqq54jugtlhyw3iu7d/include -O2 -std=c++17 -fopenmp -pthread -MD -MT CMakeFiles/pumgen.dir/src/aux/MPIConvenience.cpp.o -MF CMakeFiles/pumgen.dir/s
           rc/aux/MPIConvenience.cpp.o.d -o CMakeFiles/pumgen.dir/src/aux/MPIConvenience.cpp.o -c /tmp/ulrich/spack-stage/spack-stage-pumgen-bypass-apf-66in2kth545lc272udok4sxexxp4laib/spack-src/src/aux/MPICo
           nvenience.cpp
     73    In file included from /tmp/ulrich/spack-stage/spack-stage-pumgen-bypass-apf-66in2kth545lc272udok4sxexxp4laib/spack-src/src/aux/MPIConvenience.cpp:1:
  >> 74    /tmp/ulrich/spack-stage/spack-stage-pumgen-bypass-apf-66in2kth545lc272udok4sxexxp4laib/spack-src/src/aux/MPIConvenience.h:6:75: error: 'size_t' in namespace 'std' does not name a type
     75        6 | void sparseAlltoallv(const void* sendbuf, const int* sendsize, const std::size_t* senddisp,
     76          |                                                                           ^~~~~~
  >> 77    /tmp/ulrich/spack-stage/spack-stage-pumgen-bypass-apf-66in2kth545lc272udok4sxexxp4laib/spack-src/src/aux/MPIConvenience.h:8:33: error: 'size_t' in namespace 'std' does not name a type
     78        8 |                      const std::size_t* recvdisp, MPI_Datatype recvtype, MPI_Comm comm);
     79          |                                 ^~~~~~
  >> 80    make[2]: *** [CMakeFiles/pumgen.dir/build.make:205: CMakeFiles/pumgen.dir/src/aux/MPIConvenience.cpp.o] Error 1
     81    make[2]: *** Waiting for unfinished jobs....
     82    make[2]: Leaving directory '/tmp/ulrich/spack-stage/spack-stage-pumgen-bypass-apf-66in2kth545lc272udok4sxexxp4laib/spack-build-66in2kt'
  >> 83    make[1]: *** [CMakeFiles/Makefile2:88: CMakeFiles/pumgen.dir/all] Error 2
     84    make[1]: Leaving directory '/tmp/ulrich/spack-stage/spack-stage-pumgen-bypass-apf-66in2kth545lc272udok4sxexxp4laib/spack-build-66in2kt'
  >> 85    make: *** [Makefile:139: all] Error 2

[Thomas-Ulrich]

The previous setup worked on 30 nodes.
During the volume mesh generation the ram was around 680Gb, and during conversion, it peaked around 715 (just loosly monitoring).
In short, issue fixed! Thanks again!

Tue Oct 31 11:52:39, Info:  AR statistics:
Tue Oct 31 11:52:50, Info:  AR max: 19.1125
Tue Oct 31 11:52:50, Info:  AR (target: < ~10):
Tue Oct 31 11:52:50, Info:    [ 0.00 , 2.00 ): 593689
Tue Oct 31 11:52:50, Info:    [ 2.00 , 4.00 ): 728857614
Tue Oct 31 11:52:50, Info:    [ 4.00 , 6.00 ): 22297227
Tue Oct 31 11:52:50, Info:    [ 6.00 , 10.00 ): 2089021
Tue Oct 31 11:52:50, Info:    [ 10.00 , 20.00 ): 30328
Tue Oct 31 11:52:50, Info:    [ 20.00 , 40.00 ): 0
Tue Oct 31 11:52:50, Info:    [ 40.00 , 100.00 ): 0
Tue Oct 31 11:52:50, Info:    [ 100.00 ,inf): 0
Tue Oct 31 11:52:50, Info:  Iterating over mesh to get data...
Tue Oct 31 11:52:50, Info:  Counting part 0 / 1
Tue Oct 31 11:52:50, Info:  Local cells: 6352716
Tue Oct 31 11:52:50, Info:  Local vertices: 1112747
Tue Oct 31 11:52:50, Info:  Local vertices (with duplicates): 1288241
Tue Oct 31 11:52:50, Info:  Processing part 0 / 1
Tue Oct 31 11:52:50, Info:  Vertices: 0 to 1112747
Tue Oct 31 11:52:55, Info:  Local vertices (really, now): 258298
Tue Oct 31 11:52:55, Info:  Processing part 0 / 1
Tue Oct 31 11:52:55, Info:  Connectivity: 0 to 6352716
Tue Oct 31 11:52:56, Info:  Groups
Tue Oct 31 11:52:57, Info:  Boundaries
Tue Oct 31 11:52:57, Info:  Parsed mesh successfully, writing output...
Tue Oct 31 11:53:05, Info:  Total cell count: 753867879
Tue Oct 31 11:53:05, Info:  Total vertex count: 128515465
Tue Oct 31 11:53:29, Info:  Minimum insphere found: 4.75306
Tue Oct 31 11:53:29, Info:  Writing cells
Tue Oct 31 11:54:07, Info:  Writing vertices
Tue Oct 31 11:54:11, Info:  Writing group information
Tue Oct 31 11:54:19, Info:  Writing boundary condition
Tue Oct 31 11:54:22, Info:  Writing XDMF file
Tue Oct 31 11:54:26, Info:  Finished successfully

[davschneller]

That is great news! Then I'll try to make the PR ready some time soon-ish; it should make APF completely optional for Simmodeler and any serial mesh file (GMSH etc. should also work without APF; though I'll have to test that).

The only thing that we'd really be missing are the options described around here:

PUMGen/src/pumgen.cpp

Lines 172 to 192 in 3fb1875

	if (alignMdsMatches(mesh))
	logWarning() << "Fixed misaligned matches";
	mesh->verify();
	// Dump mesh for later usage?
	const char* dumpFile = args.getArgument<const char*>("dump", 0L);
	if (dumpFile) {
	logInfo(PCU_Comm_Self()) << "Writing native APF mesh";
	mesh->writeNative(dumpFile);
	const char* modelFile = args.getArgument<const char*>("model", 0L);
	if (modelFile)
	gmi_write_dmg(mesh->getModel(), modelFile);
	}
	// Dump VTK mesh
	const char* vtkPrefix = args.getArgument<const char*>("vtk", 0L);
	if (vtkPrefix) {
	logInfo(PCU_Comm_Self()) << "Writing VTK mesh";
	apf::writeVtkFiles(vtkPrefix, mesh);
	}