GetDP, PETSc and hybrid mode of MUMPS (MPI+OpenMP)
Hello! I'm not sure if the problem is from GetDP, but I can't localize it yet.
Starting from PETSc 3.17.0, GetDP began to "hang" before starting MUMPS if the MPI + OpenMP hybrid mode is set (for example, the -mat_mumps_use_omp_threads
option is set to 3, and GetDP is started via mpirun -np 3
; https://petsc.org /main/docs/manualpages/Mat/MATSOLVERMUMPS/).
Example:
...
E n d P r e - P r o c e s s i n g
P r o c e s s i n g . . .
Info : InitSolution[Sys_Mag]
Info : GenerateJac[Sys_Mag]
Info : SolveJac[Sys_Mag]
_hang here_
If you add "-v 10" for getdp
command, then the last output line will also be an empty line after "SolveJac". If you add "-info" into .petscrc file then the last line will be empty after something like:
[0] PetscCommDuplicate(): Using internal PETSc communicator 93875196336640 93875202320384
[1] PetscShmCommGet(): Shared memory rank 0 global rank 0
[1] PetscShmCommGet(): Shared memory rank 1 global rank 1
[1] PetscShmCommGet(): Shared memory rank 2 global rank 2
[0] PetscShmCommGet(): Shared memory rank 0 global rank 0
[0] PetscShmCommGet(): Shared memory rank 1 global rank 1
[0] PetscShmCommGet(): Shared memory rank 2 global rank 2
[2] PetscShmCommGet(): Shared memory rank 0 global rank 0
[2] PetscShmCommGet(): Shared memory rank 1 global rank 1
[2] PetscShmCommGet(): Shared memory rank 2 global rank 2
This behavior is observed on all versions of PETSc branch 3.17, regardless of the version of GetDP or the problem being tested. Tested on Ubuntu 20.04 and 22.04.
PETSc configure command: ./configure --prefix=/opt/petsc --with-openmp --with-hwloc --with-mpi --download-mumps --download-scalapack --download-slepc --download-scotch --download-ptscotch --with-debugging=0
GetDP config:
getdp --info
Version : 3.5.0-git-1c7570cd
Build OS : Linux64
Build options : 64Bit Arpack[contrib] Blas Gmsh Gsl Kernel Lapack MPI OpenMP PETSc PeWe Python SLEPc
PETSc version : 3.17.3 (real arithmetic)
Gmsh lib version : 4.9.5 (20220314)
Gmsh lib options : 64Bit ALGLIB[contrib] ANN[contrib] Bamg Blossom Cgns DIntegration Dlopen DomHex Eigen[contrib] Fltk GMP Gmm[contrib] Hxt Jpeg Kbipack LinuxJoystick MathEx[contrib] Mesh Metis[contrib] Mpeg Netgen ONELAB ONELABMetamodel OSMesa OpenACC OpenCASCADE OpenCASCADE-CAF OpenGL OpenMP OptHom Parser Plugins Png Post QuadMeshingTools QuadTri Solver TCMalloc TetGen/BR Voro++[contrib] WinslowUntangler Zlib
There are some possibly related lines in the PETSc 3.17.0 changelog (but I have no idea what this might mean in practice):
- Reuse PETSc-installed hwloc when installing OpenMPI
- Add PetscCommGetComm() and PetscCommRestoreComm() to allow reuse of MPI communicator with external packages, as some MPI implementations have broken MPI_Comm_free()
More information about PETSc configuration:
Compilers:
C Compiler: mpicc -O3 -pipe -fopenmp -march=native -fPIC -fopenmp
Version: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
C++ Compiler: mpicxx -O3 -pipe -fopenmp -march=native -fopenmp -std=gnu++17 -fPIC -fopenmp
Version: g++ (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Fortran Compiler: mpif90 -fPIC -Wall -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -march=native -O3 -fopenmp -fopenmp
Version: GNU Fortran (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Linkers:
Shared linker: mpicc -fopenmp -fopenmp -shared -O3 -pipe -fopenmp -march=native -fPIC -fopenmp
Dynamic linker: mpicc -fopenmp -fopenmp -shared -O3 -pipe -fopenmp -march=native -fPIC -fopenmp
Libraries linked against: -lrt -lquadmath -lstdc++ -ldl
BlasLapack:
Library: -llapack -lblas
Unknown if this uses OpenMP (try export OMP_NUM_THREADS=<1-4> yourprogram -log_view)
uses 4 byte integers
MPI:
Version: 3
mpiexec: /usr/bin/mpiexec
Implementation: openmpi
OMPI_VERSION: 4.0.3
hwloc:
Version: 2.1.0
Library: -lhwloc
X:
Library: -lX11
openmp:
Version: 201511
pthread:
cmake:
Version: 3.16.3
/usr/bin/cmake
slepc:
Includes: -I/opt/petsc/include
Library: -Wl,-rpath,/opt/petsc/lib -L/opt/petsc/lib -lslepc
PTScotch:
Version: 6.1.2
Includes: -I/opt/petsc/include
Library: -Wl,-rpath,/opt/petsc/lib -L/opt/petsc/lib -lptesmumps -lptscotchparmetis -lptscotch -lptscotcherr -lesmumps -lscotch -lscotcherr
regex:
MUMPS:
Version: 5.4.1
Includes: -I/opt/petsc/include
Library: -Wl,-rpath,/opt/petsc/lib -L/opt/petsc/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord
uses OpenMP; use export OMP_NUM_THREADS=<p> or -omp_num_threads <p> to control the number of threads
scalapack:
Library: -Wl,-rpath,/opt/petsc/lib -L/opt/petsc/lib -lscalapack
hpddm:
Version: 2.2.1
Includes: -I/opt/petsc/include
sowing:
Version: 1.1.26
/mnt/500GB/Sources/petsc/petsc-3.17.3/arch-linux-c-opt/bin/bfort
Language used to compile PETSc: C
PETSc:
PETSC_ARCH: arch-linux-c-opt
PETSC_DIR: /mnt/500GB/Sources/petsc/petsc-3.17.3
Prefix: /opt/petsc
Scalar type: real
Precision: double
Support for __float128
Integer size: 4 bytes
Single library: yes
Shared libraries: yes
Memory alignment from malloc(): 16 bytes
Using GNU make: /usr/bin/make
An example of a simple test task is in the attached files. But the problem is reproduced on any task. .petscrc