Parallel mumps solver and custom solver

General
1

Hello everyone,

I am working on a large multi-physics problem. My code was not originally written with parallelism in mind, but I would like to speed up the linear solve phase by using MUMPS in parallel.

Is it possible to parallelize only the MUMPS solver (via MPI) without rewriting the entire code to be parallel? In other words, can I benefit from the internal parallelism of MUMPS by simply running my executable with mpirun, even if the matrix assembly remains sequential?

Also, I am starting to develop my own solvers for specific physics. I am already using real_rhs() and real_tangent_matrix() to extract the system, but I would appreciate any examples or resources on how to structure a clean custom solver in the GetFEM workflow.

Thank you in advance for your help and guidance.

Best regards,

René

2

I assume you have read

this documentation requires some updates but it is in general helpful.

If you stick to using GetFEM to compute the rhs and tangent matrix, your code should already be able to run in parallel, both the assembly and the linear solve (with MUMPS, the latter requires the first).

MUMPS has a very nice architecture which makes parallelization so easy. If K_i is the part of the tangent matrix that the i-th process sees, then the overall tangent matrix should be:

K = K_1 + K_2 + …+ K_N

where K_1, K_2 etc. can have overlapping entries as long as the sum of these entries matches the corresponding entry in K.

If you compile GetFEM with GETFEM_PARA_LEVEL=2, then all meshes you use will be partitioned with Metis. This will define internally as many mesh regions as the processes, let’s call them Metis_1, Metis_2, … Metis_N.

Then whenever you assemble a term on region let’s say -1 (entire mesh), process “i” will perform the assembly only on region Metis_i instead.

If you perform an assembly on your own defined region, e.g. RG_RED, then process “i” will perform the assembly on the intersection of RG_RED and Metis_i

At the end in a GetFEM model, vectors are broadcasted to all processes, matrices do not. That’s how GetFEM prepares the data in the format MUMPS expects them. This parallelization is basically totally automated. If you write your entire model with GWFL expressions, and you do not use any strange model bricks, you need to do nothing else than compiling the parallel version of GetFEM.

3

Hello,

Thank you for your clear explanation.

I would just like to confirm that I am indeed in the case where the assembly of the tangent matrix and the RHS is automatically handled by GetFEM, and thus can be parallelized with MUMPS without rewriting my whole code.

In my case:

  • I use a getfem::model object (through the Python interface).
  • I don’t use “model bricks”, but instead describe the physics using GWFL expressions like:
md.add_nonlinear_term(..., expression)
  • I actually solving with:
md.solve('noisy',
         'max_iter', 100,
         'max_res', 5e-4,
         'lsolver', 'mumps',
         'lsearch', 'simplest',
         'alpha max ratio', 1.0,
         'alpha mult', 0.8,
         'alpha min', 1e-4)
  • I also extract the tangent matrix and RHS with:
K = md.tangent_matrix()
F = md.rhs()
dx = gf.linsolve_mumps(K, F)
  • And I would like to solve with this
dx = gf.linsolve_mumps(K, F)
  • I execute the code using:
mpirun -np 4 python script.py

My understanding is:
In this setup, GetFEM assembles the tangent matrix and RHS in parallel, using Metis partitions (Metis_1, Metis_2, …, Metis_N) associated with each MPI process.
MUMPS then solves the linear system in distributed mode, with no additional MPI handling on my side.
I do not need to change anything in my code logic, as long as I compile GetFEM with GETFEM_PARA_LEVEL=2 and MUMPS is linked with MPI.

Could you please confirm that this understanding is correct?
Also, is there anything important I should know if I want to go one step further by using linsolve_mumps() offers?

Thank you again for your help,
Best regards,

4

yes, I can confirm that for md.solve()

For linsolve_mumps, you just need to be on the latest version 5.4.4 or the current development version, which include this commit:

For even more advanced cases, where a matrix needs to be factorized once and used in multiple solves, I have recently (post 5.4.4) introduced a new mumps_context object, that you can learn more about by studying the examples found here:

5

Thank you very much for your confirmation.

Indeed, MPI seems to be working on my side: when I run a code with mpirun, the same process is executed on the number of processors I choose.

However, when I try the example you provided, I get the following error:

Traceback (most recent call last):
  File "/home/rene.djoumessi/getFemSim/code_c_real_col/TestMPI.py", line 36, in <module>
    if gf.util_mpi_parallelism_level() >= 2:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'getfem' has no attribute 'util_mpi_parallelism_level'

This seems to indicate that my current GetFEM installation was not compiled with GETFEM_PARA_LEVEL=2.

Thanks again for your help
I’ll reach out again if I have further questions.
Best.

6

this simply means that the util_mpi_parallelism_level function was added to GetFEM after the version you are using.

1 Like
7

Thank you again for your help. One more clarification, please.

For now, I’m using version 5.3 of GetFEM, which I installed by cloning the repository some time ago. I’m not sure if I can use this version directly for parallel MUMPS support, or if I need to recompile GetFEM with version 5.4 or the development branch.

Thank you.

8

version 5.3 is too old

9

Dear Konstantinos,
I’ve try to compile the version 5.4 in our cluster and I’m getting the following error after ```make install``

 /usr/bin/mkdir -p '/home/rene.djoumessi/share/getfem/lib/python3.12/site-packages/getfem'
 /usr/bin/install -c -m 644 _getfem.cpython-312-x86_64-linux-gnu.so '/home/rene.djoumessi/share/getfem/lib/python3.12/site-packages/getfem'
Traceback (most recent call last):
  File "<string>", line 2, in <module>
ModuleNotFoundError: No module named 'imp'
make[4]: *** [Makefile:475: install-nodist_gfpyexecPYTHON] Error 1
make[4]: Leaving directory '/home/rene.djoumessi/libs/getfem-5.4.4/interface/src/python'
make[3]: *** [Makefile:568: install-am] Error 2
make[3]: Leaving directory '/home/rene.djoumessi/libs/getfem-5.4.4/interface/src/python'
make[2]: *** [Makefile:838: install-recursive] Error 1
make[2]: Leaving directory '/home/rene.djoumessi/libs/getfem-5.4.4/interface/src'
make[1]: *** [Makefile:455: install-recursive] Error 1
make[1]: Leaving directory '/home/rene.djoumessi/libs/getfem-5.4.4/interface'
make: *** [Makefile:598: install-recursive] Error 1
rene.djoumessi@musampro3:~/libs/getfem-5.4.4$ ~/share

This is a part of what I have on the terminal. the installation works under python 3.11 but failled under python 3.12, asking the following module imp
Please any idea?
Thanks in advance
Rene

10

this looks like some misconfiguration in your cluster, does not look like a GetFEM specific issue.