Hello All,
Is it possible to parallelize matrix assembly in GetFEM? I have found that assembly has consistently been the bottleneck for large 6D problems, and am curious if that is possible.
Sincerely,
Eric A. Comstock
Hi, yes, assembly operations are in general trivial to parallelize and of course this option is already available in GetFEM. You just need to compile GetFEM with either the --enable-paralevel=1 option in the configure command, for parallelizing the assembly only, or with --enable-paralevel=2 for parallelization of assembly and linsolve with the parallel version of MUMPS.
you can compare the sequential and parallel build and runs scripts from
to see how exactly I use either sequential or para-level 2 version on our cluster.
Check also the official documentation at
Parallelization relies on mesh partitioning with Metis. I don’t think that Metis can partition 6d meshes, so probably you will need to implement some workaround there. I can help with that if you know how you want to partition your mesh.
Hello Kostas,
Thank you for the resources and help! I will look into them, and let you know if there are any issues.
Sincerely,
Eric A. Comstock
Hello Kostas,
Okay - I got a parallel build operational and working. What commands do I need to use in Python to ensure that the assembly is parallel? The reason why I am focused on assembly is because my solve is only ~0.1% of the processing. It turns out that 6D requires a huge number of quadrature points per simplex (heptapeton), and a huge number heptapetons per hypercube (hexarract) and DoF. That is why it takes so long compared to the solve in my problem.
Sincerely,
Eric A. Comstock
Hello Kostas,
To clarify, I am trying to parallelize assembly, and then run my own code for the solver. So I am using right now:
md.assembly("build_all")
to make the rhs and the matrix, and then using my own code to solve them, before injecting back to GetFEM with
md.to_variables(solution)
Right now I am running with
mpirun -n 4 python3
and it shows four programs running, and four sets of boundary conditions being added. The program then ends with the injection back to GetFEM failing due to “inconsistent shapes.”
Sincerely,
Eric A. Comstock
to begin with, try some sanity tests, do an assembly of some simple expression
VEC = gf.asm_generic(mim, 1, "1*Test_a", -1, "a", 1, mf, np.zeros(mf.nbdof()))
where mf is just some scalar mesh_fem, and save VEC in a file.
Repeat the test with mpi run -n 1, mpi run -n 2, mpi run -n 4 and check that result doesn’t change.
1 Like
Hello Kostas,
I got it working - thank you so much for your help!
Sincerely,
Eric A. Comstock