Hello Kostas,
Thank you! I am running the profiling simulation you suggested.
Sincerely,
Eric Comstock
Hello Kostas,
I ran the profiling on the 4096-vertex case, and my results (sorted by cumulative time) were:
166534 function calls (163362 primitive calls) in 1270.054 seconds
Ordered by: cumulative time
List reduced from 1188 to 20 due to restriction <20>
ncalls tottime percall cumtime percall filename:lineno(function)
120/1 0.001 0.000 1270.075 1270.075 {built-in method builtins.exec}
1 0.000 0.000 1270.075 1270.075 test6D_mmm4.py:1(<module>)
1 0.241 0.241 1251.568 1251.568 test6D_mmm4.py:46(calc_F)
88 1235.487 14.040 1235.487 14.040 {built-in method getfem._getfem.getfem}
22 0.081 0.004 1148.595 52.209 /storage/home/hcoda1/9/ecomstock3/.conda/envs/getfem/lib/python3.11/site-packages/getfem/getfem.py:2805(set)
5 0.045 0.009 1146.208 229.242 /storage/home/hcoda1/9/ecomstock3/.conda/envs/getfem/lib/python3.11/site-packages/getfem/getfem.py:3691(add_source_term_brick)
12 0.002 0.000 43.005 3.584 /storage/home/hcoda1/9/ecomstock3/.conda/envs/getfem/lib/python3.11/site-packages/getfem/getfem.py:1395(outer_faces_with_direction)
12 0.002 0.000 43.003 3.584 /storage/home/hcoda1/9/ecomstock3/.conda/envs/getfem/lib/python3.11/site-packages/getfem/getfem.py:1167(get)
2 0.003 0.001 42.015 21.007 /storage/home/hcoda1/9/ecomstock3/.conda/envs/getfem/lib/python3.11/site-packages/getfem/getfem.py:2800(get)
1 0.001 0.001 41.851 41.851 /storage/home/hcoda1/9/ecomstock3/.conda/envs/getfem/lib/python3.11/site-packages/getfem/getfem.py:2941(solve)
160/3 0.002 0.000 18.309 6.103 <frozen importlib._bootstrap>:1165(_find_and_load)
160/3 0.001 0.000 18.308 6.103 <frozen importlib._bootstrap>:1120(_find_and_load_unlocked)
147/3 0.001 0.000 18.306 6.102 <frozen importlib._bootstrap>:666(_load_unlocked)
118/3 0.000 0.000 18.306 6.102 <frozen importlib._bootstrap_external>:934(exec_module)
388/6 0.000 0.000 17.913 2.985 <frozen importlib._bootstrap>:233(_call_with_frames_removed)
6 0.006 0.001 14.535 2.422 /storage/home/hcoda1/9/ecomstock3/.conda/envs/getfem/lib/python3.11/site-packages/getfem/getfem.py:52(generic_constructor)
6 14.527 2.421 14.527 2.421 {built-in method getfem._getfem.getfem_from_constructor}
1 0.000 0.000 12.865 12.865 /storage/home/hcoda1/9/ecomstock3/.conda/envs/getfem/lib/python3.11/site-packages/numpy/__init__.py:1(<module>)
375/9 0.001 0.000 12.660 1.407 {built-in method builtins.__import__}
253/42 0.001 0.000 12.653 0.301 <frozen importlib._bootstrap>:1207(_handle_fromlist)
so it looks like add_source_term_brick is likely the bottleneck. I also sorted by total time, and got:
166534 function calls (163362 primitive calls) in 1270.054 seconds
Ordered by: internal time
List reduced from 1188 to 20 due to restriction <20>
ncalls tottime percall cumtime percall filename:lineno(function)
88 1235.487 14.040 1235.487 14.040 {built-in method getfem._getfem.getfem}
6 14.527 2.421 14.527 2.421 {built-in method getfem._getfem.getfem_from_constructor}
26/24 7.018 0.270 7.579 0.316 {built-in method _imp.create_dynamic}
618 5.844 0.009 5.844 0.009 {built-in method posix.stat}
118 4.344 0.037 4.344 0.037 {built-in method io.open_code}
14 0.487 0.035 0.994 0.071 /storage/home/hcoda1/9/ecomstock3/.conda/envs/getfem/lib/python3.11/site-packages/getfem/getfem.py:2169(eval)
26/16 0.474 0.018 2.443 0.153 {built-in method _imp.exec_dynamic}
1 0.241 0.241 1251.568 1251.568 test6D_mmm4.py:46(calc_F)
7 0.195 0.028 0.277 0.040 /storage/home/hcoda1/9/ecomstock3/.conda/envs/getfem/lib/python3.11/site-packages/numpy/core/arrayprint.py:934(fillFormat)
3 0.175 0.058 0.175 0.058 {method 'dot' of 'numpy.ndarray' objects}
14 0.145 0.010 0.145 0.010 {built-in method builtins.compile}
118 0.138 0.001 0.138 0.001 {method 'read' of '_io.BufferedReader' objects}
52 0.087 0.002 0.087 0.002 /storage/home/hcoda1/9/ecomstock3/.conda/envs/getfem/lib/python3.11/site-packages/numpy/ma/core.py:894(__init__)
22 0.081 0.004 1148.595 52.209 /storage/home/hcoda1/9/ecomstock3/.conda/envs/getfem/lib/python3.11/site-packages/getfem/getfem.py:2805(set)
6 0.080 0.013 0.098 0.016 /storage/home/hcoda1/9/ecomstock3/.conda/envs/getfem/lib/python3.11/site-packages/numpy/core/getlimits.py:34(__init__)
5 0.045 0.009 1146.208 229.242 /storage/home/hcoda1/9/ecomstock3/.conda/envs/getfem/lib/python3.11/site-packages/getfem/getfem.py:3691(add_source_term_brick)
1 0.036 0.036 0.036 0.036 /storage/home/hcoda1/9/ecomstock3/.conda/envs/getfem/lib/python3.11/site-packages/numpy/core/function_base.py:24(linspace)
1150 0.035 0.000 0.035 0.000 {built-in method builtins.min}
42 0.034 0.001 0.034 0.001 {built-in method numpy.core._multiarray_umath.dragon4_scientific}
10 0.033 0.003 0.033 0.003 {built-in method time.localtime}
I have also noticed that the main “hard step” where most of the processing time is seems to be just before the output line
Trace 2 in getfem_models.cc, line 3463: Generic linear assembly brick: generic matrix assembly
By the way, my other simulations are still running, and it seems like only one processor is used. I have used getFEM/mumps on multiple cores before in previous simulations (Comstock, Eric & Romero-Calvo, Álvaro. (2025). External Plasma-Breathing Magnetohydrodynamic Spacecraft Propulsion. 10.2514/6.2025-2038.) at up to several million DoF, so I was wondering if the 6D nature of this problem (previous work was 5D) might be causing issues with parallelization.
Sincerely,
Eric Comstock
Hello Kostas,
I got the 46656-vertex case in 13 hours (48378.26371693611 seconds, to be specific), 40 times longer than the 4096-vertex case, with the same process being the bottleneck. I think something is scaling supralinearly in the boundary conditions with the number of vertices.
Sincerely,
Eric Comstock
Some comments:
-
weird that it consumes a significant amount of time in
add_source_term_brick
This function should just define this term, it should not do any computations -
I wonder about how you could use getfem/mumps on multiple cores before. The conda package does not have any kind of parallelization for GetFEM and neither I think for MUMPS. Probably your blas was using several threads before. You can still use a multithreaded blas but the gain will not be big.
-
In more than 3 dimensions you will have problems with using a parallel GetFEM (to split all assembly procedures across several cpus) because GetFEM uses Metis for partitioning the mesh for the different cpus, and I don’t think Metis can partition higher dimension meshes. Check that.
1 Like
Hello Kostas,
Thank you again for your help!
I am wondering whether add_source_term_brick might have some sequential procedure that worked well up to 4D and 5D, but gets very complicated in 6D. It seems to be using most of the time of the code, so if that is the case and there is some solution, it would dramatically improve the feasibility of the main simulation I want to run. I am trying to see if a test case with all Dirichlet conditions has the same problem, or if it might be something else. I can run the code in 4 minutes without the profiler, and it takes about 20 minutes with it, so right now I am just using logging to determine which processes take longest. I will then run any interesting ones with the profiler and let you know of my findings.
That is probably the case, then - I mainly thought it was running on multiple cores since it showed multiple CPU usage on Task Manager.
I checked, and it seems Metis does not support partitioning above 3D. Since my mesh is hyperrectangular, might there be some way to input a partition manually? If I want to break it up over 64 CPUs, for example, I should just be able to slice it in two along each axis.
Sincerely,
Eric Comstock
I cannot imagine that.
For MPI parallelization we do not have such an interface but it is quite easy to use mesh regions to implement the same functionality semi-manually. For OpenMP parallelization I am not sure, I would need to check the implementation.
In any case it is essential for you to get a self-built installation of GetFEM from source code so that you have full control.
1 Like
Hello Kostas,
Thank you for the help!
Got it. I am trying to get a wsl-ubuntu installation of getFEM working (using the instructions from the guide here: How to install from sources on Linux — GetFEM), but I am getting stuck on the configure command for getFEM, which results in
checking for smumps_c.h... yes
checking for dmumps_c.h... yes
checking for cmumps_c.h... yes
checking for zmumps_c.h... yes
checking for library containing smumps_c... no
checking for library containing dmumps_c... no
checking for library containing cmumps_c... no
checking for library containing zmumps_c... no
configure: error: Neither MUMPS nor SuperLU was enabled. At least one linear solver is required.
Is this also a result of mumps being improperly configured? I remember having a similar problem when I was trying to install it on Windows.
My configure was
./configure --with-pic=TRUE
and I have libmumps-dev installed as my mumps solver.
Sincerely,
Eric Comstock
this is my configure command on a similar system:
./configure --prefix=/tmp/opt --with-blas=openblas --with-optimization=-O2 --with-pic --enable-python --disable-matlab --disable-superlu --enable-mumps --disable-openmp
change prefix to whatever path under your home folder you want to install GetFEM in
Hello Kostas,
Thank you! I am getting a slightly different result now,
checking for smumps_c.h... yes
checking for dmumps_c.h... yes
checking for cmumps_c.h... yes
checking for zmumps_c.h... yes
checking for library containing smumps_c... no
configure: error: The function smumps_c couldn't be found in the provided MUMPS libraries.
What is odd is that it seems to be able to find smumps_c when it checks for the file, but then throws an error when it cannot find it.
Sincerely,
Eric Comstock
check if Ubuntu has a libmumps-seq-dev package and install it
1 Like
Hello Kostas,
Thank you! Switching to libmumps-seq-dev seems to have fixed it.
Sincerely,
Eric Comstock
to run getfem python scripts with your new installation you need to run them with.
LD_LIBRARY_PATH=/../lib/ PYTHONPATH=/../lib/python3.13/site-packages/getfem python3
(all in one line, no linebreak) instead of just python3, where .. is the prefix you specified in the configure.
1 Like
Hello Kostas,
Thank you again for the help!
I managed to complete the installation process (which I did in my ~/Research/getfem folder), but when I try to use it to run a Python file (or even to open a Python console) it does not recognize the getfem module, and throws an error when I try to open it.
I have noticed that the ~/Research/getfem/lib/ folder does not exist, which is likely part of the problem. I have searched the ~/Research/getfem directory for a folder called “lib,” but have not been able to find that either.
Do you know what might be going on?
Sincerely,
Eric Comstock
you need to run
make install
to finish the installation, and it needs to execute without any errors.
Hello Kostas,
Okay - that explains it. My make install gave a couple of minor errors, which is probably why it is not working.
Its results are
Making install in m4
make[1]: Entering directory '/home/eac/Research/getfem/m4'
make[2]: Entering directory '/home/eac/Research/getfem/m4'
make[2]: Nothing to be done for 'install-exec-am'.
make[2]: Nothing to be done for 'install-data-am'.
make[2]: Leaving directory '/home/eac/Research/getfem/m4'
make[1]: Leaving directory '/home/eac/Research/getfem/m4'
Making install in cubature
make[1]: Entering directory '/home/eac/Research/getfem/cubature'
make[2]: Entering directory '/home/eac/Research/getfem/cubature'
make[2]: Nothing to be done for 'install-exec-am'.
make[2]: Nothing to be done for 'install-data-am'.
make[2]: Leaving directory '/home/eac/Research/getfem/cubature'
make[1]: Leaving directory '/home/eac/Research/getfem/cubature'
Making install in src
make[1]: Entering directory '/home/eac/Research/getfem/src'
make[2]: Entering directory '/home/eac/Research/getfem/src'
/usr/bin/mkdir -p '/Research/getfem/lib'
/usr/bin/mkdir: cannot create directory ‘/Research’: Permission denied
make[2]: *** [Makefile:801: install-libLTLIBRARIES] Error 1
make[2]: Leaving directory '/home/eac/Research/getfem/src'
make[1]: *** [Makefile:1073: install-am] Error 2
make[1]: Leaving directory '/home/eac/Research/getfem/src'
make: *** [Makefile:598: install-recursive] Error 1
Do you know how to solve these? Am I missing dependencies?
Sincerely,
Eric Comstock
in your configure command, you used --prefix=/Research which tell the build system to install all files to the /Research folder. You do not have permissions to create folders under the root folder / (and you should not.) Re-do the procedure with the configure option --prefix=/home/eac/Research/opt for example or any folder under /home/eac/.
1 Like
Hello Kostas,
Thank you so much! It works!
One other thing I ended up having to change was /home/eac/Research/opt/lib/python3.10/site-packages/getfem python3 instead of /home/eac/Research/opt/lib/python3.13/site-packages/getfem python3, since I had Python 3.10 installed. I has Python 3.13 as well, but I think the latter is classified as a different package in Ubuntu from python3 (which includes 3.10), which is why I think it used that.
Sincerely,
Eric Comstock
just use the default python of your Ubuntu, it is fine. Avoid having several python versions installed.
1 Like
Hello Kostas,
I tried running the 4096-vertex case, and this version of getFEM produced significantly different results (which make a lot less sense), and gave the following error:
Level 1 Warning in getfem_models.cc, line 595: Empty basis found for multiplier mult_on_f_8
I could not find mult_on_f_8 in my code, so I assume it has something to do with getFEM. Do you know what this could be? Do I need to format my governing equations differently?
Sincerely,
Eric Comstock
need some more information about your model. Add a md.variable_list() command before the solve (where md is your model object), and post the output of this command in the terminal.