CM3 Finite Element Solver for Multiphysics Problems

1. Install required libraries
2. Install OpenCascade (Optional)
3. Install Torch (Optional)
4. Install cm3Libraries
5. Make documentation
6. Fundings

1. Install required libraries

Follow either 1.1. Libraires under ubuntu or 1.2. Other platforms

1.1. Libraires under ubuntu (with sudo privilege)

1.1.1. Mandatory libraries

sudo apt-get install libblas-dev libopenmpi-dev openmpi-bin liblapack-dev g++ gfortran cmake-curses-gui swig libfltk1.3-dev libpng-dev libjpeg-dev petsc-dev slepc-dev python3-scipy libpython3-dev python3-numpy python3-pandas libmumps-dev git

sudo apt-get install python3-pip

1.1.2. Libraries for open-cascade (Optional)

sudo apt-get install libxi-dev libxmu-dev curl freetype2-demos libfreetype6 libfreetype6-dev libfontconfig1-dev  

1.2. Other platforms (or ubuntu without sudo privilege)

1.2.1. Mandatory libraries

Equivalent of

• libblas-dev
• libopenmpi-dev
• openmpi-bin
• liblapack-dev
• g++
• gfortran
• cmake-curses-gui
• swig
• libfltk1.3-dev
• libpng-dev
• libjpeg-dev
• petsc-dev
• slepc-dev
• python3-scipy
• libpython3-dev
• python3-numpy
• python3-pandas
• libmumps-dev

Below are somes instructions to install some of them in $HOME/local:

cd && mkdir local

1.2.1.1. SWIG > 2.0

1. Go to local

cd && cd local && mkdir swig

2. Download sources from http://www.swig.org/download.html in your $HOME/local
3. Untar the archive
4. Go to the extracted folder and configure

  ./configure --prefix=$HOME/local/swig

• If configure report a problem with pcre you have to follow the instruction provided
  • Download pcre from http://www.pcre.org, and place the archive in the swig's configuration folder.
  • Then type (more help type Tools/pcre-build.sh --help)

  Tools/pcre-build.sh

  ./configure --prefix=$HOME/local/swig

5. Install swig

make && make install

6. Add export PATH:$HOME/local/swig/bin:$PATH in your .bashrc

1.2.1.2. Cmake (if version is too old (2.6.4))

1. Go to local

cd && cd local && mkdir cmake

2. Download sources from http://www.cmake.org/cmake/resources/software.html in your $HOME/local
3. Untar the archive
4. Go to the extracted folder and configure

  ./configure --prefix=$HOME/local/cmake

5. Install cmake

gmake && gmake install

6. Add export PATH=$HOME/local/cmake/bin:$PATH in your .bashrc

1.2.1.3. PETSC

1. Go to local

cd && cd local && mkdir petsc

2. Download sources from https://petsc.org/release/install/ in your $HOME/local
3. Untar the archive
4. Type

export PETSC_DIR=$HOME/local/petsc; export PETSC_ARCH=linux-gnu-c-opt

5. Go to the extracted folder and configure. Different options are possible depending on the existing installed libraries on the machine. Either you install them with petsc or make link to the installed libraries.

• Install petsc at the same time as blas, mumps, metis, while using an installed version of MPI (you need to have first defined export MPI_HOME= ..... in your .bashrc or to export it in command line)

  ./configure --with-debugging=0 --download-fblaslapack=yes --download-mumps=yes --download-scalapack=yes --download-blacs=yes --with-mpi-dir=$MPI_HOME --with-pic --with-fpic --with-shared-libraries=1 --with-clanguage=cxx --known-mpi-shared-libraries=0 --download-parmetis=yes --download-metis=yes --download-superlu_dist=yes

• Install petsc at the same time as blas, mumps, metis, MPI

   ./configure --with-debugging=0 --download-fblaslapack=yes --download-mumps=yes --download-scalapack=yes --download-mpich=yes --with-shared-libraries=yes --known-mpi-shared-libraries=0 --download-parmetis=yes --download-metis=yes --download-superlu_dist=yes

• In case of problem for fblaslapack or fc error, use --with-fc=gfortran
• In case of error with metis you have to use --download-metis=no

6. Install petsc: copy the suggested line after step 5., which should look like

make PETSC_DIR=$HOME/local/petsc PETSC_ARCH=linux-gnu-c-opt all

7. Test petsc: copy the suggested line after step 5., which should look like

make PETSC_DIR=$HOME/local/petsc PETSC_ARCH=linux-gnu-c-opt test

8. Add export PETSC_DIR=$HOME/local/petsc and export PETSC_ARCH=linux-gnu-c-opt in your .bashrc
9. Create the file .petscrc in your $HOME, with the option of petsc. One example using mumps for parallel computing reads

#direct solver
-ksp_type preonly
-pc_type lu
-pc_factor_mat_solver_type mumps
-mat_mumps_icntl_13 1
#-mat_mumps_icntl_14 2000
-mat_mumps_icntl_23 50000
-mat_mumps_cntl_5
-mat_mumps_icntl_24 1
# iterative solver
#-pc_type jacobi
#-ksp_gmres_restart 200

2. Install OpenCascade (Optional)

2.1. Mandatory libraries

See 1.1.2. Libraries for Open Cascade

2.2. Install OpenCascade in local

This assumes the directory $HOME/local exists

1. Create installation directory

cd && cd local && mkdir occt

2. Download OpenCascade

curl -L -o occ83.tgz "https://github.com/Open-Cascade-SAS/OCCT/archive/refs/tags/V7_8_0.tar.gz" &&
	tar xf occ83.tgz

3. Compile OpenCascade

cd OCCT-7_8_0 && mkdir build && cd build && cmake .. -DCMAKE_BUILD_TYPE=Release -DBUILD_MODULE_Draw=0 -DBUILD_MODULE_Visualization=0 -DBUILD_MODULE_ApplicationFramework=0 -DINSTALL_DIR=$HOME/local/occt && make -j8 && make install

4. Add LD_LIBRARY_PATH=$HOME/local/occt/lib/:$LD_LIBRARY_PATH and export CASROOT=$HOME/local/occt in your .bashrc

3. Install Torch (Optional)

The cm3Libraries can be compiled with the Torch libraries to use torch models as constitutive laws.

To use torch you can follow the instructions either 2.1. Without GPU capability or 2.2. With GPU capability. The following assumes the directory $HOME/local exists.

3.1. Without GPU capability

1. Download the libraries to link the code with Torch

cd && cd local 

Here below to update with the last stable version from https://pytorch.org/

wget https://download.pytorch.org/libtorch/cpu/libtorch-shared-with-deps-2.3.0%2Bcpu.zip 

2. Unzip the archive, this will be extracted in $HOME/local/libtorch by default so make sure that directory does not exist:

 unzip libtorch-shared-with-deps-2.1.1+cpu.zip

3. Add export TORCHDIR=$HOME/local/libtorch in your .bashrc
4. Install the pyTorch if you need it to create and train models (Optional)

 sudo pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu

With ubuntu 24.04 (without sudo):

 sudo pip3 install --break-system-packages torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu

3.2. With GPU capability

We consider here the version using Cuda 12.1, to be adapted with the platforms.

Part 3.2.1.: Install Cuda (for ubuntu 22.04)

1. Clean up existing Cuda & Nvidia drivers (to check if still needed with lastest release of ubuntu)

sudo rm /etc/apt/sources.list.d/cuda* 

sudo apt remove --autoremove nvidia-cuda-toolkit && sudo apt remove --autoremove nvidia-*

2. Purge Cuda & Nvidia drivers (to check if still needed with lastest release of ubuntu)

sudo apt-get purge nvidia* 

sudo apt-get autoremove && sudo apt-get autoclean

sudo rm -rf /usr/local/cuda*

3. Prepare for Cuda && Nvidia drivers (to check if still needed with lastest release of ubuntu)

• Tested method

sudo apt update 

sudo add-apt-repository ppa:graphics-drivers

sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/3bf863cc.pub

sudo bash -c 'echo "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64 /" > /etc/apt/sources.list.d/cuda.list'

• Alternate method (not tested, more options on https://developer.nvidia.com/cuda-downloads)

cd && cd local

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin

sudo mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600

wget https://developer.download.nvidia.com/compute/cuda/12.1.0/local_installers/cuda-repo-ubuntu2204-12-1-local_12.1.0-530.30.02-1_amd64.deb

sudo dpkg -i cuda-repo-ubuntu2204-12-1-local_12.1.0-530.30.02-1_amd64.deb

sudo cp /var/cuda-repo-ubuntu2204-12-1-local/cuda-*-keyring.gpg /usr/share/keyrings/

4. Install Cuda

sudo apt-get update

sudo apt-get -y install cuda

Remark, alternate commands:

• bash sudo apt-get install -y cuda-drivers or
• sudo apt-get install -y nvidia-driver-550-open and bash apt-get install -y cuda-drivers-550

5. Add export PATH=/usr/local/cuda/bin/:$PATH and export LD_LIBRARY_PATH=/usr/local/cuda/lib64/:$LD_LIBRARY_PATH in your .bashrc

Part 3.2.2.: Install Cudnn (for ubuntu 22.04)

• Method 1:

sudo apt-get -y install nvidia-cudnn

• Method 2:
  • Download cudnn from nvidia: https://developer.nvidia.com/cudnn-downloads

cd && cd local

wget https://developer.download.nvidia.com/compute/cudnn/9.0.0/local_installers/cudnn-local-repo-ubuntu2204-9.0.0_1.0-1_amd64.deb

• Prepare installation

sudo dpkg -i cudnn-local-repo-ubuntu2204-9.0.0_1.0-1_amd64.deb

sudo cp /var/cudnn-local-repo-ubuntu2204-9.0.0/cudnn-*-keyring.gpg /usr/share/keyrings/

• Install

sudo apt-get update

sudo apt-get -y install cudnn-cuda-12

Part 3.2.3.: Install Torch

1. Download the libraries to link the code with Torch

cd && cd local 

wget https://download.pytorch.org/libtorch/cu121/libtorch-shared-with-deps-2.1.1%2Bcu121.zip 

2. Unzip the archive, this will be extracted in $HOME/local/libtorch by default so make sure that directory does not exist:

 unzip libtorch-shared-with-deps-2.1.1+cu121.zip

3. Add export TORCHDIR=$HOME/local/libtorch in your .bashrc
4. Install the pyTorch if you need it to create and train models (Optional)

 sudo pip3 install torch torchvision torchaudio 

With ubuntu 24.04 (without sudo):

 sudo pip3 install --break-system-packages torch torchvision torchaudio 

4. Install cm3Libraries

We assume the code is installed in $HOME/cm3

cd && mkdir cm3 && cd cm3

4.1. Download the sources

• GMSH:

git clone https://gitlab.onelab.info/gmsh/gmsh.git 

• cm3Libraries

git clone https://gitlab.onelab.info/cm3/cm3Libraries.git 

• Mean-Field-Homogenization (Optional)

git clone https://gitlab.onelab.info/cm3/cm3MFH.git 

4.2. Download the Mean-Field-Homogenization sources (Optional)

• Mean-Field-Homogenization (Optional)

git clone https://gitlab.onelab.info/cm3/cm3MFH.git 

• Copy the source in cm3Libraries

cd cm3Libraries/dG3D/src/ && ./copyFiles.sh && cd && cd cm3 

4.3. Desinstall conflicting libraries

• Gmsh (if installed), you will get a new version
• Anaconda (if installed)

4.4. Build the code

• Create a build directory (e.g. release for the Release version and debug for the Debug version):

cd cm3Libraries/dG3D && mkdir release && cd release 

• Start configure

ccmake .. 

• Select the options

CMAKE_BUILD_TYPE = Release (or Debug)
ENABLE_PRIVATE_API=ON (to get the full gmsh code)
ENABLE_MPI = ON (for // implementation)
ENABLE_SLEPC=ON (optional, only if installed)
ENABLE_PETSC=ON 
ENABLE_MUMPS=ON 
ENABLE_NUMPY=ON (optional, only if installed)
ENABLE_GMM=OFF 
ENABLE_WRAP_PYTHON=ON
ENABLE_TORCH=ON (optional, only if installed)

#Select one of the following two as ON (the other being OFF)
ENABLE_EIGEN=OFF
ENABLE_BLAS_LAPACK=ON

• On some platforms, you might need to restrict the configuration if it does not compile

ENABLE_CXX11 = OFF
ENABLE_FLTK = OFF
ENABLE_OCC  = OFF
ENABLE_METIs= OFF
ENABLE_NUMPY= OFF

• Check that the correct libraries are found by pressing t and correct if needed
• Generate the Makefiles by pressing c until you can press g (and press it)
• Compile (e.g. on 4 processors)

make -j4

4.4. Change your .bashrc

• Make sure that you have access to the installed libraries (see above)
• Give access to cm3Libraries: add in your .bashrc (to update with debug if needed- do not use both versions at the same time):

export PATH=$PATH:$HOME/cm3/cm3Libraries/dG3D/release/NonLinearSolver/gmsh
export PYTHONPATH=$PYTHONPATH:$HOME/cm3/cm3Libraries/dG3D/release:$HOME/cm3/cm3Libraries/dG3D/release/NonLinearSolver/gmsh/utils/wrappers:$HOME/cm3/cm3Libraries/dG3D/release/NonLinearSolver/gmsh/utils/wrappers/gmshpy

4.5. Test the code

cd && cd cm3/cm3Libraries/dG3D/benchmarks/taylorMPI_implicit

mpiexec -np 4 python3 taylorTetCGDynImplicit.py

5. To make documentation

• Install doxygen:

sudo apt install doxygen

• Generate doc

    cd && cd cm3/cm3Libraries/doc;
    doxygen

• Open cm3/cm3Libraries/doc/html/index.html

6. Fundings

Parts of the implementation have been conducted in the context of research projects:

• FRIA and FNRS supported PhD Theses of Gauthier Becker, Juan Manuel Calleja, Vinayak Gholap
• F.R.S-F.N.R.S. under the project number PDR T.1015.14
• ARC 09/14-02 BRIDGING - From imaging to geometrical modelling of complex micro structured materials: Bridging computational engineering and material science
• ARC S3CM3: Synthesis, Characterization, and MultiScale Model of Smart Composite Materials (S3CM3)
• SIMUCOMP: The research has been funded by the Walloon Region under the agreement no 1017232 (CT-EUC 2010-10-12) in the context of the ERA-NET +, Matera + framework.
• 3SMVIB: The research has been funded by the Walloon Region under the agreement no 1117477 (CT-INT 2011-11-14) in the context of the ERA-NET MNT framework.
• STOMMMAC: The research has been funded by the Walloon Region under the agreement no 1410246- STOMMMAC (CT-INT 2013-03-28) in the context of the M-ERA.NET Joint Call 2014.
• CARBOBRAKE: The research has been funded by the Walloon Region under the Grant Agreement no. 2010092 (2021 03 24) in the context of the M-ERA.NET Joint Call 2020.
• MRIPF: The research has been funded by the Walloon Region under the agreement no.7581-MRIPF in the context of the 16th MECATECH call
• VISCOS: The research has been funded by the Walloon Region under the agreement no.7911-VISCOS in the context of the 21st SKYWIN call.
• EntroTough: The research has been funded by the Walloon Region under the agreement no. 1610154- EntroTough in the context of the 2016 WalInnov call.
• VIBRA: The research has been funded by the Walloon Region under the agreement no.8422-SW ViBra in the context of the 28th SKYWIN call.
• PhD of Lina Homsi: This project has been funded with support of the European Commission under the grant number 2012-2624/001-001-EM. This publication reflects the view only of the author, and the Commission cannot be held responsible for any use which may be made of the information contained therein.
• MOAMMM: This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 862015 for the project Multi-scale Optimisation for Additive Manufacturing of fatigue resistant shock-absorbing MetaMaterials (MOAMMM) of the H2020-EU.1.2.1. - FET Open Programme
• DIDEAROT: This project has received funding from the European Union’s Horizon Europe Framework Programme under grant agreement No. 101056682 for the project “DIgital DEsign strategies to certify and mAnufacture Robust cOmposite sTructures (DIDEAROT)”. The contents of this publication are the sole responsibility of ULiege and do not necessarily reflect the opinion of the European Union. Neither the European Union nor the granting authority can be held responsible for them.