Tips and tricks / FOAM / Installation
From HaniWiki
General installation instructions can be found at http://openfoamwiki.net/index.php/Installation
Below are my own raw notes when installing.
General notes
The hwloc package is providing the command 'lstopo' for reliably detecting the number of processor cores on a given machine. If the command 'lstopo' is not found, a less reliable fallback solution will be used. See the script 'bin/foamGetSystemInfo' for the details.
First note on Gcc
Find available versions in /chalmers/sw/sup64/gcc*, and then select the one needed, as:
vcs-select -p gcc-8.1.0
It will then be used for all new terminal windows.
However, I had problems using it. One thing that may be needed is to also do:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/chalmers/sw/sup64/gcc-8.1.0/lib64
A hint from support is to do:
export LDFLAGS="-Wl,-rpath=/chalmers/sw/sup64/gcc-8.1.0/lib64 $LDFLAGS"
The flag -Wl,xxx will make gcc forward xxx as additional arguments to ld when linking. The flag -rpath makes ld add to the runpath of the binaries.
For OpenFOAM, it may be better to add the required Gcc in ThirdParty. See e.g. installation of OpenFOAM-v1906 below.
Unselect by:
vcs-select -u -p gcc-8.1.0
FOAM-extend-4.1_NR with solids4Foam and cfMesh, with Gcc at Tetralith (non-root), CentOS 7, 2021-01-04
cd /proj/hani/OF_inst/Tetralith/foam git clone -b nextRelease https://git.code.sf.net/p/foam-extend/foam-extend-4.1 foam-extend-4.1_NR cd foam-extend-4.1_NR sed -i s/"5\.0"/"4\.1_NR"/g etc/bashrc module load buildenv-gcc/2018a-eb #Avoid having to type Y at compilation: echo "export PARAVIEW_SYSTEM=1" > etc/prefs.sh echo "export WM_MPLIB=SYSTEMOPENMPI" >> etc/prefs.sh #The following inspired by etc/prefs.csh: echo "export OPENMPI_DIR=/software/sse/easybuild/prefix/software/OpenMPI/2.1.2-GCC-6.4.0-2.28" >> etc/prefs.sh echo "export OPENMPI_BIN_DIR=\$OPENMPI_DIR/bin" >> etc/prefs.sh #The following line gave two paths, so I had to give the correct one explicitly (in the next line) #echo 'export OPENMPI_LIB_DIR="`$OPENMPI_BIN_DIR/mpicc --showme:libdirs`"' >> etc/prefs.sh echo "export OPENMPI_LIB_DIR=/software/sse/easybuild/prefix/software/OpenMPI/2.1.2-GCC-6.4.0-2.28/lib" >> etc/prefs.sh echo 'export OPENMPI_INCLUDE_DIR="`$OPENMPI_BIN_DIR/mpicc --showme:incdirs`"' >> etc/prefs.sh echo 'export OPENMPI_COMPILE_FLAGS="`$OPENMPI_BIN_DIR/mpicc --showme:compile`"' >> etc/prefs.sh echo 'export OPENMPI_LINK_FLAGS="`$OPENMPI_BIN_DIR/mpicc --showme:link`"' >> etc/prefs.sh export FOAM_INST_DIR=/proj/hani/OF_inst/Tetralith/foam; . $FOAM_INST_DIR/foam-extend-4.1_NR/etc/bashrc foam #Fix some compilation problems: rm -r src/mesh/blockMesh/blockEdges #Not used, and had at least one file with the same name as another class (lnInclude problem). https://sourceforge.net/p/foam-extend/tickets/56/ #In src/conjugateHeatTransfer/fvPatchFields/chtRcThermalDiffusivityResistance/chtRcThermalDiffusivityResistanceFvPatchScalarField.C, comment lines 411 and 412. https://sourceforge.net/p/foam-extend/tickets/57/ #Remaining error in applications/utilities/postProcessing/POD/vectorSnapshots not fixed. https://sourceforge.net/p/foam-extend/tickets/58/ ./Allwmake.firstInstall &>log_Allwmake& ./Allwmake.firstInstall &>log_Allwmake& echo "alias fe41NR='export FOAM_INST_DIR=/proj/hani/OF_inst/Tetralith/foam; . \$FOAM_INST_DIR/foam-extend-4.1_NR/etc/bashrc'" >> ../../../addToBashrc echo "alias fe41NR_compile='module load buildenv-gcc/2018a-eb; export FOAM_INST_DIR=/proj/hani/OF_inst/Tetralith/foam; . \$FOAM_INST_DIR/foam-extend-4.1_NR/etc/bashrc'" >> ../../../addToBashrc #If not done before: #alias paraview_ml='module load ParaView/5.4.1-nsc1-gcc-2018a-eb'
Test script:
#!/bin/bash # #SBATCH -n 4 #SBATCH -t 00:20:00 #SBATCH -J damBreak #SBATCH -A snic2020-3-23 export FOAM_INST_DIR=/proj/hani/OF_inst/Tetralith/foam; . $FOAM_INST_DIR/foam-extend-4.1_NR/etc/bashrc cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak . blockMesh -case damBreak setFields -case damBreak decomposePar -case damBreak mpirun interFoam -parallel -case damBreak &> log.damBreak
Install solids4Foam:
cd extend-bazaar git clone -b development https://dr_hakan_nilsson@bitbucket.org/philip_cardiff/solids4foam-release.git find solids4foam-release -name files | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done find solids4foam-release -name files.foamextend | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done find solids4foam-release -name files | while read item; do sed -i -e 's/FOAM_USER_APPBIN/FOAM_SITE_APPBIN/g' $item; done find solids4foam-release -name files.foamextend | while read item; do sed -i -e 's/FOAM_USER_APPBIN/FOAM_SITE_APPBIN/g' $item; done find solids4foam-release -name options | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done sed -i s/FOAM_USER_APPBIN/FOAM_SITE_APPBIN/g solids4foam-release/applications/scripts/Allwmake cd solids4foam-release #The following commented should not be done. See solids4foam-release/filesToReplaceInOF/README.txt ##Next line to check which files to copy in the following lines: #(cd filesToReplaceInOF && ./Allcheck) #cp filesToReplaceInOF/solution.C /proj/hani/OF_inst/Tetralith/foam/foam-extend-4.1_NR/src/foam/matrices/solution/ #cp filesToReplaceInOF/WedgePointPatchField.C /proj/hani/OF_inst/Tetralith/foam/foam-extend-4.1_NR/src/foam/fields/PointPatchFieldTemplates/constraint/wedge/ #cp filesToReplaceInOF/tensor.C /proj/hani/OF_inst/Tetralith/foam/foam-extend-4.1_NR/src/foam/primitives/Tensor/tensor/ #cp filesToReplaceInOF/findRefCell.C /proj/hani/OF_inst/Tetralith/foam/foam-extend-4.1_NR/src/finiteVolume/cfdTools/general/findRefCell/ ##Next line to check that the files were copied correctly: #(cd filesToReplaceInOF && ./Allcheck) #INSTEAD, comment "# Replace files in the main foam libraries" in Allwmake ./Allwmake >& log_Allwmake
Test in new terminal window:
fe41NR mkdir solids4FoamTest cd solids4FoamTest scp -r $WM_PROJECT_DIR/extend-bazaar/solids4foam-release/tutorials/fluidSolidInteraction/beamInCrossFlow/elasticBeam . cd elasticBeam ./Allrun >& log&
Post-process in another terminal window:
paraview_ml touch case.foam vglrun paraview
Read in case.foam
Make available:
cd /proj/hani/OF_inst/Tetralith/foam chmod -R go+Xr foam-extend-4.1_NR chmod -R go+Xr site/4.1_NR
OpenFOAM-v1912 with solids4Foam and cfMesh, with Gcc at Hebbe (non-root), CentOS 7, 2020-12-23
cd /cephyr/NOBACKUP/groups/snic2020-6-28/hebbe mkdir OpenFOAM cd OpenFOAM/ wget https://sourceforge.net/projects/openfoam/files/v1912/OpenFOAM-v1912.tgz wget https://sourceforge.net/projects/openfoam/files/v1912/ThirdParty-v1912.tgz tar xzf OpenFOAM-v1912.tgz tar xzf ThirdParty-v1912.tgz rm OpenFOAM-v1912.tgz ThirdParty-v1912.tgz module load foss/2019a export MPI_ROOT=$EBROOTOPENMPI export FOAM_INST_DIR=/cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/OpenFOAM . $FOAM_INST_DIR/OpenFOAM-v1912/etc/bashrc WM_MPLIB=SYSTEMOPENMPI foam sed -i s/fPIC/"fPIC -fuse-ld=bfd"/g wmake/rules/linux64Gcc/c++ cd modules git clone -b development https://dr_hakan_nilsson@bitbucket.org/philip_cardiff/solids4foam-release.git find solids4foam-release -name files | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done find solids4foam-release -name files.openfoam | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done find solids4foam-release -name files | while read item; do sed -i -e 's/FOAM_USER_APPBIN/FOAM_SITE_APPBIN/g' $item; done find solids4foam-release -name files.openfoam | while read item; do sed -i -e 's/FOAM_USER_APPBIN/FOAM_SITE_APPBIN/g' $item; done find solids4foam-release -name options | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done sed -i s/FOAM_USER_APPBIN/FOAM_SITE_APPBIN/g solids4foam-release/applications/scripts/Allwmake cp modules/solids4foam-release/filesToReplaceInOF/dataTemplates.C /cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/OpenFOAM/OpenFOAM-v1912/src/OpenFOAM/meshes/data/ cp modules/solids4foam-release/filesToReplaceInOF/AMIInterpolation.C /cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/OpenFOAM/OpenFOAM-v1912/src/meshTools/AMIInterpolation/AMIInterpolation/ cp modules/solids4foam-release/filesToReplaceInOF/AMIInterpolation.H /cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/OpenFOAM/OpenFOAM-v1912/src/meshTools/AMIInterpolation/AMIInterpolation/ foam ./Allwmake >& log.make&
Test script:
#!/usr/bin/env bash #SBATCH -A SNIC2020-5-103 -p hebbe #SBATCH -n 4 #SBATCH -t 01:00:00 #----------------------------------------------------------- module load foss/2019a export MPI_ROOT=$EBROOTOPENMPI export FOAM_INST_DIR=/cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/OpenFOAM . $FOAM_INST_DIR/OpenFOAM-v1912/etc/bashrc WM_MPLIB=SYSTEMOPENMPI export FOAM_FILEHANDLER=collated #See https://www.openfoam.com/releases/openfoam-v1712/parallel.php #------------ Copy and run the damBreak tutorial ----------- #mkdir -p $FOAM_RUN/testDamBreak #cd $FOAM_RUN/testDamBreak cp -r $FOAM_TUTORIALS/multiphase/interFoam/RAS/damBreak/damBreak . cd damBreak sed -i s/"writeFormat\s*ascii"/"writeFormat binary"/g system/controlDict blockMesh #Generates the grid setFields #Sets the initial fields decomposePar #Decomposes for parallel simulation mpirun interFoam -parallel #Runs interFoam in parallel reconstructPar #Collects the decomposed results #rm -rf processor* #Deletes the decomposed files
Test solids4Foam (does not work "Duplicate entry skewCorrected in runtime selection table snGradScheme"):
mkdir solids4FoamTest cd solids4FoamTest cp -r /cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/OpenFOAM/OpenFOAM-v1912/modules/solids4foam-release/tutorials/fluidSolidInteraction/beamInCrossFlow/elasticBeam . cd elasticBeam ./Allrun >& log&
Post-process in another terminal window:
paraview_ml touch case.foam vglrun paraview #Read in case.foam
Create alias (in addToBashrc):
alias OFv1912Cephyr="module load foss/2019a; export MPI_ROOT=\$EBROOTOPENMPI; export FOAM_INST_DIR=/cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/OpenFOAM; . \$FOAM_INST_DIR/OpenFOAM-v1912/etc/bashrc WM_MPLIB=SYSTEMOPENMPI; export FOAM_FILEHANDLER=collated"
FOAM-extend-4.1_NR (nextRelease branch), with Gcc at Hebbe (non-root), CentOS 7, 2020-07-02, updated 2020-09-29 (with MRF bug-fix)
NOTE: Avoid having to type "Y" at compilation:
echo "export PARAVIEW_SYSTEM=1" >> etc/prefs.sh
Seems to be unstable, reporting floating point exception after writing files...
[hebbe23-5:00522] *** Process received signal *** [hebbe23-5:00522] Signal: Floating point exception (8)
cd /cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/foam git clone -b nextRelease https://git.code.sf.net/p/foam-extend/foam-extend-4.1 foam-extend-4.1_NR cd foam-extend-4.1_NR sed -i s/"5\.0"/"4\.1_NR"/g etc/bashrc
Fix problem with:
" /apps/Hebbe7/software/Compiler/GCCcore/8.2.0/binutils/2.31.1/bin/ld.gold: error: --add-needed is not supported but is required for libopen-rte.so.40 in /apps/Hebbe7/software/Compiler/GCC/8.2.0-2.31.1/OpenMPI/3.1.3/lib/libmpi.so "
Comment from Mikael Öhman, C3SE:
" Jag vet att det varit problem med gold-länkaren för vissa version av openfoam. Du får nog använda dig av gamla vanliga BFD-länkaren istället. Du kan specifiera ett länkningskommando på något sätt, jag vet inte hur WMake gör, men, det är "ld.bfd" dom du vill använda under länkningssteget, men, om du har något ställa där du kan lägga till en länkningsflagga; LDFLAGS=" -fuse-ld=bfd " "
sed -i s/fPIC/"fPIC -fuse-ld=bfd"/g wmake/rules/linux64Gcc/c++
Continue:
module load foss/2019a module load CMake/3.13.3 #Uses same GCCcore as foss/2019a: GCCcore/8.2.0 (checked with 'module spider') echo "export WM_MPLIB=SYSTEMOPENMPI" >> etc/prefs.sh echo "export OPENMPI_DIR=$EBROOTOPENMPI" >> etc/prefs.sh echo "export OPENMPI_BIN_DIR=$EBROOTOPENMPI/bin" >> etc/prefs.sh export FOAM_INST_DIR=/cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/foam; . $FOAM_INST_DIR/foam-extend-4.1_NR/etc/bashrc foam ./Allwmake.firstInstall >& log Y #Type CTRL-z, bg, and tailf log. Wait until finished #Do again, since parallel compilation may make things fail: ./Allwmake.firstInstall >& log Y #Type CTRL-z, bg, and tailf log. Wait until finished chgrp -R c3-fluiddyn /cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/foam/foam-extend-4.1_NR chmod -R go+Xr /cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/foam/foam-extend-4.1_NR
Submit script:
#!/usr/bin/env bash #SBATCH -A SNIC2020-5-103 -p hebbe #SBATCH -n 4 #SBATCH -t 01:00:00 #----------------------------------------------------------- ml foss/2019a export FOAM_INST_DIR=/cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/foam; . $FOAM_INST_DIR/foam-extend-4.1_NR/etc/bashrc #------------ Copy and run the damBreak tutorial ----------- #mkdir -p $FOAM_RUN/testDamBreak #cd $FOAM_RUN/testDamBreak cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak . cd damBreak blockMesh #Generates the grid setFields #Sets the initial fields decomposePar #Decomposes for parallel simulation mpirun interFoam -parallel #Runs interFoam in parallel reconstructPar #Collects the decomposed results #rm -rf processor* #Deletes the decomposed files
Experimental: For Paraview, in a separate terminal without OpenFOAM (only in Thinlinc):
ml GCC/8.3.0 OpenMPI/3.1.4 ParaView/5.6.2-Python-3.7.4-mpi vglrun paraview
FOAM-extend-3.1 with Gcc at Hebbe (non-root), CentOS 7, 2020-03-09
NOTE: Avoid having to type "Y" at compilation:
echo "export PARAVIEW_SYSTEM=1" >> etc/prefs.sh
cd $SNIC_NOBACKUP/hebbe/foam git clone git://git.code.sf.net/p/openfoam-extend/foam-extend-3.1 foam-extend-3.1 cd foam-extend-3.1 sed -i s/"http\:\/\/portal.nersc.gov\/svn\/visit\/tags\/2.4.2\/third_party\/libccmio-2.6.1.tar.gz"/"http\:\/\/downloads.sourceforge.net\/project\/foam-extend\/ThirdParty\/libccmio-2.6.1.tar.gz"/g ThirdParty/AllMake.stage3 sed -i s/fPIC/"fPIC -fuse-ld=bfd"/g wmake/rules/linux64Gcc/c++
Avoided compilation error(s):
fields/fvPatchFields/derived/advective/advectiveFvPatchField.H:224:24: error: cannot bind non-const lvalue reference of type ‘bool&’ to an rvalue of type ‘bool’
return inletOutlet_;
by deactivating things we don't need (that cause problems):
sed -i.bak1 s/"\$(derivedFvPatchFields)\/advective\/advectiveFvPatchFields.C"/""/g src/finiteVolume/Make/files sed -i.bak2 s/"\$(derivedFvPatchFields)\/waveTransmissive\/waveTransmissiveFvPatchFields.C"/""/g src/finiteVolume/Make/files sed -i.bak3 s/"\$(derivedFvPatchFields)\/waveTransmissiveInlet\/waveTransmissiveInletFvPatchFields.C"/""/g src/finiteVolume/Make/files sed -i.bak4 s/"derivedFvPatchFields\/engineTimeVaryingUniformFixedValue\/engineTimeVaryingUniformFixedValueFvPatchFields.C"/""/g src/engine/Make/files
Continued:
echo "export WM_MPLIB=SYSTEMOPENMPI" >> etc/prefs.sh echo "export OPENMPI_DIR=$EBROOTOPENMPI" >> etc/prefs.sh echo "export OPENMPI_BIN_DIR=$EBROOTOPENMPI/bin" >> etc/prefs.sh module load foss/2019a module load CMake/3.13.3 #Uses same GCCcore as foss/2019a: GCCcore/8.2.0 (checked with 'module spider') export FOAM_INST_DIR=$SNIC_NOBACKUP/hebbe/foam; . $FOAM_INST_DIR/foam-extend-3.1/etc/bashrc foam ./Allwmake.firstInstall >& log Y #Type CTRL-z, bg, and tailf log. Wait until finished #Do again, since parallel compilation may make things fail: ./Allwmake.firstInstall >& log Y #Type CTRL-z, bg, and tailf log. Wait until finished
FSI:
cd extend-bazaar/ wget http://openfoamwiki.net/images/5/52/Fsi_31.tar.gz tar zxvf Fsi_31.tar.gz rm Fsi_31.tar.gz cd FluidStructureInteraction/ find src -name options | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done find src -name files | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done find run -name options | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done #Change to -I$(WM_PROJECT_DIR)/extend-bazaar/FluidStructureInteraction/src/fluidStructureInteraction/lnInclude #in: #run/fsiFoam/HronTurekFsi3/hronTurekReport/Make/options #run/fsiFoam/beamInCrossFlow/setInletVelocity/Make/options #run/stressFoam/plateHole/setPlateHoleBC/Make/options # pushd src/fluidStructureInteraction/stressModels/constitutiveModel wget https://raw.githubusercontent.com/wyldckat/FluidStructureInteraction/fe31/src/fluidStructureInteraction/stressModels/constitutiveModel/constitutiveModel.C -O constitutiveModel.C wget https://raw.githubusercontent.com/wyldckat/FluidStructureInteraction/fe31/src/fluidStructureInteraction/stressModels/constitutiveModel/constitutiveModel.H -O constitutiveModel.H popd cd src/ ./Allwmake >& log& ./Allwmake >& log&
Set rights:
chgrp -R c3-fluiddyn $SNIC_NOBACKUP/hebbe/foam/foam-extend-3.1 chmod -R go+Xr $SNIC_NOBACKUP/hebbe/foam/foam-extend-3.1
Submit script:
#!/usr/bin/env bash #SBATCH -A SNIC2020-5-103 -p hebbe #SBATCH -n 4 #SBATCH -t 01:00:00 #----------------------------------------------------------- ml foss/2019a export FOAM_INST_DIR=/c3se/NOBACKUP/users/hani/hebbe/foam; . $FOAM_INST_DIR/foam-extend-3.1/etc/bashrc #------------ Copy and run the damBreak tutorial ----------- #mkdir -p $FOAM_RUN/testDamBreak #cd $FOAM_RUN/testDamBreak cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak . cd damBreak blockMesh #Generates the grid setFields #Sets the initial fields decomposePar #Decomposes for parallel simulation mpirun interFoam -parallel #Runs interFoam in parallel reconstructPar #Collects the decomposed results #rm -rf processor* #Deletes the decomposed files
Experimental: For Paraview, in a separate terminal without OpenFOAM (only in Thinlinc):
ml GCC/8.3.0 OpenMPI/3.1.4 ParaView/5.6.2-Python-3.7.4-mpi vglrun paraview
FOAM-extend-4.1 and solids4foam with Gcc at Hebbe (non-root), CentOS 7, 2020-02-05
NOTE: Avoid having to type "Y" at compilation:
echo "export PARAVIEW_SYSTEM=1" >> etc/prefs.sh
cd $SNIC_NOBACKUP/hebbe/foam git clone https://git.code.sf.net/p/foam-extend/foam-extend-4.1 foam-extend-4.1 cd foam-extend-4.1
Fix problem with:
" /apps/Hebbe7/software/Compiler/GCCcore/8.2.0/binutils/2.31.1/bin/ld.gold: error: --add-needed is not supported but is required for libopen-rte.so.40 in /apps/Hebbe7/software/Compiler/GCC/8.2.0-2.31.1/OpenMPI/3.1.3/lib/libmpi.so "
Comment from Mikael Öhman, C3SE:
" Jag vet att det varit problem med gold-länkaren för vissa version av openfoam. Du får nog använda dig av gamla vanliga BFD-länkaren istället. Du kan specifiera ett länkningskommando på något sätt, jag vet inte hur WMake gör, men, det är "ld.bfd" dom du vill använda under länkningssteget, men, om du har något ställa där du kan lägga till en länkningsflagga; LDFLAGS=" -fuse-ld=bfd " "
sed -i s/fPIC/"fPIC -fuse-ld=bfd"/g wmake/rules/linux64Gcc/c++
Continue:
module load foss/2019a module load CMake/3.13.3 #Uses same GCCcore as foss/2019a: GCCcore/8.2.0 (checked with 'module spider') echo "export WM_MPLIB=SYSTEMOPENMPI" >> etc/prefs.sh echo "export OPENMPI_DIR=$EBROOTOPENMPI" >> etc/prefs.sh echo "export OPENMPI_BIN_DIR=$EBROOTOPENMPI/bin" >> etc/prefs.sh export FOAM_INST_DIR=$SNIC_NOBACKUP/hebbe/foam; . $FOAM_INST_DIR/foam-extend-4.1/etc/bashrc foam ./Allwmake.firstInstall >& log Y #Type CTRL-z, bg, and tailf log. Wait until finished #Do again, since parallel compilation may make things fail: ./Allwmake.firstInstall >& log Y #Type CTRL-z, bg, and tailf log. Wait until finished chgrp -R c3-fluiddyn $SNIC_NOBACKUP/hebbe/foam chmod -R go+Xr $SNIC_NOBACKUP/hebbe/foam
Submit script:
#!/usr/bin/env bash #SBATCH -A SNIC2019-3-79 -p hebbe #SBATCH -n 4 #SBATCH -t 01:00:00 #----------------------------------------------------------- ml foss/2019a export FOAM_INST_DIR=/c3se/NOBACKUP/users/hani/hebbe/foam; . $FOAM_INST_DIR/foam-extend-4.1/etc/bashrc #------------ Copy and run the damBreak tutorial ----------- #mkdir -p $FOAM_RUN/testDamBreak #cd $FOAM_RUN/testDamBreak cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak . cd damBreak blockMesh #Generates the grid setFields #Sets the initial fields decomposePar #Decomposes for parallel simulation mpirun interFoam -parallel #Runs interFoam in parallel reconstructPar #Collects the decomposed results #rm -rf processor* #Deletes the decomposed files
Experimental: For Paraview, in a separate terminal without OpenFOAM (only in Thinlinc):
ml GCC/8.3.0 OpenMPI/3.1.4 ParaView/5.6.2-Python-3.7.4-mpi vglrun paraview
solids4Foam, installed with fe41
cd /cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/foam/foam-extend-4.1/extend-bazaar
Login at bitbucket.org (login details in usual place), and go to Repositories and solids4foam-release, AND/OR:
git clone https://dr_hakan_nilsson@bitbucket.org/philip_cardiff/solids4foam-release.git
Do some modifications and compile:
find . -name files | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done find . -name files.foamextend | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done find . -name files | while read item; do sed -i -e 's/FOAM_USER_APPBIN/FOAM_SITE_APPBIN/g' $item; done find . -name files.foamextend | while read item; do sed -i -e 's/FOAM_USER_APPBIN/FOAM_SITE_APPBIN/g' $item; done find . -name options | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done fe41 ./Allwmake &>log_Allwmake ./Allwmake &>log_Allwmake ./Allwmake &>log_Allwmake cd .. chmod -R go+Xr solids4foam-release
Test in new terminal window:
fe41 mkdir solids4FoamTest cd solids4FoamTest scp -r /cephyr/NOBACKUP/groups/snic2020-6-28/hebbe/foam/foam-extend-4.1/extend-bazaar/solids4foam-release/tutorials/fluidSolidInteraction/beamInCrossFlow/elasticBeam . cd elasticBeam ./Allrun >& log&
Post-process in another terminal window:
paraview_ml touch case.foam vglrun paraview
Read in case.foam
FOAM-extend-4.1 with Icc at Hebbe (non-root), CentOS 7, 2020-02-05
NOTE: Avoid having to type "Y" at compilation:
echo "export PARAVIEW_SYSTEM=1" >> etc/prefs.sh
DID NOT WORK, BUT IS KEPT AS TEMPLATE!
ERROR MESSAGES:
/bin/sh: cmake: command not found #See solution for Gcc (but that solution may mess up compilers)
lnInclude/meshObjectBase.H(70): error: no suitable user-defined conversion from "Foam::HashTable<Foam::meshObjectBase *, Foam::word,
Foam::string::hash>" to "Foam::HashTable<const Foam::meshObjectBase *, Foam::word, Foam::string::hash>" exists
mesh.objectRegistry::template lookupClass<meshObjectBase>();
^
compilation aborted for meshes/polyMesh/polyMeshUpdate.C (code 2)
metisDecomp.H(38): catastrophic error: cannot open source file "decompositionMethod.H" #include "decompositionMethod.H"
Procedure:
cd $SNIC_NOBACKUP/hebbe/foam git clone https://git.code.sf.net/p/foam-extend/foam-extend-4.1 foam-extend-4.1 cd foam-extend-4.1 echo "export WM_COMPILER=Icc" >> etc/prefs.sh echo "export WM_MPLIB=INTELMPI" >> etc/prefs.sh echo "export MPI_ROOT=\$I_MPI_ROOT" >> etc/prefs.sh echo "export OPENMPI_DIR=\$I_MPI_ROOT/intel64" >> etc/prefs.sh module load intel/2019a export FOAM_INST_DIR=$SNIC_NOBACKUP/hebbe/foam; . $FOAM_INST_DIR/foam-extend-4.1/etc/bashrc foam ./Allwmake.firstInstall >& log Y #Type CTRL-z, bg, and tailf log. Wait until finished #Do again, since parallel compilation may make things fail: ./Allwmake.firstInstall >& log Y #Type CTRL-z, bg, and tailf log. Wait until finished chgrp -R c3-fluiddyn $SNIC_NOBACKUP/hebbe chmod -R go+Xr $SNIC_NOBACKUP/hebbe
OpenFOAM-v1912 with Icc at Tetralith (non-root), CentOS 7, 2020-01-03
Read https://www.nsc.liu.se/software/buildenv/. There it says that one should not load the build module when using the compiled code.
cd /proj/hani/OF_inst/Tetralith/OpenFOAM wget https://sourceforge.net/projects/openfoam/files/v1912/OpenFOAM-v1912.tgz wget https://sourceforge.net/projects/openfoam/files/v1912/ThirdParty-v1912.tgz tar xzf OpenFOAM-v1912.tgz tar xzf ThirdParty-v1912.tgz rm OpenFOAM-v1912.tgz ThirdParty-v1912.tgz module load buildenv-intel/2018a-eb; export MPI_ROOT=$I_MPI_ROOT; export FOAM_INST_DIR=/proj/hani/OF_inst/Tetralith/OpenFOAM; . $FOAM_INST_DIR/OpenFOAM-v1912/etc/bashrc WM_COMPILER=Icc WM_MPLIB=INTELMPI cp /proj/hani/OF_inst/addToBashrc /proj/hani/OF_inst/addToBashrc_old echo "alias OFv1912='export MPI_ROOT=/software/sse/easybuild/prefix/software/impi/2018.1.163-iccifort-2018.1.163-GCC-6.4.0-2.28; module load ParaView/5.4.1-nsc1-gcc-2018a-eb; export FOAM_INST_DIR=/proj/hani/OF_inst/Tetralith/OpenFOAM; . $FOAM_INST_DIR/OpenFOAM-v1912/etc/bashrc WM_COMPILER=Icc WM_MPLIB=INTELMPI'" >> /proj/hani/OF_inst/addToBashrc foam ./Allwmake >& log.make&
Look for 'rror' and 'skip' in log file:
Error building: boost --------------------------------------------------- Optional component (CGAL) had build issues OpenFOAM will nonetheless remain largely functional ---------------------------------------------------
CMake Error at CMakeLists.txt:6 (cmake_minimum_required): CMake 3.6 or higher is required. You are running version 2.8.12.2 -- Configuring incomplete, errors occurred! Error building: ADIOS2-2.4.0 --------------------------------------------------- Optional component (ADIOS2) had build issues OpenFOAM will nonetheless remain largely functional ---------------------------------------------------
==> skip metis (no header) ==> skip SloanRenumber (no boost) ==> skip zoltanRenumber (no library) ==> skip ccmio (no header) ==> skip optional libccm adapter WARNING: skip runTimePostProcessing (no VTK or ParaView) ==> skip optional ccm conversion components (no libccm) ==> skip paraview-plugin (could not determine target or major.minor version) WARNING: skip ParaView Catalyst (missing or incorrrect ParaView) skip wmake target for modules doxygen directory
Note that metis is not available, but scotch!
Make it available:
cd .. chmod -R go+Xr OpenFOAM-v1912 chmod -R go+Xr ThirdParty-v1912
Test by submitting the damBreak case. First create a submit script named submitDambreak_OFv1912_Tetralith (note that mpprun did not work):
#!/bin/bash # #SBATCH -J damBreak #Name of job #SBATCH -t 00:30:00 #HH:MM:SS #SBATCH -n 4 #SBATCH --exclusive # #Same as alias OFv1912 (without ParaView): export MPI_ROOT=/software/sse/easybuild/prefix/software/impi/2018.1.163-iccifort-2018.1.163-GCC-6.4.0-2.28; export FOAM_INST_DIR=/proj/hani/OF_inst/Tetralith/OpenFOAM; . $FOAM_INST_DIR/OpenFOAM-v1912/etc/bashrc WM_COMPILER=Icc WM_MPLIB=INTELMPI #Prepare cp -r $FOAM_TUTORIALS/multiphase/interFoam/RAS/damBreak/damBreak . blockMesh -case damBreak setFields -case damBreak decomposePar -case damBreak #--nranks is used when less than the number of available cores is used mpirun interFoam -parallel -case damBreak >& log #mpprun did not work!!! # Script ends here
Submit by (add -A <projnumber> just after sbatch if you have access to more than one project):
sbatch submitDambreak_OFv1912_Tetralith
Check if it is running (change x_hakni to your user name):
squeue -u x_hakni
OpenFOAM-v1912 in StuDAT (non-root), Redhat Enterprise 7 , 2019-12-25
cd /chalmers/sw/unsup64/OpenFOAM wget https://sourceforge.net/projects/openfoam/files/v1912/OpenFOAM-v1912.tgz wget https://sourceforge.net/projects/openfoam/files/v1912/ThirdParty-v1912.tgz tar xzf OpenFOAM-v1912.tgz tar xzf ThirdParty-v1912.tgz rm OpenFOAM-v1912.tgz ThirdParty-v1912.tgz cp /chalmers/sw/unsup64/OpenFOAM/addToBashrc /chalmers/sw/unsup64/OpenFOAM/addToBashrc_backup echo "alias OFv1912='source /chalmers/sw/unsup64/OpenFOAM/OpenFOAM-v1912/etc/bashrc;alias paraview=\"echo Using system paraview;paraview-mesa paraview\";alias paraFoam=\"echo Using system paraview \\(File/Open case.foam\\);touch case.foam;paraview-mesa paraview\"'" >> /chalmers/sw/unsup64/OpenFOAM/addToBashrc . /chalmers/sw/unsup64/OpenFOAM/addToBashrc OFv1912 foam ./Allwmake >& log.make&
Make the installation available
foam cd .. chmod -R go+Xr OpenFOAM-v1912 chmod -R go+Xr ThirdParty-v1912
FOAM-extend-3.1 in StuDAT (master branch, with Gcc), Redhat Enterprise 7 , 2019-12-21
cd /chalmers/sw/unsup64/ mkdir foam; cd foam git clone git://git.code.sf.net/p/openfoam-extend/foam-extend-3.1 foam-extend-3.1 cd foam-extend-3.1 sed -i s/"http\:\/\/portal.nersc.gov\/svn\/visit\/tags\/2.4.2\/third_party\/libccmio-2.6.1.tar.gz"/"http\:\/\/downloads.sourceforge.net\/project\/foam-extend\/ThirdParty\/libccmio-2.6.1.tar.gz"/g ThirdParty/AllMake.stage3 cp /chalmers/sw/unsup64/OpenFOAM/addToBashrc /chalmers/sw/unsup64/OpenFOAM/addToBashrc_backup echo "alias fe31='export FOAM_INST_DIR=/chalmers/sw/unsup64/foam; . \$FOAM_INST_DIR/foam-extend-3.1/etc/bashrc;alias paraview=\"echo Using system paraview;paraview-mesa paraview\";alias paraFoam=\"echo Using system paraview \\(File/Open case.foam\\);touch case.foam;paraview-mesa paraview\"'" >> /chalmers/sw/unsup64/OpenFOAM/addToBashrc . /chalmers/sw/unsup64/OpenFOAM/addToBashrc fe31 ./Allwmake.firstInstall >& log& #Note that you need to bring forward and type "Y" ./Allwmake.firstInstall >& log& #Note that you need to bring forward and type "Y" chmod -R go+Xr ../foam-extend-3.1
FSI:
cd extend-bazaar/ wget http://openfoamwiki.net/images/5/52/Fsi_31.tar.gz tar zxvf Fsi_31.tar.gz rm Fsi_31.tar.gz cd FluidStructureInteraction/ find src -name options | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done find src -name files | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done find run -name options | while read item; do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item; done #Change to -I$(WM_PROJECT_DIR)/extend-bazaar/FluidStructureInteraction/src/fluidStructureInteraction/lnInclude #in: #run/fsiFoam/HronTurekFsi3/hronTurekReport/Make/options #run/fsiFoam/beamInCrossFlow/setInletVelocity/Make/options #run/stressFoam/plateHole/setPlateHoleBC/Make/options # pushd src/fluidStructureInteraction/stressModels/constitutiveModel wget https://raw.githubusercontent.com/wyldckat/FluidStructureInteraction/fe31/src/fluidStructureInteraction/stressModels/constitutiveModel/constitutiveModel.C -O constitutiveModel.C wget https://raw.githubusercontent.com/wyldckat/FluidStructureInteraction/fe31/src/fluidStructureInteraction/stressModels/constitutiveModel/constitutiveModel.H -O constitutiveModel.H popd cd src/ ./Allwmake >& log& ./Allwmake >& log& chmod -R go+Xr ../../FluidStructureInteraction/
FOAM-extend-3.2 in StuDAT (master branch, with Gcc), Redhat Enterprise 7 , 2019-12-21
cd /chalmers/sw/unsup64/ mkdir foam; cd foam git clone https://git.code.sf.net/p/foam-extend/foam-extend-3.2 foam-extend-3.2 cd foam-extend-3.2 sed -i s/"http\:\/\/portal.nersc.gov\/project\/visit\/third_party\/libccmio-2.6.1.tar.gz"/"http\:\/\/downloads.sourceforge.net\/project\/foam-extend\/ThirdParty\/libccmio-2.6.1.tar.gz"/g ThirdParty/AllMake.stage3 cp /chalmers/sw/unsup64/OpenFOAM/addToBashrc /chalmers/sw/unsup64/OpenFOAM/addToBashrc_backup echo "alias fe32='export FOAM_INST_DIR=/chalmers/sw/unsup64/foam; . \$FOAM_INST_DIR/foam-extend-3.2/etc/bashrc;alias paraview=\"echo Using system paraview;paraview-mesa paraview\";alias paraFoam=\"echo Using system paraview \\(File/Open case.foam\\);touch case.foam;paraview-mesa paraview\"'" >> /chalmers/sw/unsup64/OpenFOAM/addToBashrc . /chalmers/sw/unsup64/OpenFOAM/addToBashrc fe32 ./Allwmake.firstInstall >& log& #Note that you need to bring forward and type "Y" ./Allwmake.firstInstall >& log& #Note that you need to bring forward and type "Y" chmod -R go+Xr ../foam-extend-3.2
FOAM-extend-4.1 in StuDAT (master branch, with Gcc), Redhat Enterprise 7 , 2019-12-12
cd /chalmers/sw/unsup64/ mkdir foam; cd foam git clone https://git.code.sf.net/p/foam-extend/foam-extend-4.1 foam-extend-4.1 cd foam-extend-4.1/ cp /chalmers/sw/unsup64/OpenFOAM/addToBashrc /chalmers/sw/unsup64/OpenFOAM/addToBashrc_backup echo "alias fe41='export FOAM_INST_DIR=/chalmers/sw/unsup64/foam; . \$FOAM_INST_DIR/foam-extend-4.1/etc/bashrc;alias paraview=\"echo Using system paraview;paraview-mesa paraview\";alias paraFoam=\"echo Using system paraview \\(File/Open case.foam\\);touch case.foam;paraview-mesa paraview\"'" >> /chalmers/sw/unsup64/OpenFOAM/addToBashrc . /chalmers/sw/unsup64/OpenFOAM/addToBashrc fe41 ./Allwmake.firstInstall >& log& #Note that you need to bring forward and type "Y" ./Allwmake.firstInstall >& log& #Note that you need to bring forward and type "Y" chmod -R go+Xr ../foam-extend-4.1
Regarding FSI package, which I did not manage to compile at this point: See details for FOAM-extend-4.0 below. In addition: Remove -Werror from compiler flags and fix some problems, and compile again:
sed -i s/"-Werror "/""/g FluidSolidInteraction/src/fluidSolidInteraction/Make/options sed -i s/"procPatch.owner()"/"procPatch.master()"/g FluidSolidInteraction/src/fluidSolidInteraction/solidSolvers/solidModels/constitutiveModel/plasticityStressReturnMethods/kirchhoffMises/kirchhoffMises.C sed -i s/"procPatch.neighbour()"/"\!procPatch.master()"/g FluidSolidInteraction/src/fluidSolidInteraction/solidSolvers/solidModels/materialInterfaces/ULLSMaterialInterface/ULLSMaterialInterface.C
FOAM-extend-4.0 in StuDAT (master branch, with Gcc), Redhat Enterprise 7 , 2019-12-13
cd /chalmers/sw/unsup64/ mkdir foam; cd foam git clone https://git.code.sf.net/p/foam-extend/foam-extend-4.0 foam-extend-4.0 cd foam-extend-4.0/ #Fix libccmio link: sed -i s/"http:\/\/portal.nersc.gov\/project\/visit\/third_party\/"/"http:\/\/downloads.sourceforge.net\/project\/foam-extend\/ThirdParty\/"/g ThirdParty/AllMake.stage3 #Fix problem with setFluxRequired for FSI, see https://www.cfd-online.com/Forums/openfoam-cc-toolkits-fluid-structure-interaction/185685-fsifoam-4-0-can-not-run-tutorials-2.html sed -i '1044,1051s/.*/\/\/&/' $WM_PROJECT_DIR/src/finiteVolume/fvMatrices/fvMatrix/fvMatrix.C cp /chalmers/sw/unsup64/OpenFOAM/addToBashrc /chalmers/sw/unsup64/OpenFOAM/addToBashrc_backup echo "alias fe40='export FOAM_INST_DIR=/chalmers/sw/unsup64/foam; . \$FOAM_INST_DIR/foam-extend-4.0/etc/bashrc;alias paraview=\"echo Using system paraview;paraview-mesa paraview\";alias paraFoam=\"echo Using system paraview \\(File/Open case.foam\\);touch case.foam;paraview-mesa paraview\"'" >> /chalmers/sw/unsup64/OpenFOAM/addToBashrc . /chalmers/sw/unsup64/OpenFOAM/addToBashrc fe40 ./Allwmake.firstInstall >& log& #Note that you need to bring forward and type "Y" ./Allwmake.firstInstall >& log& #Note that you need to bring forward and type "Y" chmod -R go+Xr ../foam-extend-4.0
FSI package:
cd extend-bazaar export EXTEND_BAZAAR_TO_SITE=1 #To make available to everyone ./Allwmake >& log& #Fix some problems due to installation paths, and compile again: cd FluidSolidInteraction find src -name options | while read item do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item done find run -name options | while read item do sed -i -e 's/FOAM_USER_LIBBIN/FOAM_SITE_LIBBIN/g' $item sed -i -e 's=$(WM_PROJECT_DIR)/applications/solvers/FSI=$(WM_PROJECT_DIR)/extend-bazaar/FluidSolidInteraction/src=' $item sed -i -e 's=$(WM_THIRD_PARTY_DIR)/packages/eigen3=$(WM_PROJECT_DIR)/extend-bazaar/FluidSolidInteraction/src/ThirdParty/eigen3=' $item done find run -name files | while read item do sed -i -e 's/FOAM_SITE_LIBBIN/FOAM_USER_LIBBIN/g' $item done #Fix bug due to version update sed -i s/'"Time.H"'/'"foamTime.H"'/g MakeAxialMesh/makeAxialMesh/makeAxialMesh.C cd FluidSolidInteraction ./Allwmake >& log& #Strangely, fix linking to MPI, by: fe40 ./Allwmake >& log& chmod -R go+Xr ../../foam-extend-4.0
Test FSI package:
cp -r $WM_PROJECT_DIR/extend-bazaar/FluidSolidInteraction/run $FOAM_RUN/FSI_run cd $FOAM_RUN/FSI_run/fsiFoam/HronTurekFsi3/fluid #Change endTime to 4!!! At 2.4s there are only small movements. ./Allrun >& log&
OpenFOAM-v1906 with PreCICE and Calculix in StuDAT (non-root), Redhat Enterprise 7 , 2019-12-06
We here compile OpenFOAM-v1906 with ThirdParty gcc-8.2.0 and openmpi-1.10.7, since they are needed for PreCICE and Calculix. It was fine to compile only OpenFOAM with system gcc-4.8.5 and openmpi. Note that it seems to be important that both OpenFOAM, ThirdParty, and all packages for PreCICE are compiled using the same Gcc. It particularly seems like it was important that yaml was compiled with the same compiler as OFv1906 (8.2.0). The LD_LIBRARY_PATHs related to preCICE, yaml, boost_1_65_1, Petsc and Calculix in alias OFv1906 were added after entire procedure, since they are needed to use those libraries. It can be seen in the instructions when they were needed for the compilations.
It is mentioned in the instructions for PreCICE that the Boost version should be 1.65.1 or higher. For OpenFOAM-v1906, the ThirdParty Boost version is 1.64.0. Below I have used 1.64.0 for OpenFOAM and 1.65.1 for PreCICE. It would be nicer to use only one of them.
cd /chalmers/sw/unsup64/OpenFOAM wget https://sourceforge.net/projects/openfoam/files/v1906/OpenFOAM-v1906.tgz wget https://sourceforge.net/projects/openfoam/files/v1906/ThirdParty-v1906.tgz tar xzf OpenFOAM-v1906.tgz tar xzf ThirdParty-v1906.tgz rm OpenFOAM-v1906.tgz ThirdParty-v1906.tgz cp /chalmers/sw/unsup64/OpenFOAM/addToBashrc /chalmers/sw/unsup64/OpenFOAM/addToBashrc_backup echo "alias OFv1906='source /chalmers/sw/unsup64/OpenFOAM/OpenFOAM-v1906/etc/bashrc WM_COMPILER_TYPE=ThirdParty WM_COMPILER=Gcc82 WM_MPLIB=OPENMPI; export LD_LIBRARY_PATH=\$LD_LIBRARY_PATH:/chalmers/sw/unsup64/OpenFOAM/ThirdParty-v1906/platforms/linux64/mpfr-4.0.1/lib64; export LD_LIBRARY_PATH=\$LD_LIBRARY_PATH:/chalmers/sw/unsup64/OpenFOAM/ThirdParty-v1906/preCICE/precice-1.6.1/lib; export LD_LIBRARY_PATH=\$LD_LIBRARY_PATH:/chalmers/sw/unsup64/CalculiX/yaml-cpp-yaml-cpp-0.6.2/build; export LD_LIBRARY_PATH=/chalmers/sw/unsup64/OpenFOAM/ThirdParty-v1906/preCICE/boost_1_65_1/lib:\$LD_LIBRARY_PATH; export LD_LIBRARY_PATH=/chalmers/sw/unsup64/OpenFOAM/ThirdParty-v1906/preCICE/petsc/arch-linux2-c-opt/lib:\$LD_LIBRARY_PATH; export PATH=\$PATH:/chalmers/sw/unsup64/CalculiX/CalculiX/calculix-adapter-master/bin; alias paraview=\"echo Using system paraview;paraview-mesa paraview\";alias paraFoam=\"echo Using system paraview \\(File/Open case.foam\\);touch case.foam;paraview-mesa paraview\"'" >> /chalmers/sw/unsup64/OpenFOAM/addToBashrc . /chalmers/sw/unsup64/OpenFOAM/addToBashrc OFv1906 foam
Install Gcc-8.2.0, INCLUDING FORTRAN - FOR CALCULIX THAT NEEDS gfortran and it was important to be from the same gcc, (this takes hours! See dependencies at e.g. http://www.linuxfromscratch.org/lfs/view/8.3-systemd-rc1/chapter05/gcc-pass1.html):
cd ../ThirdParty-v1906 #Include fortran (gfortran, for Calculix): sed -i s/"--enable-languages=c,c++"/"--enable-languages=c,c++,fortran"/g makeGcc wget https://ftp.gnu.org/gnu/gcc/gcc-8.2.0/gcc-8.2.0.tar.gz tar xzf gcc-8.2.0.tar.gz rm gcc-8.2.0.tar.gz wget https://ftp.gnu.org/gnu/mpfr/mpfr-4.0.1.tar.gz tar xzf mpfr-4.0.1.tar.gz rm mpfr-4.0.1.tar.gz wget https://ftp.gnu.org/gnu/gmp/gmp-6.1.2.tar.xz tar xf gmp-6.1.2.tar.xz rm gmp-6.1.2.tar.xz wget https://ftp.gnu.org/gnu/mpc/mpc-1.1.0.tar.gz tar xzf mpc-1.1.0.tar.gz rm mpc-1.1.0.tar.gz export WM_ARCH_OPTION=64 ./makeGcc gcc-8.2.0 gmp-6.1.2 mpfr-4.0.1 mpc-1.1.0 >& log_makeGcc& #Check: which gfortran #Should give /chalmers/sw/unsup64/OpenFOAM/ThirdParty-v1906/platforms/linux64/gcc-8.2.0/bin/gfortran
Install OpenMPI, INCLUDING FORTRAN - FOR PETSC THAT NEEDS mpif90, and it was important to be from the same gcc. Petsc was needed for the OpenFOAM FSI adaptor by PreCICE (RBF mapping).
sed -i s/"--enable-mpi-fortran=none"/"--enable-mpi-fortran=yes"/g makeOPENMPI ./makeOPENMPI openmpi-1.10.7 >& log_makeOPENMPI& #Check: which mpif90 #Should give /chalmers/sw/unsup64/OpenFOAM/ThirdParty-v1906/platforms/linux64Gcc82/openmpi-1.10.7/bin/mpif90
Compile OpenFOAM
foam ./Allwmake >& log.make&
Make the installation available
foam cd .. chmod -R go+Xr OpenFOAM-v1906 chmod -R go+Xr ThirdParty-v1906
Tried also to install ParaView (but problems with OpenGL, so commented - don't do this):
#cd /chalmers/sw/unsup64/OpenFOAM/ThirdParty-v1906 #./makeParaView >& log.makeParaView& #cd $WM_PROJECT_DIR/applications/utilities/postProcessing/graphics/PVReaders #./Allwclean #./Allwmake #chmod -R go+Xr ../../graphics
Now complete the instructions at http://www.tfd.chalmers.se/~hani/wikis/haniWiki/index.php?title=Tips_and_tricks_/_preCICE_/_Installation#preCICE-1.6.1_for_OFv1906_in_StuDAT_.28non-root.29.2C_Redhat_Enterprise_7_.2C_2019-12-26
FOAM-extend-4.1 (master branch, with Gcc) at Tetralith (non-root), CentOS 7, 2019
Note that the following lines were typed after the installation, so be careful copy-pasting! The instructions are guidelines.
cd /proj/hani/OF_inst/Tetralith/foam git clone git://git.code.sf.net/p/foam-extend/foam-extend-4.1 foam-extend-4.1_hani cd foam-extend-4.1_hani cp etc/bashrc etc/bashrc_hani sed -i s/"export WM_PROJECT_VERSION=4.1"/"export WM_PROJECT_VERSION=4.1_hani"/g etc/bashrc_hani module load buildenv-gcc/2018a-eb export FOAM_INST_DIR=/proj/hani/OF_inst/Tetralith/foam; . $FOAM_INST_DIR/foam-extend-4.1_hani/etc/bashrc_hani WM_MPLIB=SYSTEMOPENMPI OPENMPI_DIR=/software/sse/easybuild/prefix/software/OpenMPI/2.1.2-GCC-6.4.0-2.28 OPENMPI_BIN_DIR=/software/sse/easybuild/prefix/software/OpenMPI/2.1.2-GCC-6.4.0-2.28/bin
foam ./Allwmake.firstInstall >& log& #Note that you need to bring forward and type "Y" ./Allwmake.firstInstall >& log& #Note that you need to bring forward and type "Y" chgrp -R pg_hani ../foam-extend-4.1_hani chmod -R go+Xr ../foam-extend-4.1_hani
Submit-script to test:
#!/bin/bash # #SBATCH -n 4 #SBATCH -t 00:20:00 #SBATCH -J damBreak #SBATCH -A snic2018-3-678 #OpenFOAM versions: #module load buildenv-gcc/2018a-eb export FOAM_INST_DIR=/proj/hani/OF_inst/Tetralith/foam; . $FOAM_INST_DIR/foam-extend-4.1_hani/etc/bashrc_hani WM_MPLIB=SYSTEMOPENMPI OPENMPI_DIR=/software/sse/easybuild/prefix/software/OpenMPI/2.1.2-GCC-6.4.0-2.28 OPENMPI_BIN_DIR=/software/sse/easybuild/prefix/software/OpenMPI/2.1.2-GCC-6.4.0-2.28/bin cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak . blockMesh -case damBreak setFields -case damBreak decomposePar -case damBreak mpirun interFoam -parallel -case damBreak &> result.out
FOAM-extend-4.0 at Tetralith (non-root), CentOS 7, 2018
cd /proj/hani/OF_inst/Tetralith/foam git clone -b nextRelease git://git.code.sf.net/p/foam-extend/foam-extend-4.0 foam-extend-4.0 cd foam-extend-4.0 git tag -l git checkout tags/v4.0 echo "export WM_COMPILER=Icc" >> etc/prefs.sh echo "export WM_MPLIB=INTELMPI" >> etc/prefs.sh echo "export MPI_ROOT=\$I_MPI_ROOT" >> etc/prefs.sh echo "export OPENMPI_DIR=\$I_MPI_ROOT/intel64" >> etc/prefs.sh module load buildenv-intel/2018a-eb export FOAM_INST_DIR=/proj/hani/OF_inst/Tetralith/foam; . $FOAM_INST_DIR/foam-extend-4.0/etc/bashrc foam ./Allwmake.firstInstall >& log& chgrp -R pg_hani Tetralith chmod -R go+Xr Tetralith
Test by submitting the damBreak case. First create a submit script named submitDambreak:
#!/bin/bash # #SBATCH -n 4 #SBATCH -t 00:20:00 #SBATCH -J damBreak #SBATCH -A snic2017-1-651 module load buildenv-intel/2018a-eb export FOAM_INST_DIR=/proj/hani/OF_inst/Tetralith/foam; source $FOAM_INST_DIR/foam-extend-4.0/etc/bashrc cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak . blockMesh -case damBreak setFields -case damBreak decomposePar -case damBreak mpirun interFoam -parallel -case damBreak &> result.out #mpprun did not work!!! # Script ends here
OpenFOAM-v1706+ in StuDAT (non-root), Redhat Enterprise 6.2 , 2017-09-07
Followed some of the instructions at http://openfoamwiki.net/index.php/Installation/Linux/OpenFOAM%2B-v1612%2B/CentOS_SL_RHEL
Skipping Qt and Paraview at the moment
cd /chalmers/sw/unsup64/OpenFOAM git clone https://develop.openfoam.com/Development/OpenFOAM-plus.git #This requires a user name and password from your registration mv OpenFOAM-plus OpenFOAM-v1706 cd OpenFOAM-v1706 git checkout OpenFOAM-v1706 cd ..
The following on one line (starting with wget, ending with vorboss, and no spaces after https):
wget -O ThirdParty-v1706.tgz https://downloads.sourceforge.net/project/openfoamplus/v1706/ThirdParty-v1706.tgz?r=http%3A%2F%2Fopenfoam.com%2Fcode%2Frepositories.php&ts=1502098894&use_mirror=vorboss
Press enter to get prompt back.
md5sum ThirdParty-v1706.tgz #Should give 003d38fe3f97ccaa291bafbf94c768eb tar xzf ThirdParty-v1706.tgz rm ThirdParty-v1706.tgz
gcc needs a higher version that 4.4.7, which is in the system. It was recommended at the above link to also install binutils, so I do that (it is fast). I also installed openmpi since it is not in the system. cmake is already version 3.7.2, which is above required for paraview and CGAL
sed -i -e 's=gmp-system=gmp-5.1.2=' -e 's=mpfr-system=mpfr-3.1.2=' -e 's=mpc-system=mpc-1.0.1=' OpenFOAM-v1706/etc/config.sh/compiler source /chalmers/sw/unsup64/OpenFOAM/OpenFOAM-v1706/etc/bashrc WM_COMPILER_TYPE=ThirdParty WM_MPLIB=OPENMPI cd $WM_THIRD_PARTY_DIR wget "https://raw.github.com/wyldckat/scripts4OpenFOAM3rdParty/master/getGcc" wget "https://raw.github.com/wyldckat/ThirdParty-2.0.x/binutils/makeBinutils" wget "https://raw.github.com/wyldckat/ThirdParty-2.0.x/binutils/getBinutils" chmod +x get* make* ./getGcc gcc-4.8.5 gmp-5.1.2 mpfr-3.1.2 mpc-1.0.1 ./makeGcc > log.mkgcc 2>&1 wmREFRESH #I actually didn't do this, but the next lines worked anyway. ./getBinutils ./makeBinutils gcc-4.8.5 >& log.mkbinutils #Not until gcc-4.8.5 is built ./makeOPENMPI >& log_makeOPENMPI
The following echo command once destroyed the original file. Make a copy of it first and make sure that the echo command worked.
cp /chalmers/sw/unsup64/OpenFOAM/addToBashrc /chalmers/sw/unsup64/OpenFOAM/addToBashrc_backup echo "alias OF1706+='source /chalmers/sw/unsup64/OpenFOAM/OpenFOAM-v1706/etc/bashrc WM_COMPILER_TYPE=ThirdParty WM_MPLIB=OPENMPI'" >> /chalmers/sw/unsup64/OpenFOAM/addToBashrc
Opened a new terminal window and typed:
OF1706+ foam ./Allwmake >& log.make
Make the installation available
foam cd .. chmod -R go+Xr OpenFOAM-v1706 chmod -R go+Xr ThirdParty-v1706
OpenFOAM-3.0.1 at Triolith (non-root), CentOS, 2017-09-07
HP SL230s compute nodes, equipped with two Intel E5-2660 (2.2 GHz Sandybridge) processors with 8 cores each, i.e 16 cores per node. Hyper-threading is not enabled. Infiniband from Mellanox (FDR IB, 56 Gb/s)
I do not bother about paraview at the moment
mkdir -p /proj/hani/OF_inst/Triolith/OpenFOAM cd /proj/hani/OF_inst/Triolith/OpenFOAM wget http://downloads.sourceforge.net/foam/OpenFOAM-3.0.1.tgz?use_mirror=mesh --output-document=OpenFOAM-3.0.1.tgz wget http://downloads.sourceforge.net/foam/ThirdParty-3.0.1.tgz?use_mirror=mesh --output-document=ThirdParty-3.0.1.tgz tar -xzf OpenFOAM-3.0.1.tgz tar -xzf ThirdParty-3.0.1.tgz rm OpenFOAM-3.0.1.tgz rm ThirdParty-3.0.1.tgz cp -r /proj/hani/OF_inst/Triolith/OpenFOAM/OpenFOAM-3.0.1/wmake/rules/linux64Icc /proj/hani/OF_inst/Triolith/OpenFOAM/OpenFOAM-3.0.1/wmake/rules/linux64IccHani sed -i s/fPIC/"fPIC -Nmpi -Nmkl -xavx"/g /proj/hani/OF_inst/Triolith/OpenFOAM/OpenFOAM-3.0.1/wmake/rules/linux64IccHani/c++
Used the currently recommended version at the cluster for impi, intel compiler, and mkl. (see instructions for v1606+):
alias OF301='export FOAM_INST_DIR=/proj/hani/OF_inst/Triolith/OpenFOAM; module load impi/5.0.2.044 intel/15.0.1.133 mkl/11.2.1; export MPI_ROOT=$I_MPI_ROOT; . $FOAM_INST_DIR/OpenFOAM-3.0.1/etc/bashrc WM_COMPILER=IccHani WM_MPLIB=INTELMPI' OF301 foam ./Allwmake >& log&
Problem with SloanRenumber: cannot find -lboost_thread
Commented the compilation of SloanRenumber in src/renumber/Allwmake:
- Code from here (not to paste)
#if [ -n "$BOOST_ARCH_PATH" ]
#then
# wmake $targetType SloanRenumber
#else
echo
echo "Skipping SloanRenumber"
echo
#fi
- Code to here (not to paste)
Problem with foamyMesh: cannot open source file "CGAL/Delaunay_triangulation_3.h", "CGAL/Delaunay_triangulation_2.h"
Commented compilation in applications/utilities/mesh/generation/Allwmake:
- Code from here (not to paste)
#if [ -n "$CGAL_ARCH_PATH" ] #then # foamyMesh/Allwmake #fi
- Code to here (not to paste)
The above are related, since CGAL requires boost. I do not bother at the moment since we don't use the above functionalities.
Note that as the above errors occurred it seems like some additional parts were not compiled, so run the next line a couple of times until all are already "up to date".
./Allwmake >& log&
Make it available:
cd .. chmod -R go+Xr OpenFOAM-3.0.1 ThirdParty-3.0.1 chgrp -R pg_hani OpenFOAM-3.0.1 ThirdParty-3.0.1
FOAM-extend-4.0 at Triolith (non-root), CentOS, 2017-04-28
cd /proj/hani/OF_inst/Triolith/foam git clone git://git.code.sf.net/p/foam-extend/foam-extend-3.2 foam-extend-4.0 cd foam-extend-4.0 git checkout nextRelease
Do required modifications to the installation procedure, in this case use Qt-4.8.6 from module (check version in ThirdParty/AllMake.stage4 and etc/prefs.sh-EXAMPLES):
module load qt/4.8.6 paraview/4.4.0 echo "export QT_DIR=/software/apps/qt/4.8.6" >> etc/prefs.sh echo "export QT_BIN_DIR=\$QT_DIR/bin" >> etc/prefs.sh echo "export PARAVIEW_SYSTEM=1" >> etc/prefs.sh echo "export PARAVIEW_DIR=/software/apps/paraview/4.4.0" >> etc/prefs.sh echo "export PARAVIEW_BIN_DIR=\$PARAVIEW_DIR/bin" >> etc/prefs.sh echo "export WM_COMPILER=Icc" >> etc/prefs.sh echo "export WM_MPLIB=INTELMPI" >> etc/prefs.sh echo "export MPI_ROOT=\$I_MPI_ROOT" >> etc/prefs.sh echo "export OPENMPI_DIR=\$I_MPI_ROOT/intel64" >> etc/prefs.sh
There is a problem with compilation of cfMesh using the Intel compiler giving messages like:
polyMeshGenModifierReorderBoundaryFaces.C(224): internal error: null pointer
Found a solution in:
https://www.cfd-online.com/Forums/openfoam-meshing-utilities/163578-cfmesh-installation-errors-supercomputer.html
Applied by:
sed -i.orig 's/EXE_INC/OMP_FLAGS =\nEXE_INC/g' $FOAM_SRC/mesh/cfMesh/Make/options
Used the currently recommended version at the cluster for impi, intel compiler, and mkl. Get the recommended version numbers by:
module load impi module load intel module load mkl
It was realized that mkl was needed by a compilation error message (when compiling v1606+). I don't know what it is used for.
It is necessary to add a couple of compilation flags for mpi at Triolith:
sed -i s/fPIC/"fPIC -Nmpi -Nmkl -xavx"/g $FOAM_INST_DIR/foam-extend-4.0/wmake/rules/linux64Icc/c++
(error message otherwise: Could not retrieve MPI tag from /proj/hani/OF_inst/Triolith/foam/foam-extend-4.0/applications/bin/linux64IccDPOpt/interFoam)
Source and prepare:
alias OFextend40='export FOAM_INST_DIR=/proj/hani/OF_inst/Triolith/foam; module load impi/5.0.2.044 intel/15.0.1.133 mkl/11.2.1 qt/4.8.6 paraview/4.4.0; . $FOAM_INST_DIR/foam-extend-4.0/etc/bashrc' OFextend40 foam
Compile all (perhaps do this a couple of times and check that all is compiled in the log file):
./Allwmake.firstInstall >& log&
Make it available:
cd .. chmod -R go+Xr foam-extend-4.0 chgrp -R pg_hani foam-extend-4.0
Test by submitting the damBreak case. First create a submit script named submitDambreak:
#!/bin/bash # #SBATCH -J damBreak #Name of job #SBATCH -t 00:30:00 #HH:MM:SS #SBATCH -N 1 #SBATCH --exclusive # #Same as alias OFextend40: export FOAM_INST_DIR=/proj/hani/OF_inst/Triolith/foam module load impi/5.0.2.044 intel/15.0.1.133 mkl/11.2.1 qt/4.8.6 paraview/4.4.0 . $FOAM_INST_DIR/foam-extend-4.0/etc/bashrc #Prepare cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak . blockMesh -case damBreak setFields -case damBreak decomposePar -case damBreak #--nranks is used when less than the number of available cores is used mpprun --nranks=4 `which interFoam` -parallel -case damBreak >& log # Script ends here
Submit by:
sbatch submitDambreak
Check if it is running (change x_hakni to your user name):
squeue -u x_hakni
OpenFOAM-4.x on Hebbe, 2017-04-10
mkdir -p /c3se/NOBACKUP/users/hani/hebbe/OpenFOAM cd /c3se/NOBACKUP/users/hani/hebbe/OpenFOAM git clone git://github.com/OpenFOAM/OpenFOAM-4.x.git git clone git://github.com/OpenFOAM/ThirdParty-4.x.git . /apps/new_modules.sh #To use the new modules ml load iimpi/2016b #For Intel compiler and MPI ml load Boost/1.61.0 #For SloanRenumber) ml load CGAL/4.8.1 #For foamyMesh)
Compiler: either GCC version 4.5 or above; or LLVM Clang version 3.6 or above; or Intel ICC version 15.0.3 or above. GCC is most commonly available and the version can be checked by typing
icc --version
Gives:
icc (ICC) 16.0.3 20160415
Source (note that the installation path is automatically detected):
. OpenFOAM-4.x/etc/bashrc WM_COMPILER=Icc WM_MPLIB=INTELMPI MPI_ROOT=/apps/Hebbe/software/Compiler/intel/2016.3.210-GCC-5.4.0-2.26/impi/5.1.3.181
Compile ThirdParty:
cd ThirdParty-4.x ./Allwmake >& log& ./Allwmake >& log& #Check log file for errors
Skipped compiling Paraview
Compile OpenFOAM:
cd ../OpenFOAM-4.x ./Allwmake -j >& log& ./Allwmake -j >& log& #Got some HANG-UP the first time, so continued with this line ./Allwmake -j >& log& #Check for errors in the log file rm log
Meanwhile, check revision:
git rev-parse --short HEAD
In this case:
56a4152
Example of a submit script for parallel simulation (submit by "sbatch -A PROJ_NUMBER scriptName"):
#!/usr/bin/env bash #SBATCH -N 1 -n 4 #Will use only 4 cores on 1 node (there are 20 cores per node on Hebbe) #SBATCH -p hebbe #SBATCH -t 00:01:00 #---------------- Uncomment ONE module line ------------------------- . /apps/new_modules.sh; ml load iimpi/2016b Boost/1.61.0 CGAL/4.8.1; . /c3se/NOBACKUP/users/hani/hebbe/OpenFOAM/OpenFOAM-4.x/etc/bashrc WM_COMPILER=Icc WM_MPLIB=INTELMPI MPI_ROOT=/apps/Hebbe/software/Compiler/intel/2016.3.210-GCC-5.4.0-2.26/impi/5.1.3.181 #------------ Copy and run the damBreak tutorial -------------------- cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak/damBreak . cd damBreak blockMesh >& log_blockMesh #Generates the grid setFields >& log_setFields #Sets the initial fields decomposePar >& log_decomposePar #Decomposes for parallel simulation mpirun interFoam -parallel >& log #Runs interFoam in parallel on 4 cores (flags -N 1 -n 4 above) reconstructPar >& log_reconstructPar #Collects the decomposed results rm -rf processor* #Deletes the decomposed files
Example of submitscript for sequential simulation:
#!/usr/bin/env bash #SBATCH -N 1 -n 1 #SBATCH -p hebbe #SBATCH -t 00:01:00 #---------------- Uncomment ONE module line ------------------------- . /apps/new_modules.sh; ml load iimpi/2016b Boost/1.61.0 CGAL/4.8.1; . /c3se/NOBACKUP/users/hani/hebbe/OpenFOAM/OpenFOAM-4.x/etc/bashrc WM_COMPILER=Icc WM_MPLIB=INTELMPI MPI_ROOT=/apps/Hebbe/software/Compiler/intel/2016.3.210-GCC-5.4.0-2.26/impi/5.1.3.181 #------------ Copy and run the damBreak tutorial -------------------- cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak/damBreak . cd damBreak blockMesh >& log_blockMesh #Generates the grid setFields >& log_setFields #Sets the initial fields interFoam >& log #Runs interFoam in sequential mode
Run at log-in node (only for very short tests and small cases): Add to the end of your $HOME/.bashrc file:
. /c3se/NOBACKUP/users/hani/hebbe/addToBashrc
Then open a new window at Hebbe and type:
OF4x mkdir -p $FOAM_RUN cd $FOAM_RUN cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak/damBreak . cd damBreak blockMesh >& log_blockMesh #Generates the grid setFields >& log_setFields #Sets the initial fields interFoam >& log& #Runs interFoam in sequential mode
The rest is not tested!!!
Paraview is already available in two versions. You see them by typing:
module load extras module avail
Then either:
module load paraview/3.14
or:
module load paraview/4.4.0
and then:
paraFoam -nativeReader
OpenFOAM-v16.06+ on Triolith, CentOS, 2017-01-25
HP SL230s compute nodes, equipped with two Intel E5-2660 (2.2 GHz Sandybridge) processors with 8 cores each, i.e 16 cores per node. Hyper-threading is not enabled. Infiniband from Mellanox (FDR IB, 56 Gb/s)
I do not bother about paraview at the moment
mkdir -p /proj/hani/OF_inst/Triolith/OpenFOAM cd /proj/hani/OF_inst/Triolith/OpenFOAM wget "http://downloads.sourceforge.net/openfoamplus/files/OpenFOAM-v1606%2B.tgz?use_mirror=mesh" -O OpenFOAM-v1606+.tgz wget "http://downloads.sourceforge.net/openfoamplus/files/ThirdParty-v1606%2B.tgz?use_mirror=mesh" -O ThirdParty-v1606+.tgz tar -xzf OpenFOAM-v1606+.tgz tar -xzf ThirdParty-v1606+.tgz rm OpenFOAM-v1606+.tgz rm ThirdParty-v1606+.tgz cp -r $FOAM_INST_DIR/OpenFOAM-v1606+/wmake/rules/linux64Icc $FOAM_INST_DIR/OpenFOAM-v1606+/wmake/rules/linux64IccHani sed -i s/fPIC/"fPIC -Nmpi -Nmkl -xavx"/g $FOAM_INST_DIR/OpenFOAM-v1606+/wmake/rules/linux64IccHani/c++
Used the currently recommended version at the cluster for impi, intel compiler, and mkl. Get the recommended version numbers by:
module load impi module load intel module load mkl
It was realized that mkl was needed by a compilation error message. I don't know what it is used for.
alias OF1606+='export FOAM_INST_DIR=/proj/hani/OF_inst/Triolith/OpenFOAM; module load impi/5.0.2.044 intel/15.0.1.133 mkl/11.2.1; export MPI_ROOT=$I_MPI_ROOT; . $FOAM_INST_DIR/OpenFOAM-v1606+/etc/bashrc WM_COMPILER=IccHani WM_MPLIB=INTELMPI' OF1606+ foam
Remove two lines with InterfaceForce in src/lagrangian/intermediate/parcels/include/makeParcelForces.H and the final backslash on the preceding line. There was an error message about the templating. We are currently not interested in that code, so let's leave it for now.
./Allwmake >& log&
Make it available:
cd .. chmod -R go+Xr OpenFOAM-v1606+ ThirdParty-v1606+ chgrp -R pg_hani OpenFOAM-v1606+ ThirdParty-v1606+
Test by submitting the damBreak case. First create a submit script:
#!/bin/bash # #SBATCH -J damBreak #Name of job #SBATCH -t 00:30:00 #HH:MM:SS #SBATCH -N 1 #SBATCH --exclusive # #Same as alias OF1606+: export FOAM_INST_DIR=/proj/hani/OF_inst/Triolith/OpenFOAM module load impi/5.0.2.044 intel/15.0.1.133 mkl/11.2.1 export MPI_ROOT=$I_MPI_ROOT . $FOAM_INST_DIR/OpenFOAM-v1606+/etc/bashrc WM_COMPILER=IccHani WM_MPLIB=INTELMPI #Prepare cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak/damBreak . blockMesh -case damBreak setFields -case damBreak decomposePar -case damBreak #--nranks is used when less than the number of available cores is used mpprun --nranks=4 `which interFoam` -parallel -case damBreak >& log # Script ends here
Submit by:
sbatch submitDambreak
Check if it is running (change x_hakni to your user name):
squeue -u x_hakni
OpenFOAM-4.x in StuDAT (non-root), Redhat Enterprise 6.2 , 2016-11-07
This is work in progress. I have come to the point where you see a marker text.
For next time, first have a look at the following link, which has most of the information in a different way: [1]
cd /chalmers/sw/unsup64/OpenFOAM git clone git://github.com/OpenFOAM/OpenFOAM-4.x.git git clone git://github.com/OpenFOAM/ThirdParty-4.x.git cp OpenFOAM-4.x/etc/bashrc OpenFOAM-4.x/etc/bashrcHani cp OpenFOAM-4.x/etc/config.sh/settings OpenFOAM-4.x/etc/config.sh/settingsHani sed -i s/settings/settingsHani/g OpenFOAM-4.x/etc/bashrcHani
Check available versions:
gcc --version #Gives 4.4.7. Required: 4.7+. NOT OK! - using ThirdParty Gcc flex --version #Gives 2.5.35. Seems to have worked ok!
Check out the makeGcc?? files and Gcc in settings.sh to decide which gcc version to use - in this case we need to upgrade through ThirdParty (Gcc version can also be checked in etc/config.sh/compiler)
sed -i s/"WM_COMPILER_TYPE=system"/"WM_COMPILER_TYPE=ThirdParty"/g OpenFOAM-4.x/etc/bashrcHani sed -i s/"bashrc"/"bashrcHani"/g OpenFOAM-4.x/etc/config.sh/aliases
The following line destroyed the original file when installing 2.4.x (never happened before or after). Make a back-up copy first!
echo "alias OF4x='export FOAM_INST_DIR=/chalmers/sw/unsup64/OpenFOAM;. \$FOAM_INST_DIR/OpenFOAM-4.x/etc/bashrcHani'" >> addToBashrc
Ignore the warning after the following line is executed (we use system Gcc for compilation of Gcc etc.):
export FOAM_INST_DIR=/chalmers/sw/unsup64/OpenFOAM;. $FOAM_INST_DIR/OpenFOAM-4.x/etc/bashrcHani
See ThirdParty-4.x/makeGcc for which versions of dependent packages you need.
cd ThirdParty-4.x wget http://ftp.gnu.org/gnu/gmp/gmp-5.1.2.tar.bz2 bunzip2 gmp-5.1.2.tar.bz2 tar xf gmp-5.1.2.tar rm gmp-5.1.2.tar wget http://www.mpfr.org/mpfr-3.1.2/mpfr-3.1.2.tar.gz tar xzf mpfr-3.1.2.tar.gz rm mpfr-3.1.2.tar.gz wget http://www.multiprecision.org/mpc/download/mpc-1.0.1.tar.gz tar xzf mpc-1.0.1.tar.gz rm mpc-1.0.1.tar.gz wget https://ftp.gnu.org/gnu/gcc/gcc-4.8.5/gcc-4.8.5.tar.gz tar xzf gcc-4.8.5.tar.gz rm gcc-4.8.5.tar.gz ./makeGcc gcc-4.8.5 >& log_makeGcc rm log_makeGcc #After looking at it, making sure that it went through
Check that gcc is installed:
which gcc
Returns /chalmers/sw/unsup64/OpenFOAM/ThirdParty-3.0.x/platforms/linux64/gcc-4.8.4/bin/gcc
(For the following, see http://openfoamwiki.net/index.php/Installation/Linux/OpenFOAM-4.x/CentOS_SL_RHEL):
wget http://www.cmake.org/files/v3.2/cmake-3.2.1.tar.gz wget https://github.com/CGAL/cgal/releases/download/releases%2FCGAL-4.8/CGAL-4.8.tar.xz wget http://sourceforge.net/projects/boost/files/boost/1.55.0/boost_1_55_0.tar.bz2 tar xzf cmake-3.2.1.tar.gz tar -xJf CGAL-4.8.tar.xz tar -xjf boost_1_55_0.tar.bz2 rm cmake-3.2.1.tar.gz rm CGAL-4.8.tar.xz rm boost_1_55_0.tar.bz2 sed -i s/"config.sh\/CGAL"/"config.sh\/CGALHani"/g ../OpenFOAM-4.x/etc/bashrcHani sed -i s/"config.sh\/CGAL"/"config.sh\/CGALHani"/g makeCGAL cp ../OpenFOAM-4.x/etc/config.sh/CGAL ../OpenFOAM-4.x/etc/config.sh/CGALHani sed -i -e 's/\(boost_version=\)boost-system/\1boost_1_55_0/' ../OpenFOAM-4.x/etc/config.sh/CGALHani sed -i -e 's/\(cgal_version=\)cgal-system/\1CGAL-4.8/' ../OpenFOAM-4.x/etc/config.sh/CGALHani ./makeCmake >& log_makeCmake rm log_makeCmake wmRefresh wget "https://raw.github.com/wyldckat/ThirdParty-2.0.x/binutils/makeBinutils" wget "https://raw.github.com/wyldckat/ThirdParty-2.0.x/binutils/getBinutils" chmod +x get* make* ./getBinutils ./makeBinutils gcc-4.8.5 > log.makeBinutils 2>&1 rm log.makeBinutils #CGAL compilation did not work the first attempt, but after opening a new terminal window and sourcing it worked: ./makeCGAL boost_1_55_0 gmp-5.1.2 mpfr-3.1.2 >& log_makeCGAL #Makes sure to use the ones we installed rm log_makeCGAL wmRefresh
ParaView 5.0.1 needs at least Qt 4.7. Therefore, we'll need to do a custom build of Qt 4.8.6. This failed the first attempt. Second attemt worked, after opening a new terminal window, sourcing, and compiling CGAL.
While compiling Qt it is also possible to compile OpenFOAM (Allwmake below)
cd $WM_THIRD_PARTY_DIR wget https://github.com/wyldckat/scripts4OpenFOAM3rdParty/raw/master/getQt wget https://github.com/OpenFOAM/ThirdParty-2.4.x/raw/master/makeQt wget -P etc/tools/ https://github.com/OpenFOAM/ThirdParty-2.4.x/raw/master/etc/tools/QtFunctions chmod +x getQt makeQt sed -i -e 's=4\.6=4.8=' -e 's=4\.8\.4=4.8.6=' -e 's=/\$major/\$tarFile=/$major/$version/$tarFile=' getQt ./getQt ./makeQt qt-4.8.6 > log.makeQt 2>&1 rm log.makeQt #After looking at it, making sure that it went through
Had a problem with MPI when compiling OF (try it first by compiling OF), so I did this to switch to an older mpi version that will be compiled as ThirdParty:
foam cp -r ../ThirdParty-2.3.x/openmpi-1.6.5 ../ThirdParty-4.x sed -i s/"openmpi-1.10.2"/"openmpi-1.6.5"/g etc/config.sh/mpi sed -i s/SYSTEMOPENMPI/OPENMPI/g etc/bashrcHani
Update the shell environment
wmRefresh
Check that the correct gcc is used (the one just installed):
which gcc foam ./Allwmake -j 4 >& log& ./Allwmake -j 4 >& log& rm log #Had several errors first time. Second time CGAL was compiled, and only error with conformalVoronioMesh (foamymesh) which I do not care about.
I am here.
Compile ParaView with python and mpi (this requires mpi to be compiled first)
./makeParaView -qmake $WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER/qt-4.8.6/bin/qmake -mpi -python > log.makePV 2>&1 wmRefresh
Compile ParaFoam reader:
cd $FOAM_UTILITIES/postProcessing/graphics/PVReaders ./Allwclean ./Allwmake
Make the installation available
foam cd .. chmod -R go+Xr OpenFOAM-4.x chmod -R go+Xr ThirdParty-4.x
FOAM-extend-4.0 in StuDAT (non-root), RedHat Enterprise 6.2, 2016-10-18
This is work in progress
cd /chalmers/sw/unsup64/OpenFOAM git clone git://git.code.sf.net/p/foam-extend/foam-extend-3.2 foam-extend-4.0 cd foam-extend-4.0 git checkout nextRelease
Do required modifications to the installation procedure, in this case point at the Qt installation of FOAM-extend-3.1, since the system installed is below the required Qt version 4.7.0, by ParaView-4.3.1 (check etc/prefs.sh-EXAMPLES):
echo "export QT_DIR=/chalmers/sw/unsup64/OpenFOAM/foam-extend-3.1/ThirdParty/packages/qt-everywhere-opensource-src-4.8.5/platforms/linux64GccDPOpt" > etc/prefs.sh echo "export QT_BIN_DIR=\$QT_DIR/bin" >> etc/prefs.sh
alias OFextend40='export FOAM_INST_DIR=/chalmers/sw/unsup64/OpenFOAM;. $FOAM_INST_DIR/foam-extend-4.0/etc/bashrc' OFextend40 foam
Compile all:
./Allwmake.firstInstall >& log&
libccmio failed compiling since the tarball was not found. I changed in AllMake.stage3 to ftp://www.daba.lv/pub/TIS/bibliotekas/dazadas/libccmio-2.6.1.tar.gz and recompiled successfully.
Make it available:
chmod -R go+Xr .
Updated 161123
Made a tar file of the existing installation, as a backup
Checked modifications done by me on original files, using 'git status'
Moved ThirdParty/AllMake.stage3 to a backup file, since it was modified
Checked difference between my installation and remote:
git fetch origin
git diff HEAD...origin/nextRelease
Downloaded modifications:
git pull
Compiled:
./Allwmake (not running installations of ThridParty again)
Made it available:
chmod -R go+Xr .
Compile Debug version, without recompiling ThirdParty:
cd /chalmers/sw/unsup64/OpenFOAM echo "alias OFextend40Debug='export FOAM_INST_DIR=/chalmers/sw/unsup64/OpenFOAM;. \$FOAM_INST_DIR/foam-extend-4.0/etc/bashrc WM_COMPILE_OPTION=Debug'" >> addToBashrc
Open new terminal window
OFextend40Debug cd $WM_THIRD_PARTY_DIR/packages/cmake-3.2.2/platforms ln -s linux64GccDPOpt linux64GccDPDebug cd $WM_THIRD_PARTY_DIR/packages/hwloc-1.10.1/platforms ln -s linux64GccDPOpt linux64GccDPDebug cd $WM_THIRD_PARTY_DIR/packages/libccmio-2.6.1/platforms ln -s linux64GccDPOpt linux64GccDPDebug cd $WM_THIRD_PARTY_DIR/packages/mesquite-2.1.2/platforms ln -s linux64GccDPOpt linux64GccDPDebug cd $WM_THIRD_PARTY_DIR/packages/metis-5.1.0/platforms ln -s linux64GccDPOpt linux64GccDPDebug cd $WM_THIRD_PARTY_DIR/packages/openmpi-1.6.5/platforms ln -s linux64GccDPOpt linux64GccDPDebug cd $WM_THIRD_PARTY_DIR/packages/ParaView-4.3.1/platforms ln -s linux64GccDPOpt linux64GccDPDebug cd $WM_THIRD_PARTY_DIR/packages/parmetis-4.0.3/platforms ln -s linux64GccDPOpt linux64GccDPDebug cd $WM_THIRD_PARTY_DIR/packages/ParMGridGen-1.0/platforms ln -s linux64GccDPOpt linux64GccDPDebug cd $WM_THIRD_PARTY_DIR/packages/scotch-6.0.4/platforms ln -s linux64GccDPOpt linux64GccDPDebug
Open a new terminal:
OFextend40Debug foam ./Allwmake >& log&
Made it available:
chmod -R go+Xr .
OpenFOAM-3.0.x in StuDAT (non-root), Redhat Enterprise 6.2 , 2015-12-10
cd /chalmers/sw/unsup64/OpenFOAM git clone git://github.com/OpenFOAM/OpenFOAM-3.0.x.git #Try this next time (but then additional tar files need to be downloaded, see wiki link below): git clone https://github.com/OpenFOAM/ThirdParty-3.0.x.git wget http://downloads.sourceforge.net/project/foam/foam/3.0.0/ThirdParty-3.0.0.tgz?r=http%3A%2F%2Fopenfoam.org%2Fdownload%2Fgit.php&ts=1449766237&use_mirror=netix
Maybe necessary to rename the downloaded file to ThirdParty-3.0.0.tgz
tar xzf ThirdParty-3.0.0.tgz rm ThirdParty-3.0.0.tgz mv ThirdParty-3.0.0 ThirdParty-3.0.x cp OpenFOAM-3.0.x/etc/bashrc OpenFOAM-3.0.x/etc/bashrcHani cp OpenFOAM-3.0.x/etc/config/settings.sh OpenFOAM-3.0.x/etc/config/settingsHani.sh sed -i s/settings.sh/settingsHani.sh/g OpenFOAM-3.0.x/etc/bashrcHani
Check available versions:
gcc --version #Gives 4.4.7. Required: 4.7+. NOT OK! - using ThirdParty Gcc flex --version #Gives 2.5.35. Seems to have worked ok!
Check out the makeGcc?? files and Gcc in settings.sh to decide which gcc version to use - in this case we need to upgrade through ThirdParty
sed -i s/"foamCompiler=system"/"foamCompiler=ThirdParty"/g OpenFOAM-3.0.x/etc/bashrcHani sed -i s/". \$WM\_PROJECT\_DIR\/etc\/bashrc"/". \$WM\_PROJECT\_DIR\/etc\/bashrcHani"/g OpenFOAM-3.0.x/etc/config/aliases.sh
The following line destroyed the original file when installing 2.4.x (never happened before or after). Make a back-up copy first!
echo "alias OF30x='export FOAM_INST_DIR=/chalmers/sw/unsup64/OpenFOAM;. \$FOAM_INST_DIR/OpenFOAM-3.0.x/etc/bashrcHani'" >> addToBashrc
Ignore the warning after the following line is executed (we use system Gcc for compilation of Gcc etc.):
export FOAM_INST_DIR=/chalmers/sw/unsup64/OpenFOAM;. $FOAM_INST_DIR/OpenFOAM-3.0.x/etc/bashrcHani WM_NCOMPPROCS=4
See ThirdParty-3.0.x/makeGcc for which versions of dependent packages you need.
cd ThirdParty-3.0.x wget http://ftp.gnu.org/gnu/gmp/gmp-5.1.2.tar.bz2 bunzip2 gmp-5.1.2.tar.bz2 tar xf gmp-5.1.2.tar rm gmp-5.1.2.tar wget http://www.mpfr.org/mpfr-3.1.2/mpfr-3.1.2.tar.gz tar xzf mpfr-3.1.2.tar.gz rm mpfr-3.1.2.tar.gz wget http://www.multiprecision.org/mpc/download/mpc-1.0.1.tar.gz tar xzf mpc-1.0.1.tar.gz rm mpc-1.0.1.tar.gz wget https://ftp.gnu.org/gnu/gcc/gcc-4.8.4/gcc-4.8.4.tar.gz tar xzf gcc-4.8.4.tar.gz rm gcc-4.8.4.tar.gz ./makeGcc gcc-4.8.4 >& log_makeGcc rm log_makeGcc #After looking at it, making sure that it went through
Check that gcc is installed:
which gcc
Returns /chalmers/sw/unsup64/OpenFOAM/ThirdParty-3.0.x/platforms/linux64/gcc-4.8.4/bin/gcc
(For the following, see http://openfoamwiki.net/index.php/Installation/Linux/OpenFOAM-3.0.x/CentOS_SL_RHEL#CentOS_6.7):
(I complied CGAL before cmake contrary to the description in the wiki)
wget http://www.cmake.org/files/v3.2/cmake-3.2.1.tar.gz tar xzf cmake-3.2.1.tar.gz rm cmake-3.2.1.tar.gz wget http://sourceforge.net/projects/boost/files/boost/1.55.0/boost_1_55_0.tar.bz2 tar -xjf boost_1_55_0.tar.bz2 rm boost_1_55_0.tar.bz2 sed -i s/"CGAL.sh"/"CGALHani.sh"/g ../OpenFOAM-3.0.x/etc/bashrcHani cp ../OpenFOAM-2.4.x/etc/config/CGAL.sh ../OpenFOAM-3.0.x/etc/config/CGALHani.sh sed -i -e 's=boost-system=boost_1_55_0=' ../OpenFOAM-3.0.x/etc/config/CGALHani.sh ./makeCGAL boost_1_55_0 gmp-5.1.2 mpfr-3.1.2 >& log_makeCGAL #Makes sure that also boost, gmp and mpfr are compiled rm log_makeCGAL ./makeCmake >& log_makeCmake rm log_makeCmake
ParaView 4.4.0 needs at least Qt 4.7. Therefore, we'll need to do a custom build of Qt 4.8.6
wget https://github.com/wyldckat/scripts4OpenFOAM3rdParty/raw/master/getQt chmod +x getQt sed -i -e 's=4\.6=4.8=' -e 's=4\.8\.4=4.8.6=' -e 's=/\$major/\$tarFile=/$major/$version/$tarFile=' getQt ./getQt #makeQt missing in download. Get it from old installation: cp ../ThirdParty-2.4.x/makeQt . cp ../ThirdParty-2.4.x/etc/tools/QtFunctions etc/tools/ ./makeQt qt-4.8.6 >& log_makeQt #YOU HAVE TO TYPE "yes" TO ACCEPT THE LGPL LICENCE rm log_makeQt #After looking at it, making sure that it went through
Had a problem with MPI when compiling OF (try it first by compiling OF), so I did this to switch to an older mpi version that will be compiled as ThirdParty:
foam cp -r ../ThirdParty-2.3.x/openmpi-1.6.5 ../ThirdParty-3.0.x sed -i s/"openmpi-1.10.0"/"openmpi-1.6.5"/g etc/config/settingsHani.sh sed -i s/SYSTEMOPENMPI/OPENMPI/g etc/bashrcHani
Update the shell environment
wmSET $FOAM_SETTINGS
Check that the correct gcc is used (the one just installed):
which gcc foam ./Allwmake -j 4 >& log& ./Allwmake -j 4 >& log& rm log
Compile ParaView with python and mpi (this requires mpi to be compiled first)
#We need to apply a fix for a problem with reading multiple solids in an STL wget https://raw.githubusercontent.com/Kitware/VTK/40937e934308e5009e80769dc0c451ee4f157749/IO/Geometry/vtkSTLReader.cxx -O ParaView-4.4.0/VTK/IO/Geometry/vtkSTLReader.cxx #this will take a while... somewhere between 30 minutes to 2 hours or more ./makeParaView4 -qmake $WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER/qt-4.8.6/bin/qmake -mpi -python >& log_makeParaView4
Compile ParaFoam reader:
cd $FOAM_UTILITIES/postProcessing/graphics/PV4Readers wmSET $FOAM_SETTINGS ./Allwclean ./Allwmake
Make the installation available
foam cd .. chmod -R go+Xr OpenFOAM-3.0.x chmod -R go+Xr ThirdParty-3.0.x
cfMesh-1.1.1 for OpenFOAM-2.4.x in StuDAT, 2015-12-07
This was done after completing the OpenFOAM-2.4.x installation (see below).
See e.g. http://www.cfd-online.com/Forums/openfoam-meshing-utilities/160757-cfmesh-v1-1-1-a.html
OF24x foam cd ../ThirdParty-2.4.x/ wget http://downloads.sourceforge.net/project/cfmesh/v1.1.1/cfMesh-v1.1.1.tgz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fcfmesh%2F&ts=1449478534&use_mirror=skylink mv cfMesh-v1.1.1.tgz\?r\=http\:%2F%2Fsourceforge.net%2Fprojects%2Fcfmesh%2F cfMesh-v1.1.1.tgz tar xzf cfMesh-v1.1.1.tgz rm cfMesh-v1.1.1.tgz cd cfMesh-v1.1.1
Prepare for site installation:
sed -i s/FOAM_USER_LIBBIN/FOAM_LIBBIN/g meshLibrary/Make/files sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g executables/cartesian2DMesh/Make/files sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g executables/cartesianMesh/Make/files sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g executables/pMesh/Make/files sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g executables/tetMesh/Make/files sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/checkSurfaceMesh/Make/files sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/copySurfaceParts/Make/files sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/extrudeEdgesInto2DSurface/Make/files sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/FLMAToSurface/Make/files sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/FMSToSurface/Make/files sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/FMSToVTK/Make/files sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/generateBoundaryLayers/Make/files sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/importSurfaceAsSubset/Make/files sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/improveMeshQuality/Make/files sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/improveSymmetryPlanes/Make/files sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/mergeSurfacePatches/Make/files sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/meshToFPMA/Make/files sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/patchesToSubsets/Make/files sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/preparePar/Make/files sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/removeSurfaceFacets/Make/files sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/scaleMesh/Make/files sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/scaleSurfaceMesh/Make/files sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/subsetToPatch/Make/files sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/surfaceFeatureEdges/Make/files sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/surfaceGenerateBoundingBox/Make/files sed -i s/FOAM_USER_APPBIN/FOAM_APPBIN/g utilities/surfaceToFMS/Make/files
Compile:
./Allwmake
Change rights:
cd .. chmod -R go+Xr cfMesh-v1.1.1 foam chmod -R go+Xr platforms
FOAM-extend-3.2 compiled with Gcc on Hebbe, 2015-12-02
After the compilation with Intel (below) I did the following:
Set the alias in the bashrc file:
alias OFextend32Gcc="module load gcc/4.9/4.9.2 openmpi/1.10.0; export FOAM_INST_DIR=/c3se/NOBACKUP/users/hani/hebbe/foam;. \$FOAM_INST_DIR/foam-extend-3.2/etc/bashrc WM_MPLIB=SYSTEMOPENMPI PARAVIEW_SYSTEM=1 FOAM_VERBOSE=1; unset WM_THIRD_PARTY_USE_OPENMPI_165; unset WM_THIRD_PARTY_USE_PARAVIEW_431"
Compile, in a new terminal window:
OFextend32Gcc foam ./Allwmake.firstInstall >& log_Allwmake.firstInstall& chgrp -R c3-fluiddyn /c3se/NOBACKUP/users/hani/hebbe chmod -R g-w+Xr /c3se/NOBACKUP/users/hani/hebbe chmod -R o-wrx /c3se/NOBACKUP/users/hani/hebbe
Exchange the "module" line in the submit script with the corresponding parts of the above alias.
FOAM-extend-3.2 compiled with Intel on Hebbe, 2015-10-15
mkdir -p /c3se/NOBACKUP/users/hani/hebbe/foam cd /c3se/NOBACKUP/users/hani/hebbe/foam git clone git://git.code.sf.net/p/foam-extend/foam-extend-3.2 foam-extend-3.2 cd foam-extend-3.2 git checkout nextRelease echo 'alias OFextend32="module load intel-compilers/15.0/090 intel-mpi/5.0.3.048; export FOAM_INST_DIR=/c3se/NOBACKUP/users/hani/hebbe/foam;. \$FOAM_INST_DIR/foam-extend-3.2/etc/bashrc WM_COMPILER=Icc WM_MPLIB=INTELMPI MPI_ROOT=/c3se/apps/Common/intel/impi/5.0.3.048 PARAVIEW_SYSTEM=1 FOAM_VERBOSE=1; unset WM_THIRD_PARTY_USE_OPENMPI_165; unset WM_THIRD_PARTY_USE_PARAVIEW_431; export OPENMPI_DIR=/c3se/apps/Common/intel/impi/5.0.3.048/intel64"' >> ~/.bashrc
(OPENMPI_DIR is needed for parMetisDecomp, to find the lib directory of INTELMPI - ticket filed)
In a new terminal window:
OFextend32 foam ./Allwmake.firstInstall >& log_Allwmake.firstInstall& chgrp -R c3-fluiddyn /c3se/NOBACKUP/users/hani/hebbe chmod -R g-w+Xr /c3se/NOBACKUP/users/hani/hebbe chmod -R o-wrx /c3se/NOBACKUP/users/hani/hebbe
Example of a submit script for parallel simulation (submit by "sbatch scriptName -A PROJ_NUMBER"):
#!/usr/bin/env bash #SBATCH -N 1 -n 4 #Will use only 4 cores on 1 node (there are 20 cores per node on Hebbe) #SBATCH -p hebbe #SBATCH -t 00:01:00 #---------------- Uncomment ONE module line ------------------------- module load intel-compilers/15.0/090 intel-mpi/5.0.3.048; export FOAM_INST_DIR=/c3se/NOBACKUP/users/hani/hebbe/foam;. $FOAM_INST_DIR/foam-extend-3.2/etc/bashrc WM_COMPILER=Icc WM_MPLIB=INTELMPI MPI_ROOT=/c3se/apps/Common/intel/impi/5.0.3.048 PARAVIEW_SYSTEM=1 FOAM_VERBOSE=1; unset WM_THIRD_PARTY_USE_OPENMPI_165; unset WM_THIRD_PARTY_USE_PARAVIEW_431; export OPENMPI_DIR=/c3se/apps/Common/intel/impi/5.0.3.048/intel64 #------------ Copy and run the damBreak tutorial -------------------- cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak . cd damBreak blockMesh >& log_blockMesh #Generates the grid setFields >& log_setFields #Sets the initial fields decomposePar >& log_decomposePar #Decomposes for parallel simulation mpirun interFoam -parallel >& log #Runs interFoam in parallel on 4 cores (flags -N 1 -n 4 above) reconstructPar >& log_reconstructPar #Collects the decomposed results rm -rf processor* #Deletes the decomposed files
Example of submitscript for sequential simulation:
#!/usr/bin/env bash #SBATCH -N 1 -n 1 #SBATCH -p hebbe #SBATCH -t 00:01:00 #---------------- Uncomment ONE module line ------------------------- module load intel-compilers/15.0/090 intel-mpi/5.0.3.048; export FOAM_INST_DIR=/c3se/NOBACKUP/users/hani/hebbe/foam;. $FOAM_INST_DIR/foam-extend-3.2/etc/bashrc WM_COMPILER=Icc WM_MPLIB=INTELMPI MPI_ROOT=/c3se/apps/Common/intel/impi/5.0.3.048 PARAVIEW_SYSTEM=1 FOAM_VERBOSE=1; unset WM_THIRD_PARTY_USE_OPENMPI_165; unset WM_THIRD_PARTY_USE_PARAVIEW_431; export OPENMPI_DIR=/c3se/apps/Common/intel/impi/5.0.3.048/intel64 #------------ Copy and run the damBreak tutorial -------------------- cp -r $FOAM_TUTORIALS/multiphase/interFoam/ras/damBreak . cd damBreak blockMesh >& log_blockMesh #Generates the grid setFields >& log_setFields #Sets the initial fields interFoam >& log #Runs interFoam in sequential mode
This should probably also work, to compile with system open-mpi, but I didn't verify it completely:
echo 'alias OFextend32="module load intel-compilers/15.0/090 openmpi/1.8.3-icc; export FOAM_INST_DIR=/c3se/NOBACKUP/users/hani/hebbe/foam;. \$FOAM_INST_DIR/foam-extend-3.2/etc/bashrc WM_COMPILER=Icc WM_MPLIB=SYSTEMOPENMPI OPENMPI_DIR=/c3se/apps/Hebbe/openmpi/1.8.3 OPENMPI_BIN_DIR=/c3se/apps/Hebbe/openmpi/1.8.3/bin PARAVIEW_SYSTEM=1 FOAM_VERBOSE=1; unset WM_THIRD_PARTY_USE_OPENMPI_165; unset WM_THIRD_PARTY_USE_PARAVIEW_431"' >> ~/.bashrc
Paraview is already available in two versions. You see them by typing:
module load extras module avail
Then either:
module load paraview/3.14
or:
module load paraview/4.4.0
and then:
paraFoam -nativeReader
FOAM-extend-3.2 in StuDAT (non-root), Redhat Enterprise 6.2 , 2015-09-24
cd /chalmers/sw/unsup64/OpenFOAM
Note: downloaded after official release of 3.2, but still as branch in 3.1 (can also be done as git clone --branch v3.2 http://git.code.sf.net/p/openfoam-extend/foam-extend-3.1 foam-extend-3.2)
git clone http://git.code.sf.net/p/openfoam-extend/foam-extend-3.1 foam-extend-3.2 cd foam-extend-3.2 git checkout v3.2 alias OFextend32='export FOAM_INST_DIR=/chalmers/sw/unsup64/OpenFOAM;. $FOAM_INST_DIR/foam-extend-3.2/etc/bashrc' OFextend32 foam
Do required modifications to the installation procedure, in this case point at the Qt installation of FOAM-extend-3.1, since the system installed is below the required Qt version 4.7.0, by ParaView-4.3.1 (check etc/prefs.sh-EXAMPLES):
echo "export QT_DIR=/chalmers/sw/unsup64/OpenFOAM/foam-extend-3.1/ThirdParty/packages/qt-everywhere-opensource-src-4.8.5/platforms/linux64GccDPOpt" > etc/prefs.sh echo "export QT_BIN_DIR=\$QT_DIR/bin" >> etc/prefs.sh
Compile all:
./Allwmake.firstInstall >& log&
Make it available:
chmod -R go+Xr .
paraFoam did not work remotely, but I will test locally later.
Old notes, to check:
ParaView probably fails compiling. If so, do:
export WM_NCOMPPROCS=1 ./Allwmake.firstInstall >& log& rm log chmod -R go+Xr .
OpenFOAM-2.4 in StuDAT (non-root), Redhat Enterprise 6.2 , 2015-08-18
git clone git://github.com/OpenFOAM/OpenFOAM-2.4.x.git wget http://downloads.sourceforge.net/project/foam/foam/2.4.0/ThirdParty-2.4.0.tgz?r=http%3A%2F%2Fwww.openfoam.org%2Fdownload%2Fgit.php&ts=1439907495&use_mirror=netcologne
Maybe necessary to rename the downloaded file to ThirdParty-2.4.0.tgz
tar xzf ThirdParty-2.4.0.tgz rm ThirdParty-2.4.0.tgz mv ThirdParty-2.4.0 ThirdParty-2.4.x cp OpenFOAM-2.4.x/etc/bashrc OpenFOAM-2.4.x/etc/bashrcHani cp OpenFOAM-2.4.x/etc/config/settings.sh OpenFOAM-2.4.x/etc/config/settingsHani.sh sed -i s/settings.sh/settingsHani.sh/g OpenFOAM-2.4.x/etc/bashrcHani
Check available versions:
gcc --version #Gives 4.4.7. Required: 4.7+. NOT OK! - using ThirdParty Gcc flex --version #Gives 2.5.35. Seems to have worked ok!
Check out the makeGcc?? files and Gcc in settings.sh to decide which gcc version to use - in this case we need to upgrade through ThirdParty
sed -i s/"foamCompiler=system"/"foamCompiler=ThirdParty"/g OpenFOAM-2.4.x/etc/bashrcHani sed -i s/". \$WM\_PROJECT\_DIR\/etc\/bashrc"/". \$WM\_PROJECT\_DIR\/etc\/bashrcHani"/g OpenFOAM-2.4.x/etc/config/aliases.sh
The following line destroyed the original file for some strange reason (never happened before). Make a back-up copy first!
echo "alias OF24x='export FOAM_INST_DIR=/chalmers/sw/unsup64/OpenFOAM;. \$FOAM_INST_DIR/OpenFOAM-2.4.x/etc/bashrcHani'" >> addToBashrc
Ignore the warning after the following line is executed (we use system Gcc for compilation of Gcc etc.):
export FOAM_INST_DIR=/chalmers/sw/unsup64/OpenFOAM;. $FOAM_INST_DIR/OpenFOAM-2.4.x/etc/bashrcHani WM_NCOMPPROCS=4
See ThirdParty-2.4.x/makeGcc for which versions of depandent packages you need.
cd ThirdParty-2.4.x wget http://ftp.gnu.org/gnu/gmp/gmp-5.1.2.tar.bz2 bunzip2 gmp-5.1.2.tar.bz2 tar xf gmp-5.1.2.tar rm gmp-5.1.2.tar wget http://www.mpfr.org/mpfr-3.1.2/mpfr-3.1.2.tar.gz tar xzf mpfr-3.1.2.tar.gz rm mpfr-3.1.2.tar.gz wget http://www.multiprecision.org/mpc/download/mpc-1.0.1.tar.gz tar xzf mpc-1.0.1.tar.gz rm mpc-1.0.1.tar.gz wget https://ftp.gnu.org/gnu/gcc/gcc-4.8.2/gcc-4.8.2.tar.gz tar xzf gcc-4.8.2.tar.gz rm gcc-4.8.2.tar.gz ./makeGcc gcc-4.8.2 >& log_makeGcc rm log_makeGcc #After looking at it, making sure that it went through
Check that gcc is installed:
which gcc
Returns /chalmers/sw/unsup64/OpenFOAM/ThirdParty-2.4.x/platforms/linux64/gcc-4.8.2/bin/gcc (For the following, see http://openfoamwiki.net/index.php/Installation/Linux/OpenFOAM-2.4.0/CentOS_SL_RHEL#CentOS_6.5):
sed -i s/"CGAL.sh"/"CGALHani.sh"/g ../OpenFOAM-2.4.x/etc/bashrcHani cp ../OpenFOAM-2.4.x/etc/config/CGAL.sh ../OpenFOAM-2.4.x/etc/config/CGALHani.sh wget "https://raw.github.com/wyldckat/scripts4OpenFOAM3rdParty/master/getBoost" chmod +x getBoost ./getBoost sed -i -e 's=boost-system=boost_1_54_0=' ../OpenFOAM-2.4.x/etc/config/CGALHani.sh ./makeCGAL gmp-5.1.2 mpfr-3.1.2
ParaView 4.1.0 needs at least Qt 4.7. Therefore, we'll need to do a custom build of Qt 4.8.4
wget "https://raw.github.com/wyldckat/scripts4OpenFOAM3rdParty/master/getQt" chmod +x getQt sed -i -e 's=4\.6=4.8=' -e 's=4\.8\.4=4.8.6=' -e 's=/\$major/\$tarFile=/$major/$version/$tarFile=' getQt ./getQt ./makeQt qt-4.8.6 > log_makeQt 2>&1 rm log_makeQt #After looking at it, making sure that it went through
CentOS 6.5 comes with CMake 2.6.4, but ParaView 4.1.0 needs a one of the versions from the more recent CMake 2.8.x series and CGAL also needs CMake.
cd $WM_THIRD_PARTY_DIR wget "https://raw.github.com/wyldckat/scripts4OpenFOAM3rdParty/master/getCmake" sed -i -e 's=version}\.3=version}.12.1=' getCmake chmod +x getCmake ./getCmake ./makeCmake wmSET $FOAM_SETTINGS
Close the terminal window and open a new one, then type
OF24x . $FOAM_INST_DIR/OpenFOAM-2.4.x/etc/bashrcHani WM_NCOMPPROCS=4
Check that the correct gcc is used (the one just installed):
which gcc foam ./Allwmake >& log& ./Allwmake >& log&
SOME PROBLEMS RELATED TO MPI, SEE END OF SECTION!!!
rm log
ParaFoam:
cd $WM_THIRD_PARTY_DIR ./makeParaView4 -qmake $WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER/qt-4.8.6/bin/qmake PARAVIEW_INSTALL_THIRD_PARTY_LIBRARIES:BOOL=ON cd $FOAM_UTILITIES/postProcessing/graphics/PV4Readers wmSET $FOAM_SETTINGS ./Allwclean ./Allwmake
foam cd .. chmod -R go+Xr ThirdParty-2.4.x OpenFOAM-2.4.x
Compile Debug version, without recompiling ThirdParty:
cd /chalmers/sw/unsup64/OpenFOAM echo "alias OF24xDebug='export FOAM_INST_DIR=/chalmers/sw/unsup64/OpenFOAM;. \$FOAM_INST_DIR/OpenFOAM-2.4.x/etc/bashrcHani WM_COMPILE_OPTION=Debug'" >> addToBashrc
Open new terminal window
OF24xDebug cd $WM_THIRD_PARTY_DIR/platforms ln -s linux64GccDPOpt linux64GccDPDebug
Open a new terminal:
OF24xDebug foam ./Allwmake >& log& ./Allwmake >& log& cd $FOAM_UTILITIES/postProcessing/graphics/PV4Readers #Skip wmSET since it will not set Debug, see alias wmSET ./Allwclean ./Allwmake
foam cd .. chmod -R go+Xr OpenFOAM-2.4.x;echo "done: chmod -R go+Xr OpenFOAM-2.4.x"; chmod -R go+Xr ThirdParty-2.4.x; echo "done: chmod -R go+Xr ThirdParty-2.4.x" >& log_chmod&
The problem with MPI was solved after the above using:
OF24x foam cp -r ../ThirdParty-2.3.x/openmpi-1.6.5 ../ThirdParty-2.4.x sed -i s/"openmpi-1.8.5"/"openmpi-1.6.5"/g etc/config/settingsHani.sh sed -i s/SYSTEMOPENMPI/OPENMPI/g etc/bashrcHani
New terminal window
OF24x foam ./Allwmake >& log& ./Allwmake >& log&
New terminal window
OF24xDebug foam ./Allwmake >& log& ./Allwmake >& log& cd .. chmod -R go+Xr OpenFOAM-2.4.x;echo "done: chmod -R go+Xr OpenFOAM-2.4.x"; chmod -R go+Xr ThirdParty-2.4.x; echo "done: chmod -R go+Xr ThirdParty-2.4.x" >& log_chmod&
I later also added cfMesh-1.1.1. See separate section above.
OpenFOAM-2.3 in StuDAT (non-root), Redhat Enterprise 6.2 , 2014-08-25
NOTE:
This installation did not complete zoltanRenumber since a fix for that is not possible without root access:
The Boost C++ library is installed automatically for GCAL under the name libboost_thread-mt.so, but it requires the library to be named libboost_thread.so. To work around this inconsistency, a soft link should be created as follows:
ln -s /usr/lib64/libboost_thread-mt.so /usr/lib64/libboost_thread.so
INSTALLATION NOTES START HERE:
git clone git://github.com/OpenFOAM/OpenFOAM-2.3.x.git wget http://downloads.sourceforge.net/project/foam/foam/2.3.0/ThirdParty-2.3.0.tgz?r=http%3A%2F%2Fwww.openfoam.org%2Fdownload%2Fgit.php&ts=1408790768&use_mirror=master
Mayby necessary to rename the downloaded file to ThirdParty-2.3.0.tgz
tar xzf ThirdParty-2.3.0.tgz rm ThirdParty-2.3.0.tgz mv ThirdParty-2.3.0 ThirdParty-2.3.x cp OpenFOAM-2.3.x/etc/bashrc OpenFOAM-2.3.x/etc/bashrcHani cp OpenFOAM-2.3.x/etc/config/settings.sh OpenFOAM-2.3.x/etc/config/settingsHani.sh sed -i s/settings.sh/settingsHani.sh/g OpenFOAM-2.3.x/etc/bashrcHani
Check out the makeGcc?? files and Gcc in settings.sh to decide which gcc version to use
sed -i s/"foamCompiler=system"/"foamCompiler=ThirdParty"/g OpenFOAM-2.3.x/etc/bashrcHani sed -i s/". \$WM\_PROJECT\_DIR\/etc\/bashrc"/". \$WM\_PROJECT\_DIR\/etc\/bashrcHani"/g OpenFOAM-2.3.x/etc/config/aliases.sh echo "alias OF23x='export FOAM_INST_DIR=/chalmers/sw/unsup64/OpenFOAM;. \$FOAM_INST_DIR/OpenFOAM-2.3.x/etc/bashrcHani'" >> addToBashrc
Check versions:
gcc --version #Gives 4.4.7. Required: 4.7+. NOT OK! - using ThirdParty Gcc flex --version #Gives 2.5.35. Seems to have worked ok! export FOAM_INST_DIR=/chalmers/sw/unsup64/OpenFOAM;. $FOAM_INST_DIR/OpenFOAM-2.3.x/etc/bashrcHani WM_NCOMPPROCS=4 cd ThirdParty-2.3.x wget http://ftp.gnu.org/gnu/gmp/gmp-5.1.2.tar.bz2 bunzip2 gmp-5.1.2.tar.bz2 tar xf gmp-5.1.2.tar rm gmp-5.1.2.tar wget http://www.mpfr.org/mpfr-3.1.2/mpfr-3.1.2.tar.gz tar xzf mpfr-3.1.2.tar.gz rm mpfr-3.1.2.tar.gz wget http://www.multiprecision.org/mpc/download/mpc-1.0.1.tar.gz tar xzf mpc-1.0.1.tar.gz rm mpc-1.0.1.tar.gz wget https://ftp.gnu.org/gnu/gcc/gcc-4.8.2/gcc-4.8.2.tar.gz tar xzf gcc-4.8.2.tar.gz rm gcc-4.8.2.tar.gz ./makeGcc48
(see http://openfoamwiki.net/index.php/Installation/Linux/OpenFOAM-2.3.0/CentOS_SL_RHEL#CentOS_6.5):
sed -i s/"CGAL.sh"/"CGALHani.sh"/g ../OpenFOAM-2.3.x/etc/bashrcHani cp OpenFOAM-2.3.x/etc/config/CGAL.sh ../OpenFOAM-2.3.x/etc/config/CGALHani.sh wget "https://raw.github.com/wyldckat/scripts4OpenFOAM3rdParty/master/getBoost" chmod +x getBoost ./getBoost sed -i -e 's=boost-system=boost_1_54_0=' ../OpenFOAM-2.3.x/etc/config/CGALHani.sh ./makeCGAL gmp-5.1.2 mpfr-3.1.2
ParaView 4.1.0 needs at least Qt 4.7. Therefore, we'll need to do a custom build of Qt 4.8.4
wget "https://raw.github.com/wyldckat/scripts4OpenFOAM3rdParty/master/getQt" chmod +x getQt sed -i -e 's=4\.6=4.8=' -e 's=/\$major/\$tarFile=/$major/$version/$tarFile=' getQt ./getQt ./makeQt qt-4.8.4 >& log_makeQt & (NOTE: Have to say "y")
CentOS 6.5 comes with CMake 2.6.4, but ParaView 4.1.0 needs a one of the versions from the more recent CMake 2.8.x series and CGAL also needs CMake.
cd $WM_THIRD_PARTY_DIR wget "https://raw.github.com/wyldckat/scripts4OpenFOAM3rdParty/master/getCmake" sed -i -e 's=version}\.3=version}.12.1=' getCmake chmod +x getCmake ./getCmake ./makeCmake wmSET $FOAM_SETTINGS
. $FOAM_INST_DIR/OpenFOAM-2.3.x/etc/bashrcHani
Check that the correct gcc is used (the one just installed):
which gcc foam ./Allwmake ./Allwmake
ParaFoam:
cd $WM_THIRD_PARTY_DIR ./makeParaView4 -qmake $WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER/qt-4.8.4/bin/qmake PARAVIEW_INSTALL_THIRD_PARTY_LIBRARIES:BOOL=ON cd $FOAM_UTILITIES/postProcessing/graphics/PV4Readers wmSET $FOAM_SETTINGS ./Allwclean ./Allwmake
cd .. chmod -R go+Xr ThirdParty-2.3.x OpenFOAM-2.3.x
Can't run paraFoam remotely:
/nfs4/sol.ita.chalmers.se/ufs/groups/manual03/unsup/unsup64/OpenFOAM/ThirdParty-2.3.x/platforms/linux64Gcc/ParaView-4.1.0/lib/paraview-4.1/paraview: symbol lookup error: /usr/lib64/libGL.so.1: undefined symbol: _XGetRequest
See also:
http://www.cfd-online.com/Forums/blogs/wyldckat/1203-paraview-3-12-0-superbuild-openfoam.html
Compile Debug version, without recompiling ThirdParty:
cd /chalmers/sw/unsup64/OpenFOAM echo "alias OF23xDebug='export FOAM_INST_DIR=/chalmers/sw/unsup64/OpenFOAM;. \$FOAM_INST_DIR/OpenFOAM-2.3.x/etc/bashrcHani WM_COMPILE_OPTION=Debug'" >> addToBashrc cd $WM_THIRD_PARTY_DIR/platforms ln -s linux64GccDPOpt linux64GccDPDebug
Open a new terminal:
OF23xDebug foam ./Allwmake cd $FOAM_UTILITIES/postProcessing/graphics/PV3Readers #Skip wmSET since it will not set Debug, see alias wmSET ./Allwclean ./Allwmake
foam cd .. chmod -R go+Xr OpenFOAM-2.3.x;echo "done: chmod -R go+Xr OpenFOAM-2.3.x"; chmod -R go+Xr ThirdParty-2.3.x; echo "done: chmod -R go+Xr ThirdParty-2.3.x" >& log_chmod&
