OpenFOAM at Ada

Log in to Ada

First of all, read the information at the Ada homepage .

Log in to Ada:
ssh ada.c3se.chalmers.se

Basic set-up

In order to use the pre-installed version 1.4, do the following:
Right now there are problems with the bashrc files in the installation, so download OF1.4_Ada_bashrc to your home diretory at Ada.

Add the necessary MPICH module to your $HOME/.bashrc file at Ada by typing the following at command-line:
module initadd mpich/1.2.7-PIC/gcc/4.1

Add the following at the end of your $HOME/.bashrc file at Ada:
-------------------------------------------------------------------
# User specific aliases, functions and default modules below
export WM_64=on #Use the 64 bit version (the only pre-installed)
export FOAM_SILENT=1 #No echo in bashrc
export FOAM_SIGFPE=1 #Halts on Nan/Floating Point Exceptions
#export WM_NCOMPPROCS=8 #Compile in parallel
. $HOME/OF1.4_Ada_bashrc
#. /c3se/apps/OpenFOAM/OpenFOAM-1.4/.OpenFOAM-1.4/bashrc # This is the correct one, when it works...
-------------------------------------------------------------------

If you want to use your own installation, just modify the final sourcing to the corresponding bashrc file of your own installation.

Log out and log in to Ada again (ssh ada.c3se.chalmers.se), so that the files have been sourced correctly.

Test

Now it's time to test the installation at the login node. Note that this is only for testing purposes, you should not run at the login node later on.

Make sure that you have the following directory structure:
$HOME/OpenFOAM/$LOGNAME-1.4/run

Copy the pitzDaily case from the tutorials directory to your run directory:
cp -r /c3se/apps/OpenFOAM/OpenFOAM-1.4/tutorials/simpleFoam/pitzDaily $HOME/OpenFOAM/$LOGNAME-1.4/run

Move to the run directory:
run

Generate the grid:
blockMesh . pitzDaily

Run simpleFoam on the pitzDaily case:
simpleFoam . pitzDaily

Your computation should now run. Otherwise you should go through this set-up again before you consult me.

How to submit OpenFOAM jobs at Ada

Sequential jobs

A simple submit script that submits the above case to Ada:

-----------------------------------------------------------------------
# Arguments to qsub can be submitted via the script as well by starting
# the line with #PBS
#
# Choose the que:
#PBS -q ada
#
# Specify time for job (here 1h 10 min 5 sec)
#PBS -l walltime=1:10:05
#
# Request 1 processor (node)
#PBS -l nodes=1:ppn=1
#
# Set the name of the job
#PBS -N pitzDailySequential
#
# End of arguments to qsub

# Go to work submission directory (Don't remove this line)
# This is where output from standard output will appear.
# Output from OpenFOAM will appear in the OpenFOAM directories.
cd $PBS_O_WORKDIR

# Time the program
time simpleFoam $FOAM_RUN pitzDaily > $FOAM_RUN/pitzDaily/log

#End of script (make sure line before this gets run)
-----------------------------------------------------------------------

If you named the above script as 'submitPitzDailySequential', submit the job as:
qsub submitPitzDailySequential

Check the status of your jobs with:
qstat

You should get one 'pitzDailySequential.o*', and one 'pitzDailySequential.e*' file in the directory you submitted from. 'pitzDailySequential.o*' contains the standard output. 'pitzDailySequential.e*' contains output from the time command, or possible error messages. The results will be in the pitzDaily time directories as usual.

Parallel jobs

This does not work with the pre-installed version at the moment. It has not yet been compiled with MPICH at Ada, and I don't have the right to do it. However, if you have your own installation, the following information might make sense. Note that your own installation must then be re-compiled with MPICH (see below).

For parallel jobs you need to decompose your case before you run it. Download decomposeParDict to the 'system' directory in your pitzDaily case. This decomposeParDict will decompose your case into 8 parts using the Metis method.

Here is a script that decomposes your case, runs it and reconstructs it:
-----------------------------------------------------------------------
# Arguments to qsub can be submitted via the script as well by starting
# the line with #PBS
#
# Choose the que:
#PBS -q ada
#
# Specify time for job (here 1h 10 min 5 sec)
#PBS -l walltime=1:10:05
#
# Request 8 processors (2 nodes and 4 processors per node)
#PBS -l nodes=2:ppn=4
#
# Set the name of the job
#PBS -N pitzDailyParallel
#
# End of arguments to qsub
#
export P4_GLOBMEMSIZE=10240000 #This seems to be case-dependent, and this
#value works with 100 000 cells per process
#It may be possible to recompile MPICH
#to avoid setting this value
#
# Go to work submission directory (Don't remove this line)
# This is where output from standard output will appear.
# Output from OpenFOAM will appear in the OpenFOAM directories.
cd $PBS_O_WORKDIR

# Decompose the case for parallel computations
decomposePar $FOAM_RUN pitzDaily

# Time the program
time mpiexec simpleFoam $FOAM_RUN pitzDaily -parallel > $FOAM_RUN/pitzDaily/log

# Reconstruct the case after parallel computations
reconstructPar $FOAM_RUN pitzDaily

#End of script (make sure line before this gets run)
-----------------------------------------------------------------------

If you named the above script as 'submitPitzDailyParallel', submit the job as:
qsub submitPitzDailyParallel

Check the status of your jobs with:
qstat

You should get one 'pitzDailyParallel.o*', and one 'pitzDailyParallel.e*' file in the directory you submitted from. 'pitzDailyParallel.o*' contains the standard output. 'pitzDailyParallel.e*' contains output from the time command, or possible error messages. The results will be in the pitzDaily time directories as usual.

How to compile your own applications

Although there is a central installation of OpenFOAM at Ada you are still able to compile your own libraries and applications. First make sure that you have the following directory structure:
~/OpenFOAM/$LOGNAME-1.4
Put all your source files in the $LOGNAME-1.4 directory. Use a nice structure for this. Have a look at how it is organized in /c3se/apps/OPenFOAM/OpenFOAM-1.4-2/applications.

When you compile your applications you must make sure that you use $FOAM_USER_APPBIN in your Make/files file, so that you executable will be written to your own directory. If you keep $FOAM_APPBIN wmake will try to write to the root directories, and the compilation will fail.

Note that you can submit compilation jobs by making a submit script as described above, but with the wmake command. You must then submit your job from the directory of your specific directory and make sure that you keep the line with 'cd $PBS_O_WORKDIR' in your submit script.

How to install OpenFOAM-1.4 for MPICH at Ada

This section describes the steps you need to take when you install OpenFOAM-1.4 directly from the official release. Later on there are instructions on how to use Subversion and SourceForge to keep you installation updated with the most recent bug fixes. You will also learn how to get the most recent extended version of OpenFOAM.

Note that you should not install the official release yourself at Ada. An updated bug-fixed version is kept up to date for you in the central installation.

Download

Download the Linux 64 bit version according to the instructions at www.openfoam.org. Include the binaries for double precision. Double precision is default and sufficient. ParaView and Java are not needed if you will not use FoamX or paraFoam.

Modify for MPICH

Open file OpenFOAM/OpenFOAM-1.4/.OpenFOAM-1.4/bashrc
Change lines 127-128 to:
#export WM_MPLIB=LAM
export WM_MPLIB=MPICH
Open file OpenFOAM/OpenFOAM-1.4/.bashrc
Change lines 214-215 to:
export MPICH_VERSION=1.2.7
export MPICH_PATH=/c3se/apps/mpich/1.2.7__gcc-4.1.0-PIC

Basic set-up

Make sure that your $HOME/.bashrc file is updated according to the instructions above. Make sure that you now source the bashrc in your own installation. Log out/log in to update everything. You have to do this for the above changes to be effective.

Re-compile for MPICH

Parts of the code needs to be re-compiled for MPICH, do the following:
cd $WM_PROJECT_DIR/src/Pstream/mpi
wmake libso
cd $WM_PROJECT_DIR/applications/utilities/parallelProcessing/decompositionMethods/
./Allwmake
cd $WM_PROJECT_DIR/applications/utilities/parallelProcessing/decomposePar
wmake

It might in fact be a good idea to re-compile all the applications:
cd $WM_PROJECT_DIR/applications
./Allwmake
(Note that there is a bug in wmake in 1.4, so you will get an error message! Find the corrected wmake in the Forum.)

Now you should be able to run the tests above. Good luck!

How to get and install OpenFOAM-1.4_patch at Ada

OpenFOAM-1.4_patch is the unofficial bug-fixed version of the original distribution of OpenFOAM-1.4. It is located at SourceForge, and it is available through svn (Subversion). Here is a short description on how to keep updated with this version of OpenFOAM. Here it is assumed that you will only be checking out files from SourceForge (i.e. not write to it). You can find more information on how to use svc at the SourceForge homepage.

Note that you should not do this at Ada. The central installation will be kept updated for you. If you find out that it is not updated, please tell Håkan to update it.

Check out the most recent patch

Move to your OpenFOAM directory:
cd $HOME/OpenFOAM
Use the svc check-out command (this will take approximately 30min):
svn co https://openfoam-extend.svn.sourceforge.net/svnroot/openfoam-extend/branches/OpenCFD_Release/OpenFOAM-1.4_patch

Note that external packages are not part of the svn, so you will have to get them from elsewhere:
src/lam-7.1.2
src/mico-2.3.12
src/openmpi-1.2b3
src/zlib-1.2.1

There might still be a problem with an STL file. See how to fix it at:
http://openfoam.cfd-online.com/cgi-bin/forum/show.cgi?tpc=126&post=13043#POST13043
There might still be a problem with paths to Allwmake. See the bugs report.

Make sure that cmake is available in your path:
module add cmake

Download paraview-2.4.4.tar.gz from:
http://powerlab.fsb.hr/ped/kturbo/OpenFOAM/release/
into ~/OpenFOAM/linuxSrc

Install Paraview:

(You don't have to do this, but then you have to compile parts 'by hand')
cd ~/OpenFOAM/linuxSrc
tar xvf paraview-2.4.4.tar.gz
mkdir paraview-2.4.4-obj
cd paraview-2.4.4-obj
cmake -i ../paraview-2.4.4

Change options:

enable shared library: YES
prefix: ~/OpenFOAM/linuxSrc/paraview-2.4.4

make
make install

cd ~/OpenFOAM/linuxSrc

mkdir ~/OpenFOAM/linux64/paraview-2.4.4/include
cp ./paraview-2.4.4/VTK/Common/vtk* ~/OpenFOAM/linux64/paraview-2.4.4/include/
cp ./paraview-2.4.4/VTK/Filtering/vtk* ~/OpenFOAM/linux64/paraview-2.4.4/include/
cp ./paraview-2.4.4-obj/VTK/vtk* ~/OpenFOAM/linux64/paraview-2.4.4/include/
mkdir ~/OpenFOAM/linux64/paraview-2.4.4/lib
mkdir ~/OpenFOAM/linux64/paraview-2.4.4/lib/paraview-2.4
cp ./paraview-2.4.4-obj/ParaViewConfig.cmake ~/OpenFOAM/linux64/paraview-2.4.4/lib/paraview-2.4/

mkdir ~/OpenFOAM/linux64/paraview-2.4.4/bin
cp ./paraview-2.4.4-obj/bin/paraview ../paraview-2.4.4/bin
cp ./paraview-2.4.4-obj/bin/pvbatch ../paraview-2.4.4/bin
cp ./paraview-2.4.4-obj/bin/pvclient ../paraview-2.4.4/bin
cp ./paraview-2.4.4-obj/bin/pvdataserver ../paraview-2.4.4/bin
cp ./paraview-2.4.4-obj/bin/pvrenderserver ../paraview-2.4.4/bin
cp ./paraview-2.4.4-obj/bin/pvserver ../paraview-2.4.4/bin
cp ./paraview-2.4.4/lib/paraview-2.4/paraview-real ../linux64/paraview-2.4.4/lib/paraview-2.4/
or...
cp paraview-2.4.4-obj/bin/paraview-real ../linux64/paraview-2.4.4/lib/paraview-2.4/
cp paraview-2.4.4-obj/bin/lib* ../linux64/paraview-2.4.4/lib/paraview-2.4/
cp -r paraview-2.4.4-obj/lib/tcl8.4 ../linux64/paraview-2.4.4/lib/paraview-2.4/
cp -r paraview-2.4.4-obj/lib/tk8.4 ../linux64/paraview-2.4.4/lib/paraview-2.4/
~/OpenFOAM/OpenFOAM-1.4/bin/paraFoam.pvs is also needed, copy it from the original distribution?
(see also Hrv's build instructions)

Now you have the source code for the patched version, and you have to modify the set-up and compile the whole stuff (approx 4h!).

Modify set-up and compile

Keeping the name of the installation as OpenFOAM-1.4_patch and installing in ~/OpenFOAM, do the following:

In ~/OpenFOAM/OpenFOAM-1.4_patch/.OpenFOAM-1.4/bashrc:
Line 52:
export WM_PROJECT_DIR=$WM_PROJECT_INST_DIR/$WM_PROJECT-${WM_PROJECT_VERSION}_patch
Line 127-128:
#export WM_MPLIB=LAM
export WM_MPLIB=MPICH

In ~/OpenFOAM/OpenFOAM-1.4_patch/.bashrc:
Line 107-109: This requires that you have downloaded the gcc binaries from the OpenFOAM homepage. You can change this to a central installation if available.
export MPICH_VERSION=1.2.7
In ~/.bashrc:
Source ~/OpenFOAM/OpenFOAM-1.4_patch/.OpenFOAM-1.4/bashrc

Ada (RedHat) does not have demangling:
In ~/OpenFOAM/OpenFOAM-1.4/src/OpenFOAM/lnInclude/error.C:
Line 37:
#undef HAS_DEMANGLING

Ada (RedHat) has the ssl library in another place than expected by OpenFOAM:
The file is located in /lib/libssl.so.0.9.7a, but FoamX needs it to be located in /usr/lib.
Creating a symbolic link named /usr/lib/libssl.so.0.9.7 to /lib/libssl.so.0.9.7a (you have to be root):
ln -s /lib/libssl.so.0.9.7a /usr/lib/libssl.so.0.9.7

Log out and log in to set all the environment variables correctly.

Compile (4h - get lots of coffee!):
foam
./Allwmake >& Allwmake.log &

Keep updated

Every now and then you should update your installation against SourceForge:
Move to your OpenFOAM-1.4_patch directory:
cd $HOME/OpenFOAM/OpenFOAM-1.4_patch
Use the svc update command (this should be fast):
svn update

If your working copy is up to date you will just get something like:
At revision 7.

If there has been some updates you will see which files have been updated. Then you need to re-compile the updated stuff.

More svn commands

Find more information through Sourceforge, or just use the following:
Make a diff between the svn and the original distribution (note that you should update your working copy first):
svn diff -R .

How to make an OpenFOAM module for Ada (just for system maintainers)

Start with a copy of the .OpenFOAM/bashrc-file. Insert the sourced files. Do all the if-statements by hand and remove everything that is not done at Ada and for the specific installation. Note that the path to cmake must be changed to the central installation. You will end up with something like this file.