Central ALICE software installations at GSI

Introduction

The HPC Computing farm for analysis is Prometheus. Details on the batch farms can be found here (in IT wiki).

The login/submit nodes can be reached via:
  • pro-ib.hpc.gsi.delxsub0[1-4], connected with InfiniBand to Prometheus. NO, connection to the outside !!
  • pro.hpc.gsi.delxsub[05-..], connection to outside possible ( eg. git, svn, etc.)

Be careful, connection to the outside (eg. svn co) can only be reached from the pro.hpc machines ( lxsub05 upwards )

File systems and storage

The computing cluster is accompanied by a mass storage system HERA, based on the lustre file system. This has to be taken as "scratch space" and is not back up'ed. The ALICE branch can be accessed via

/hera/alice/

Note that your home directory /u/<user-name> is not available on the compute cluster. It might only be accessed on the submission nodes.

The application software of the central installation is distributed via CVMFS (http://wiki.gsi.de/Linux/CvmFs) , which creates local copies of the files being used and caches them.

The ALICE branch can be accessed via

/cvmfs/alice.gsi.de/

Software organization

The application software is organized in modules:
  • modules (http://sourceforge.net/projects/modules/) provides an easy access to the different versions
  • One main module ALICE has to be loaded and loads all subsequent modules, needed in order to setup the required environment.

As description of how to setup and use the modules environment can be found below. Every ALICE module loads several sub modules such as :
  • BASE (sets basic environmental variables)
  • ALIEN
  • FASTJET
  • ROOT
  • GEANT3
  • ALIROOT
  • Pythia8

To load an ALICE environment, one just has to load the top level ALICE module : eg

jthaeder@lxi059:~$ jthaeder@lxsub05:~$ ml ALICE/vAN-20140709-prod
ALICE version loaded: (GLOBAL) vAN-20140709-prod
AliRoot version: vAN-20140709
Alien version: v2-19.190
Root version: v5-34-08
Geant3 version: v1-15a
FastJet version: 3.0.3
Pythia8 version: 8175

Attention: one has to load the main module ALICE, it will not work if you load for example only the ALIROOT module

Possible ALICE versions

The versions of the ALICE module is mainly defined by the ALIROOT module below :
  • ALIROOT master ⇒ eg.: ALICE/HEAD_2012-02-08-prod
  • ALIROOT Daily Analysis Tag ⇒ eg.: ALICE/vAN-20140709-prod
  • ALIROOT Branch ⇒ eg. ALICE/v5-02-Branch_2012-02-03-prod
  • ALIROOT Release ⇒ eg. ALICE/v5-01-Rev-21-prod

The modules environment

Setup modules for user

Every user as to adjust his/her environment in order to use modules

Edit the $HOME/.bashrc file and add the following lines :

#####################################################
declare -f module> /dev/null;
hasMODULE=$?

if [ ${hasMODULE} -eq 0 ] ; then

if [ -d /cvmfs/alice.gsi.de/modules ] ; then
module use /cvmfs/alice.gsi.de/modules
else
echo "ERROR : No modules can be found !!!"
fi
fi
#####################################################

Usage of modules for user

The module command gives access to several options. Executing it without arguments will show a list of options. The most common ones are mentioned here :

module avail

Shows the available software modules, eg. module avail

module load <moduleName>/<versionNo>

Loads a software module, eg. module load ALICE/vAN-20140709-prod

module unload <moduleName>

Unloads a software module, eg. module unload ALICE

module list

Shows the loaded software modules, eg. module list

jthaeder@lxsub05:~$ module list
Currently Loaded Modulefiles:
1) BASE/base 3) FASTJET/3.0.3 5) ALIROOT/vAN-20140709 7) GEANT3/v1-15a_v5-34-08
2) ALIEN/v2-19.190 4) PYTHIA8/8175 6) ROOT/v5-34-08 8) ALICE/vAN-20140709-prod

Hints

It is useful to define aliases like :

# module aliases
alias ma="module avail"
alias mu="module unload"
alias ml="module load"
alias mls="module list"
alias mrm="module unload ALICE"
alias mh="module avail -l 2>&1 | grep ALICE"

Use your private ALIROOT installation on the batch farm

It is possible to use a private aliroot installation. For this follow these steps belpw

Prepare the environment

Create a folder with your private ALIROOT installation on /hera/alice

/hera/alice/<user-name>/<software folder>/aliroot

and for your private module files

/hera/alice/<user-name>/<software folder>/modules/ALIROOT

To create a private module file, copy one of the central installation and modify it accordingly eg.

cp /cvmfs/alice.gsi.de/modules/ALIROOT/<aliroot-module-name>  /hera/alice/<user-name>/<software folder>/modules/ALIROOT/<private-aliroot-name>

Change the lines inside the copied module file:

setenv       ALIROOT_RELEASE    <aliroot-version>

to

setenv       ALIROOT_RELEASE    <private-aliroot-version>
setenv       ALIROOT_BASEDIR    /hera/alice/<user-name>/<software folder>/aliroot

in such a way the ALIROOT will be installed in

/hera/alice/<user-name>/<software folder>/aliroot/AliRoot_<private-aliroot-version>

and the module filename matches the <private-aliroot-version>

ALIROOT/<private-aliroot-version>

Enable the usage of private modules and load them

Edit the $HOME/.bashrc file and add the following line below the existing module use command
module use --append /hera/alice/<user-name>/<software folder>/modules/

Then load the general ALICE module and unload the ALIROOT one

module load ALICE/<version>
module unload ALIROOT

Now you can load your private ALIROOT module
module load ALIROOT/<private-aliroot-version>

Installation of private ALIROOT version

Then ALIROOT can be checked out and installed eg. for master. It is assumed that the proper modules are loaded.
cd $ALIROOT_BASEDIR
# for users with no commit rights
git clone http://git.cern.ch/pub/AliRoot AliRoot_<private-aliroot-version>
# more on http://aliweb.cern.ch/Offline/AliRoot/Installation.html
# and http://aliceinfo.cern.ch/Offline/node/2912#ParallelGit

If a specific branch/tag (eg. a daily analysis tag) should be checked out used the following line in addition
git checkout <aliroot-version> 

Then ALIROOT can be compiled with -g -O2 options via:

mkdir -p $ALICE_ROOT/build
cd $ALICE_ROOT/build
cmake -DCMAKE_BUILD_TYPE=RelRelWithDebInfo $ALICE_ROOT
make install -j 16

Special Issues

ALIEN USER

This setup assumes that your GSI username is also your alien username. If this is NOT the case, then you have to set the ALIEN_USER environment variable in your $HOME/.bashrc. eg.:
export  ALIEN_USER=myAlienUserName

Steps to initialize AliEn from central installation at GSI:

#1.) load BASE module (defines install dir for software)
module load BASE/base
#2.) load ALIEN module:
module load ALIEN/current
#3.) check AliEn path
echo $ALIEN_ROOT
#4.) token init 
alien-token-init
#5 source alien environment
/tmp/gclient_env_$UID

Additional information:
  • Code described above should work after copy/paste to your terminal.
  • Please note worker node have to have direct access to the outside word.
  • Special setting for pass-word free alien registration is needed

CVMFS installation on Laptop ...

CVMFS - the CERN Virtual Machine FileSystem - can also be setup on your laptop. It can be used to retrieve common files. Even the installations could be used if you use the same OS flavor (Debian at GSI and Scientifc Linux at CERN).

Installation

Details can be found here http://cernvm.cern.ch/portal/filesystem/downloads#cvmfs:
  1. First you need to install CernVM-FS Public Key
  2. Then install the appropriate package for your distribution CernVM-FS downloads

Setup

Documentation can be found here. Please use this one if this basic/simple setup scenario is not sufficient for your needs.

Run the setup script:
sudo cvmfs_config setup
Create a default local config file and open it for editing
sudo touch /etc/cvmfs/default.local

Add the following lines
CVMFS_REPOSITORIES=alice-ocdb.cern.ch,alice.cern.ch
CVMFS_HTTP_PROXY=DIRECT

Create a special file for the location of the ALICE OCDB:
sudo touch /etc/cvmfs/config.d/alice-ocdb.cern.ch.local

and insert
CVMFS_SERVER_URL="http://cvmfs-stratum-one.cern.ch/opt/@org@"

Check the setup

A reboot of your laptop might help at that point as well ...

Check if CVMFS mounts the specified repositories by

cvmfs_config probe

Now, if all works fine, you should be able to find the ALICE OCDB and the ALICE software

/cvmfs/alice.cern.ch
/cvmfs/alice.cern.ch/calibration/
/cvmfs/alice-ocdb.cern.ch/calibration/
Topic revision: r34 - 16 Aug 2014, JochenThaeder