Installation on Karolina (IT4I)

The Karolina cluster is located at IT4I, Technical University of Ostrava.

Introduction

If you are new to this system, please see the following resources:

• IT4I user guide

• Batch system: PBS

• Filesystems:

  • $HOME: per-user directory, use only for inputs, source and scripts; backed up (25GB default quota, 5k entries)

  • /scatch/: production directory; very fast for parallel jobs (10TB, 10M entries per user)

  • /mnt/: project file system (20TB, 5M entries per project)

For convenience, you can add the following variables to your .bashrc:

export SCRDIR="/scratch/project/dd-23-83/${USER}"
export WRKDIR="/mnt/proj2/dd-23-83/${USER}"

where dd-23-83 is the project identifier, which can be different in your case.

Preparation

Use the following commands to download the fbpic source code:

# optionally, remove any previous installs if necessary
rm -rf $HOME/src/fbpic
rm -rf $HOME/sw/karolina/gpu/venvs/fbpic

git clone https://github.com/berceanu/fbpic.git $HOME/src/fbpic

On Karolina, we recommend running on the accelerator nodes with fast A100 GPUs.

We use system software modules, add environment hints and further dependencies via the file $HOME/karolina_fbpic.profile. Create it now:

cp $HOME/src/fbpic/Tools/machines/karolina-it4i/karolina_fbpic.profile.example $HOME/karolina_fbpic.profile

Script Details

$HOME/src/fbpic/Tools/machines/karolina-it4i/karolina_fbpic.profile.example.

# please set your project account
export proj="DD-23-83"  # change me!

# remembers the location of this script
export MY_PROFILE=$(cd $(dirname $BASH_SOURCE) && pwd)"/"$(basename $BASH_SOURCE)
if [ -z ${proj-} ]; then echo "WARNING: The 'proj' variable is not yet set in your $MY_PROFILE file! Please edit its line 2 to continue!"; return; fi

# required dependencies
module purge
ml CMake/3.23.1-GCCcore-11.3.0
ml Boost/1.79.0-GCC-11.3.0
ml OpenBLAS/0.3.20-GCC-11.3.0
ml Python/3.10.4-GCCcore-11.3.0-bare
ml OpenMPI/4.1.4-GCC-11.3.0-CUDA-11.7.0

ml git/2.36.0-GCCcore-11.3.0-nodocs

# TODO: remove
#ml CuPy/8.5.0-fosscuda-2020b 


# Python virtual env
if [ -d "${HOME}/sw/karolina/gpu/venvs/fbpic" ]
then
  source ${HOME}/sw/karolina/gpu/venvs/fbpic/bin/activate
fi

# compiler environment hints
export CC=$(which gcc)
export CXX=$(which g++)
export FC=$(which gfortran)
export CUDACXX=$(which nvcc)
export CUDAHOSTCXX=${CXX}

Edit the 2nd line of this script, which sets the export proj="" variable. For example, if you are member of the project DD-23-83, then run vi $HOME/karolina_fbpic.profile. Enter the edit mode by typing i and edit line 2 to read:

export proj="DD-23-83"

Exit the vi editor with Esc and then type :wq (write & quit).

Important

Now, and as the first step on future logins to Karolina, activate these environment settings:

source $HOME/karolina_fbpic.profile

You can also add the line above to your $HOME/.bashrc file so that it is loaded on each login.

Finally, since Karolina does not yet provide software modules for some of our dependencies, install them once, and activate the newly created Python virtual environment. Further environment activations will be done automatically from inside karolina_fbpic.profile.

bash $HOME/src/fbpic/Tools/machines/karolina-it4i/install_dependencies.sh
source $HOME/sw/karolina/gpu/venvs/fbpic/bin/activate

Script Details

$HOME/src/fbpic/Tools/machines/karolina-it4i/install_dependencies.sh.

#!/bin/bash

# Exit on first error encountered #############################################
#
set -eu -o pipefail


# Check: ######################################################################
#
#   Was karolina_fbpic.profile sourced and configured correctly?
if [ -z ${proj-} ]; then echo "WARNING: The 'proj' variable is not yet set in your karolina_fbpic.profile file! Please edit its line 2 to continue!"; exit 1; fi


# Remove old dependencies #####################################################
#
SW_DIR="${HOME}/sw/karolina/gpu"
rm -rf ${SW_DIR}
mkdir -p ${SW_DIR}

# remove common user mistakes in python, located in .local instead of a venv
python3 -m pip uninstall -qq -y fbpic
python3 -m pip uninstall -qqq -y mpi4py 2>/dev/null || true


# Python ######################################################################
#
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade virtualenv
python3 -m pip cache purge
rm -rf ${SW_DIR}/venvs/fbpic
python3 -m venv ${SW_DIR}/venvs/fbpic
source ${SW_DIR}/venvs/fbpic/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade wheel
python3 -m pip install --upgrade black 
python3 -m pip install --upgrade numpy
python3 -m pip install --upgrade pandas
python3 -m pip install --upgrade scipy
python3 -m pip install --upgrade matplotlib
python3 -m pip install --upgrade openpmd-api
python3 -m pip install --upgrade openpmd-viewer
python3 -m pip install --upgrade mpi4py --no-cache-dir --no-build-isolation --no-binary mpi4py
# TODO: remove commented lines
#MPICC="cc -shared -target-accel=nvidia80" python3 -m pip install --upgrade --force --no-cache-dir --no-build-isolation --no-binary=mpi4py mpi4py
#python3 -m pip install --upgrade llvmlite tbb
# TODO: remove synchrad dependencies
python3 -m pip install --upgrade mako
python3 -m pip install --upgrade pyopencl
python3 -m pip install --upgrade numba
python3 -m pip install --upgrade python-dateutil
python3 -m pip install --upgrade h5py
python3 -m pip install --upgrade cupy-cuda117

Finally, install fbpic itself. This will install the package in “editable” mode, meaning any changes you make to the local source code will immediately be reflected in the installed package:

cd $HOME/src/fbpic
python3 -m pip install -e .

Now, you can submit Karolina compute jobs for fbpic Python scripts.

Update fbpic & Dependencies

If you already installed fbpic in the past and want to update it, start by getting the latest source code:

cd $HOME/src/fbpic

# read the output of this command - does it look ok?
git status

# get the latest fbpic source code
git fetch
git pull

# read the output of these commands - do they look ok?
git status
git log # press q to exit

And, if needed,

• update the karolina_fbpic.profile file,

• log out and into the system, activate the now updated environment profile as usual,

• execute the dependency install script.

As a last step, reinstall fbpic:

python3 -m pip install -e .

This is only needed in case the dependencies have changed, otherwise the “editable” install should automatically reflect the latest changes pulled from git, without needing to reinstall the package.

Running

The batch script below can be used to run a fbpic simulation on TODO GPU nodes (change #PBS -l select= accordingly) on the supercomputer Karolina at IT4I. This partition has up to 72 nodes. Every node has 8x A100 (40GB) GPUs and 2x AMD EPYC 7763, 64-core, 2.45 GHz processors.

Replace descriptions between chevrons <> by relevant values, for instance <proj> could be DD-23-83. Note that we run one MPI rank per GPU.

Script Details

$HOME/src/fbpic/Tools/machines/karolina-it4i/karolina_gpu.qsub.

#!/bin/bash -l

#PBS -q qgpu
#PBS -N fbpic
#PBS -l select=1:ncpus=16:ngpus=1:mpiprocs=1:ompthreads=16,walltime=00:10:00
#PBS -A DD-23-83

cd ${PBS_O_WORKDIR}

# Python interpreter & input script here
INTERPRETER=python3
SCRIPT=${PBS_O_WORKDIR}/fbpic_script.py

# OpenMP threads
export OMP_NUM_THREADS=16

# run
mpirun -n 1 bash -c "
    export CUDA_VISIBLE_DEVICES=\${OMPI_COMM_WORLD_LOCAL_RANK};
    ${INTERPRETER} ${SCRIPT}" \
  > output_1_gpu.txt

To run a simulation, copy the lines above to a file karolina_gpu.qsub

mkdir -p $SCRDIR/runs/fbpic
cp $HOME/src/fbpic/Tools/machines/karolina-it4i/karolina_gpu.qsub $SCRDIR/runs/fbpic

and run

cd $SCRDIR/runs/fbpic
qsub karolina_gpu.qsub

to submit the job.