ImpactX

ImpactX: an s-based beam dynamics code including space charge effects. This is the next generation of the IMPACT-Z code.

Documentation

In order to learn how to install and run the code, please see the online documentation: https://impactx.readthedocs.io

ImpactX Doxygen: https://impactx.readthedocs.io/en/latest/_static/doxyhtml
AMReX Doxygen: https://amrex-codes.github.io/amrex/doxygen
WarpX Doxygen: https://warpx.readthedocs.io/en/latest/_static/doxyhtml

Contributing

Our workflow is described in CONTRIBUTING.rst.

Developer Environment

Please prepare you local development environment as follows. Pick one of the methods below:

Perlmutter (NERSC)

ssh perlmutter-p1.nersc.gov

Now cd to your ImpactX source directory.

module load cmake/3.22.0
module load cray-hdf5-parallel/1.12.2.1

# CCache for faster rebuilds
export PATH=/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/ccache-4.5.1-ybl7xefvggn6hov4dsdxxnztji74tolj/bin:$PATH

# Python
module load cray-python/3.9.13.1
if [ -d "$HOME/sw/perlmutter/venvs/impactx" ]
then
  source $HOME/sw/perlmutter/venvs/impactx/bin/activate
else
  python3 -m pip install --user --upgrade pip
  python3 -m pip install --user virtualenv
  python3 -m venv $HOME/sw/perlmutter/venvs/impactx
  source $HOME/sw/perlmutter/venvs/impactx/bin/activate

  python3 -m pip install --upgrade pip
  MPICC="cc -target-accel=nvidia80 -shared" python3 -m pip install -U --no-cache-dir -v mpi4py
  python3 -m pip install --upgrade pytest
  python3 -m pip install --upgrade -r requirements.txt
  python3 -m pip install --upgrade -r examples/requirements.txt
fi

# GPU-aware MPI
export MPICH_GPU_SUPPORT_ENABLED=1

# necessary to use CUDA-Aware MPI and run a job
export CRAY_ACCEL_TARGET=nvidia80

# optimize CUDA compilation for A100
export AMREX_CUDA_ARCH=8.0

# compiler environment hints
export CC=cc
export CXX=CC
export FC=ftn
export CUDACXX=$(which nvcc)
export CUDAHOSTCXX=CC

# work on an interactive node
salloc -N 1 --ntasks-per-node=4 -t 1:00:00 -C gpu --gpu-bind=single:1 -c 32 -G 4 --qos=interactive -A m3906_g

# configure
cmake -S . -B build_perlmutter -DImpactX_COMPUTE=CUDA -DImpactX_PYTHON=ON

# compile
cmake --build build_perlmutter -j 64

# test
ctest --test-dir build_perlmutter -E AMReX --output-on-failure

# run
cd build_perlmutter/bin
srun ./impactx ../../examples/fodo/input_fodo.in

Cori KNL (NERSC)

ssh cori.nersc.gov

Now cd to your ImpactX source directory.

module swap craype-haswell craype-mic-knl
module swap PrgEnv-intel PrgEnv-gnu
module load cmake/3.22.1
module load cray-hdf5-parallel/1.10.5.2
module load cray-fftw/3.3.8.10

# Python
module load cray-python/3.9.7.1
if [ -d "$HOME/sw/knl/venvs/impactx" ]
then
  source $HOME/sw/knl/venvs/impactx/bin/activate
else
  python3 -m pip install --user --upgrade pip
  python3 -m pip install --user virtualenv
  python3 -m venv $HOME/sw/knl/venvs/impactx
  source $HOME/sw/knl/venvs/impactx/bin/activate

  python3 -m pip install --upgrade pip
  MPICC="cc -shared" python3 -m pip install -U --no-cache-dir -v mpi4py
  python3 -m pip install --upgrade pytest
  python3 -m pip install --upgrade -r requirements.txt
  python3 -m pip install --upgrade -r examples/requirements.txt
fi

# tune exactly for KNL sub-architecture
export CXXFLAGS="-march=knl"
export CFLAGS="-march=knl"

# configure
cmake -S . -B build_knl

# compile
cmake --build build_knl -j 8

# test
srun -C knl -N 1 -t 30 -q debug ctest --test-dir build_knl --output-on-failure

# run
cd build_knl/bin
srun -C knl -N 1 -t 30 -q debug ./impactx ../../examples/fodo/input_fodo.in

Homebrew (macOS)

brew update
brew install adios2      # for openPMD
brew install ccache
brew install cmake
brew install fftw
brew install git
brew install hdf5-mpi    # for openPMD
brew install libomp      # for OpenMP
brew install pkg-config  # for fftw
brew install open-mpi

Apt (Debian/Ubuntu)

sudo apt update
sudo apt install build-essential ccache cmake g++ git libfftw3-mpi-dev libfftw3-dev libhdf5-openmpi-dev libopenmpi-dev pkg-config python3 python3-matplotlib python3-numpy python3-scipy

Spack (Linux)

spack env create impactx-dev
spack env activate impactx-dev
spack add adios2        # for openPMD
spack add ccache
spack add cmake
spack add fftw
spack add hdf5          # for openPMD
spack add mpi
spack add pkgconfig     # for fftw
spack add python
spack add py-pip
spack add py-setuptools
spack add py-wheel

# OpenMP support on macOS
[[ $OSTYPE == 'darwin'* ]] && spack add llvm-openmp

# optional: Linux only
#spack add cuda

spack install
python3 -m pip install matplotlib numpy openpmd-api pandas pytest scipy

In new terminals, re-activate the environment with spack env activate impactx-dev again.

Conda (Linux/macOS/Windows)

conda create -n impactx-dev -c conda-forge adios2 ccache cmake compilers git hdf5 fftw matplotlib ninja numpy openpmd-api pandas pytest scipy
conda activate impactx-dev

# compile with -DImpactX_MPI=OFF

Get the Source Code

Before you start, you will need a copy of the ImpactX source code:

git clone [email protected]:ECP-WarpX/impactx.git
cd impactx

Compile

# find dependencies & configure
cmake -S . -B build

# compile
cmake --build build -j 4

That's all! ImpactX binaries are now in build/bin/. Most people execute these binaries directly or copy them out.

You can inspect and modify build options after running cmake -S . -B build with either

ccmake build

or by adding arguments with -D<OPTION>=<VALUE> to the first CMake call, e.g.:

cmake -S . -B build -DImpactX_COMPUTE=CUDA -DImpactX_MPI=OFF

Python Compile

# find dependencies & configure
cmake -S . -B build -DImpactX_PYTHON=ON

# compile & install
cmake --build build -j 4 --target pip_install

Run

An executable ImpactX binary with the current compile-time options encoded in its file name will be created in build/bin/.

Additionally, a symbolic link named impactx can be found in that directory, which points to the last built ImpactX executable.

The command-line syntax for this executable is:

Usage: impactx <inputs-file> [some.overwritten.option=value]...

Mandatory arguments (remove the <>):
  inputs-file     the path to an input file; can be relative to the current
                  working directory or absolute.
                  Example: input_fodo.in

Optional arguments (remove the []):
  options         this can overwrite any line in an inputs-file
                  Example: quad1.ds=0.5 sbend1.rc=1.5

Examples:
  In the current working directory, there is a file "input_fodo.in" and the
  "impactx" executable.
  The line to execute would look like this:
    ./impactx input_fodo.in

  In the current working directory, there is a file "input_fodo.in" and the
  executable "impactx" is in a directory that is listed in the "PATH"
  environment variable.
  The line to execute would look like this:
    impactx input_fodo.in

  In the current working directory, there is a file "input_fodo.in" and the
  "impactx" executable. We want to voerwrite the segment length of the beamline
  element "quad1" that is already defined in it. We also want to change the
  radius of curvature of the bending magnet "sbend1" to a different value than
  in the file "input_fodo.in".
  The line to execute would look like this:
    ./impactx input_fodo.in quad1.ds=0.5 sbend1.rc=1.5

Test

In order to run our tests, you need to have a few Python packages installed:

python3 -m pip install --upgrade pip setuptools wheel pytest
python3 -m pip install --upgrade -r examples/requirements.txt

You can run all our tests with:

ctest --test-dir build --output-on-failure

Further options:

help: ctest --test-dir build --help
list all tests: ctest --test-dir build -N
only run tests that have "FODO" in their name: ctest --test-dir build -R FODO

Acknowledgements

This work was supported by the Laboratory Directed Research and Development Program of Lawrence Berkeley National Laboratory under U.S. Department of Energy Contract No. DE-AC02-05CH11231.

ImpactX is supported by the CAMPA collaboration, a project of the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research and Office of High Energy Physics, Scientific Discovery through Advanced Computing (SciDAC) program.

Copyright Notice

Copyright (c) 2022-2023, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved.

If you have questions about your rights to use or distribute this software, please contact Berkeley Lab's Intellectual Property Office at [email protected].

NOTICE. This Software was developed under funding from the U.S. Department of Energy and the U.S. Government consequently retains certain rights. As such, the U.S. Government has been granted for itself and others acting on its behalf a paid-up, nonexclusive, irrevocable, worldwide license in the Software to reproduce, distribute copies to the public, prepare derivative works, and perform publicly and display publicly, and to permit others to do so.

Please see the full license agreement in LICENSE.txt, which is the BSD-3-Clause-LBNL license.

RTSandberg/impactx

README