flow123d / swrap

Complete MPI wrapping for HPC singularity applications.

Flow123d ·

Processes in Fractured Media

Flow123d is a simulator of underground water flow, transport, and mechanical processes in fractured porous media. Novelty of this software is support of computations on complex meshes consisting of simplicial elements of different dimensions. Therefore we can combine continuum models and discrete fracture network models. For more information see the project pages: flow123d.github.io.

Getting started

Please refer to a User Guide and Input Reference manual available at our official website where there is a entire section dedicated to this topic. You can find step-by-step tutorial explaining geometries, yaml input files and more. Below you can see a result from the tutorial problem.

Installation

You have several options when it comes to installation. You can use our prebuilt Docker images or build Flow123d from the source (requires a moderate experience with Linux OS).

For detailed instructions, see the installation guide.

Developers

Build

Two step build:

     host> bin/fterm     # start the docker developing container
container> make all      # produce "build_tree/bin/flow123d"
container> bin/flow123d  # lunch the simulator using a simple wrapper script

That would start the debug docker image and build the debug version of the simulator. For the release (optimized) version run:

bin/fterm rel
make all

For details see installation guide or the manual.

Troubleshooting

• When problem occurs during the compilation process it may be due to a leftover files in a build folder. Cleaning this directory can solve this issue. You can either remove build-<branch> folder (the folder is located one level above repository root) via make clean-all, which removes build folders and also remove any symlinks.
  To clean all the build folders manually run rm -rf ../build-* while in a repository root.
  Running rm -rf can quite easily cause a lot of damage, double check that you're in a correct folder.

• The build tools may fail if the root path contains folders with spaces.

• During an installation under Windows, some scenarios can cause problems. Please refer to an installation guide for a Docker Toolbox. You can also check out Troubleshooting page where the most common error are described and solved.

Building the reference manual

The reference manual can be built by while in docker container

make ref-doc

To copy out reference manual from docker use command docker cp.

Singularity

Singularity is a container system targeting HPC applications. Singularity containers can be created from the docker images and in contrast to the docker one is not allowed to modify system of running containers.

For large images may be necessary set temp directory.

export SINGULARITY_TMPDIR="/some_absolute_path/tmp"

Run simulator in singularity container

singularity exec docker://flow123d/3.1.0 flow123d simulation.yaml

Parallel run, mpiexec out of the image

module add mpich-3.0.2-gcc
mpiexec -host host1,host2 -np 4 singularity exec docker://flow123d/3.1.0 flow123d simulation.yaml

Build

     host> git clone https://github.com/flow123d/flow123d.git # clone flow123d repository
     host> singularity shell -B flow123d/:/flow123d docker://flow123d/flow-dev-gnu-rel:3.1.0 # starts developing container
container> cd /flow123d
container> make all # produce "build_tree/bin/flow123d"
container> bin/flow123d  # lunch the simulator

Docker images

Hosted on dockerhub under organisation flow123d.

production images

tagged by the release version (e.g. 3.1.0)

• flow123-gnu : based on gnu libraries and tools
• flow123-intel : based on intel libraries (intelmpi, mkl, intel compilers)

alfa images

built by CI, tagged by {branch}-{commit}

• ci-gnu
• ci-intel

swrap's Issues

Errors in integration tests

1. Now on claster charon test end with error:
   INFO: Using cached SIF image
   Host key verification failed.
   =>> PBS: job killed: walltime 1241 exceeded limit 1200

recursive jobs

Starting ned jobs running in the same container need to take care of:

• proper bindings, have to filter SINGULARITY_BIND and exclude /tmp and other non-shared directories
  possibly take care of new agent directories etc.
• propper passing of env variables

Reliably remove auxiliary directory

review start fo ssh-agent

Proposed start of ssh-agent for local passphrases in .bashrc:

env=~/.ssh/agent.env

agent_load_env () { test -f "$env" && . "$env" >| /dev/null ; }

agent_start () {
    (umask 077; ssh-agent >| "$env")
    . "$env" >| /dev/null ; }

• study meaning of agent.env
• review initialization of the ssh-agent, at least we should check that it is already running

Minimize dependency

Python implementation still depends on the Python. Moreover if we want to separate it to more modules it must probably be installed. That could make problems if pip and/or venv is not available. To this end we propose:

1. use only standard Python packages
2. Do not use setuptools, make main bash scripts that calls python with modified PYTHONPATH.
3. For stable pieces of the code reimplement it in pure bash/sh.

stop ssh-agent on completion

swrap should start ssh-agent only if it is not running, or stop it after completion.

see the gist