bwvdnbro / cmacionize

C++ Monte Carlo photoionization and radiation hydrodynamics code.

CMacIonize

C++ Monte Carlo photoionization and radiation hydrodynamics code.

• Obtaining and compiling the code
• Code dependencies
• Documentation
• Support

Compilation

First, clone the repository using

git clone https://github.com/bwvdnbro/CMacIonize.git

This creates a new folder named CMacIonize. Enter this folder and create a new build folder:

cd CMacIonize
mkdir build

Enter the build folder and call CMake:

cd build
cmake ..

Now compile the program using make.

Documentation can be generated using make doc.

Unit tests can be run using make check.

A number of benchmark problems is provided in the folder build/rundir/benchmark. Each benchmark problem consists of at least a parameter file that can be used to run the benchmark, and a .txt file containing some more information about the benchmark problem.

Dependencies

The only thing you need to compile and run CMacIonize is the CMake build system and a C++ compiler. However, some plugins for reading and writing files require the HDF5 library as well. The CMake build system automatically tries to locate a working HDF5 installation on your system. If this fails, the program will still compile, but the following functionality will not work:

• reading a density field from a Gadget snapshot file
• reading a density field from a FLASH snapshot file
• reading ionizing sources from a Gadget snapshot file
• writing Gadget output files. Since this is currently the only supported output format, this means output will not work, and the program is in fact useless.

To install the HDF5 (development) libraries on a Linux system, use

sudo apt-get install libhdf5-dev

On Mac OS X, the easiest way to install the HDF5 (development) libraries is a manual installation. Get the latest HDF5 release tarball from the HDF5 website (make sure to choose a distribution with UNIX line endings). Extract the tarball in a folder of choice, and configure and build the program:

./configure
make
make install

Depending on where the HDF5 library is installed, you might need to tell CMake where to find HDF5: locate the folder containing libhdf5.so (or similar). This folder should be named lib, and should be part of a parent folder that also contains folders named include and bin. You should set the environment variable HDF5_ROOT to the name of this folder:

export HDF5_ROOT=/name/of/the/folder

Check the messages generated when you run CMake, as it will tell you whether or not a working HDF5 development installation was found.

CMacIonize uses Doxygen to generate code documentation, but this is in no way necessary to compile or run the program. CMake will automatically try to find a Doxygen installation. If it finds one, you can generate documentation using make doc. If not, make doc will not work.

Some scripts used for analyzing benchmark results, or for writing files used by the unit tests, require Python, and the numpy, matplotlib and h5py libraries. However, all files that are necessary for the unit tests are also included in the repository (even if the scripts to generate them are), so it is still possible to compile and run the code without Python.

Documentation

The CMacIonize code contains a full inline documentation using Doxygen. A recent version of this documentation is available from an online mirror. A small number of online tutorials is available from the CMacIonize webpage. Other sources of documentation include:

• the journal article describing the original photoionization code on which CMacIonize was based, Wood, Mathis & Ercolano (2004)
• the journal article describing CMacIonize 1.0, Vandenbroucke & Wood (2018)
• the journal article describing the task-based algorithm underlying CMacIonize 2.0, Vandenbroucke & Camps (2020)

Support

If you have any issues obtaining, compiling or running the code, please consult the wiki or the online tutorials, or let us know by email ([email protected]). A Slack workspace is available for regular users and contributors upon invitation. Try to use github issue were applicable to

• submit bug reports
• request new code features

cmacionize's Issues

Tutorials are out of date

The tutorials on the web page still contain explicit references to the now removed stable and other branches.

MD5 checksum for large input files

Reason for the proposed new feature

Currently we try to improve reproducibility by making the parameter file and all configuration flags part of every snapshot output. However, this approach does not guarantee reproducibility if the simulation requires additional input files, e.g. a density function that is read in from a simulation snapshot. It is impossible to include these in every output, but we could add a check to make sure the file we are reading is actually the same. MD5 checksums are widely used for this, so that seems like a good option.

Example of the new feature

When running a simulation that reads input from a snapshot file, we could compute the checksum when we open the file and then add it to every CMacIonize output, just like we add all parameters and configuration flags. When rerunning a simulation, we could check that the input file has the expected checksum and display a warning/error if it does not (we would need to tell CMacIonize we are trying to rerun a simulation of course). Or we could leave it up to the user to compute the checksum, in which case we need to make sure we implement the same checksum algorithm as available on Linux systems.

Implementation ideas

The MD5 checksum algorithm is described in detail on Wikipedia: https://en.wikipedia.org/wiki/MD5. We would need to find a good place to implement it (maybe find some general way of reading/logging external files?) and then add the resulting checksum to either the parameter block in the snapshots, or to a new block.

Restart files

Reason for the proposed new feature

RHD runs can take a long time and many things could go wrong while they are running. Intermediate binary dumps that can be used to quickly restart a simulation as if it never stopped could be a useful feature.

Example of the new feature

Restart files should be written out automatically at specific intervals during the run (the time interval could be a parameter). Additionally, there should be a mechanism to force a restart file write (e.g. using a clean up phase when the user manually kills the simulation with CTRL+C). The code should be able to restart from these files, basically as if it never stopped.

Implementation ideas

We will first need to figure out if it is possible to capture the CTRL+C kill command and use it to run the simulation until a good point to write a dump. If not, we will need another way to force a restart file write (like the touch stop trick Gadget2 uses). We then need to decide which information needs to be present in the restart files and set up a general mechanism of dumping and reloading classes. It seems like a good idea to start by only doing this for RHD simulations.

testCMILibrary and testCMICLibrary are too slow

After merging in the new density mapping, these two unit tests take more than a minute to complete, which is way too long for a unit test.

Possible solutions: use less resolution elements or optimise the mapping algorithm so that the tests run faster again.

DensityGrid initialization should be postponed until after parameters-usedvalues write

Currently, the DensityGrid is initialized in the constructor. Since this initialization can be computationally expensive, this means the program can take quite some time, and then crash because of a small mistake in the parameter file.
It would be cleaner if the DensityGrid initialization was postponed until after all parameters have been read, so that parameter file bugs can be found very quickly. This also means we could make a --dry-run version of the program that only parses the parameter file and then exits. This makes it easier to iteratively construct a new parameter file.

Add parameter file version number and change scripts

Some code changes affect the parameter file: parameters become obsolete or get replaced by other parameters, new parameter blocks are added and sometimes have non-trivial default values... This makes (re)using old parameter files problematic in some cases.

The solution to this would be to keep track of the code version when writing a parameter file (especially when writing the parameters-usedvalues file). Then it would become possible to provide Python scripts that automatically update old parameter files whenever such changes occur.

GadgetDensityGridWriter uses too much memory

Since GadgetDensityGridWriter first allocates internal buffers, then fills them, and then writes them to the snapshot file, it uses way too much memory for large grid sizes.

There are a few solutions for this:

• writing one dataset at a time
• writing datasets in chunks
• both of the above

This is certainly something to consider before releasing v1.0.

CMAKE_BUILD_TYPE not set

Doing mkdir build && cd build && cmake .. after cloning throws

 CMake Error at CMakeLists.txt:82 (if):
    if given arguments:

      "STREQUAL" "Debug"

    Unknown arguments specified

for me. May I suggest adding an additional check on whether CMAKE_BUILD_TYPE is set or not? Like so:

 if(${CMAKE_BUILD_TYPE})
   if(${CMAKE_BUILD_TYPE} STREQUAL "Debug")
     add_compiler_flag("-fsanitize=address -fno-omit-frame-pointer" OPTIONAL)
   endif(${CMAKE_BUILD_TYPE} STREQUAL "Debug")
 endif(${CMAKE_BUILDTYPE})

Wrong units for face vertices in OldVoronoiGrid with REMAP

When OLDVORONOIGRID_REMAP is enabled, the face vertex coordinates have the wrong units, as they are not converted back to the original units.

Repair or remove the convergence checkers.

The convergence checkers are broken in the current development version of the code, since they are not compatible with the distributed memory parallelization, and various other reasons. Do we really need them? If so, we need to think about repairing them.

Two versions of AsciiFilePhotonSourceDistribution

After merging in the new SPHArrayInterface, we are left with two versions of AsciiFilePhotonSourceDistribution that are different... This makes a rebase of the pmacionize branch against the development branch currently impossible.

Solution: rename one of them.

Check sizes of integer variables

The fact that unsigned long integers are not necessarily 64 bit on all operating systems has raised some concerns about the usage of unsigned long throughout the code.

There are basically two use cases: either you use unsigned long because you need a 64 bit integer, and then you should definitely use uint64_t instead. Or you use unsigned long as an index for an array that could potentially contain more than 2^32 elements, in which case you should use size_t.

It should be worth the effort to go through the entire code and check if the right integer type is being used.

write_compiler_info.cmake does not work on Windows

write_compiler_info.cmake uses the Unix command date, which is not available on Windows. This leads to errors when trying to compile CMacIonize, as CompilerInfo.cpp cannot be configured correctly.

Install from .zip/.tar.gz does not work

Since CMake contains a command that generates a version file based on git, installing from a .zip/.tar.gz directly downloaded from GitHub does not work if git is installed (not sure about the case where git is not installed). Since this is obviously a possible way people would want to install the code, this should definitely be fixed.

Add He++

For some sources (e.g. a black body at 100,000 K or the spectrum of a very old star), the temperatures calculated with CMacIonize defer significantly from those of Cloudy: the CMacIonize temperature is high but pretty flat near the centre for a uniform box, while in Cloudy the central temperature is much higher and then falls of. The reason turns out to be double ionized helium: the Cloudy output indicates high ionic fractions for He++ in the central region with the higher temperature. CMacIonize does not have He++ and hence significantly underestimates the photoionization heating.

Adding He++ is very complicated: it requires the addition of an additional helium ion and a thorough change to the ionization balance calculation. Additionally, the addition of a number of other elements with ionization energies close to that of He++ is required to get the cooling right.

The best strategy to embark upon this ambitious change would be to

• first add the atomic data required for He++ and the other metal ions that need to be included. These can be tested independently as part of the existing unit tests.
• then add He++ to the ionization balance by running some fixed temperature simulations. Hopefully it will be possible to benchmark these against Cloudy.
• finally, the additional metals can be added and a full comparison with Cloudy for a high temperature black body can be done. Since the computation of the ionization balance of metals can be implemented independently of the H/He balance, this should not be too hard.

Add version command line option/parameter

For some simulations, it would be useful to have a check on the used program version, e.g. if you are running using a workflow system and want to make absolutely sure that every remote node runs the same version. So you could add a program version parameter or command line option, and check the version at the start of the simulation and crash if the wrong version is being run.

Code does not compile without OpenMP

Some of the new task-based algorithm code does not properly check if OpenMP is available before including the omp.h header file. This breaks compilation on systems that do not have OpenMP (e.g. the Clang compiler used by Travis CI).

Decrease initial photon packet number when periodic boundaries are used.

Since the neutral fractions are initialized to very low values at the start of a run, the initial step can take a very long time in simulations with periodic boundaries, especially if boundaries are periodic in more than 1 direction. It is probably better to decrease the initial number of photon packages in this case, especially since the very low attenuation per cell means even a small number of photon packages will already visit a large number of cells.