swiftsim / swiftsimio

Python library for reading SWIFT data. Uses unyt and h5py.

SWIFTsimIO

The SWIFT astrophysical simulation code (http://swift.dur.ac.uk) is used widely. There exists many ways of reading the data from SWIFT, which outputs HDF5 files. These range from reading directly using h5py to using a complex system such as yt; however these either are unsatisfactory (e.g. a lack of unit information in reading HDF5), or too complex for most use-cases. swiftsimio provides an object-oriented API to read (dynamically) data from SWIFT.

Full documentation is available at ReadTheDocs.

Getting set up with swiftsimio is easy; it (by design) has very few requirements. There are a number of optional packages that you can install to make the experience better and these are recommended.

Requirements

This requires python v3.8.0 or higher. Unfortunately it is not possible to support swiftsimio on versions of python lower than this. It is important that you upgrade if you are still a python2 user.

Python packages

• numpy, required for the core numerical routines.
• h5py, required to read data from the SWIFT HDF5 output files.
• unyt, required for symbolic unit calculations (depends on sympy`).

Optional packages

• numba, highly recommended should you wish to use the in-built visualisation tools.
• scipy, required if you wish to generate smoothing lengths for particle types that do not store this variable in the snapshots (e.g. dark matter)
• tqdm, required for progress bars for some long-running tasks. If not installed no progress bar will be shown.
• py-sphviewer, if you wish to use our integration with this visualisation code.

Installing

swiftsimio can be installed using the python packaging manager, pip, or any other packaging manager that you wish to use:

pip install swiftsimio

Citing

Please cite swiftsimio using the JOSS paper:

@article{Borrow2020,
  doi = {10.21105/joss.02430},
  url = {https://doi.org/10.21105/joss.02430},
  year = {2020},
  publisher = {The Open Journal},
  volume = {5},
  number = {52},
  pages = {2430},
  author = {Josh Borrow and Alexei Borrisov},
  title = {swiftsimio: A Python library for reading SWIFT data},
  journal = {Journal of Open Source Software}
}

If you use any of the subsampled projection backends, we ask that you cite our relevant SPHERIC paper. Note that citing the arXiv version here is recommended as the ADS cannot track conference proceedings well.

@article{Borrow2021
  title={Projecting SPH Particles in Adaptive Environments}, 
  author={Josh Borrow and Ashley J. Kelly},
  year={2021},
  eprint={2106.05281},
  archivePrefix={arXiv},
  primaryClass={astro-ph.GA}
}

swiftsimio's Issues

Generation of the smoothing lengths

Hi,

When generating some initial conditions, you recommend in the documentation the following command:

# Generate initial guess for smoothing lengths based on MIPS
x.gas.generate_smoothing_lengths(boxsize=boxsize, dimension=3)

Here, it only works for a uniform box. Thus I think it would be nice to be more general and use the function in the visualization. It might be a good idea to keep both as the current implementation is a lot faster than the visualization one. Maybe we can add an argument to the function in order to mention if the box is uniform or not.

`cosmo_array` does not work with `matplotlib_support`

For some reason, the cosmo array doesn't work automatically with the matplotlib support, but if the array you wish to plot is cast to an unyt_array, this works:

with unyt.matplotlib_support:
    plt.plot(data.gas.masses)

fails, while

with unyt.matplotlib_support:
    plt.plot(unyt.unyt_array(data.gas.masses))

works.

Write a snapshot from the cell meta-data info

A feature that could be useful to have is that once a SWIFT snapshots has been open and the user specified a region of interest, the library could somehow write a mini stand-alone valid snapshot with only the particles in the region selected but copying over all the fields and meta-data.
That could be useful for people that only want to analyse one halo/region out of a big run. We can then provide a simple tool to extract things once only and then have a much more manageable data size to provide them.

This could also be used a service in the long run. For instance linked to some DB. A user identifies a halo of interest and then requests a mini-snapshot around that region.

For a rainy day I guess...

[JOSS review] some fixes required in the documentation

JOSS review issue: openjournals/joss-reviews#2430

Summary

First of all, swiftsimio is a really well written Python package, filling a need to analyse and visualise large SWIFT datasets. The particle masking feature is particularly cool. It has excellent internal documentation via docstrings. I like that you use black for standardised formatting.

There are a few relatively minor issues that need addressing detailed below.

Code

One minor thing is that there is a slightly confusing message: "You do not have numba installed. Please consider installing if you are going to be doing visualisation or indexing large arrays (pip install numba)" when importing. However, I do have numba (v0.48.0) installed. I think numba.core, which is the missing import that generates this message, was added in v0.49.1.

I have a small suggestion that it would be useful to add an environment.yml file to set up a conda environment for users or for development. E.g. I used this while testing the documentation code examples:

name: swiftsimio

channels:
  - conda-forge

dependencies:
  - python>=3.6
  - black
  - h5py
  - ipython
  - matplotlib
  - numba>=0.50
  - numpy
  - pip
  - pykdtree
  - pytest
  - scipy
  - tqdm
  - unyt
  - pip:
    - h5glance
    - py-sphviewer
    - swiftsimio
    - velociraptor
    - wily

Manuscript

The only comments I have about the manuscript are:

• there is a typo in the "Background" section in paragraph 1: "gallaxy";
• a small style-point: I think Python should be capitalised.

Documentation

There are several (mostly minor) issues with the documentation on readthedocs. (I was working off the stable version v4.0.0; commit hash 16e79ce8, using the provided dataset from http://virgodb.cosma.dur.ac.uk/swift-webstorage/IOExamples/cosmological_volume.hdf5.)

There are two changes that I suggest are important additions to the documentation.

1. I would like to see the output figures generated by the code examples in the "Visualisation" sections to check if the code is working when I run it.
2. It would be good to include a link to an example "catalogue file" required in the code examples in the "Visualisation/Projection" page.

In what follows I've detailed issues I found while running the example code in the documentation.

Loading Data

Using metadata

• Where it states "This will output [4.38929954e+26 4.38929954e+26 4.38929954e+26] cm" instead I get: [142.24751067 142.24751067 142.24751067] Mpc.
• There is a typo in "(a :mod:datetime object describing when the snapshot was written to disk)". I think you just need to replace ":mod:" with ":py:mod:".

Reading particle data

• Where it states "will output 3.08567758e+24*cm." instead I get: Mpc.
• Where it states "which will output '\frac{M_\odot}{\rm{Mpc}^{3}}'." instead I get: 2.0 \times 10^{30}\ \frac{\rm{kg}}{\rm{Mpc}^{3}}.

Named columns

• I think this is not formatted as intended. There seems to be a syntax error. I couldn't really run the code.

Non-unyt properties

• Where it states "which will output Co-moving mass densities of the particles." instead I get: AttributeError: 'cosmo_array' object has no attribute 'description'.

Masking

Spatial-only masking

• In the example, I think data = load(filename, mask=mask) should be sw.data = load(filename, mask=mask).

Full mask

• In the example on the line containing mask.constrain_mask("gas", "density", 0.4 * density_units, 0.8 * density_units) I get AttributeError: 'SWIFTUnits' object has no attribute 'gas'.

Writing a subset of snapshot

• In the example on the line containing sw.subset_writer.write_subset("test_subset.hdf5", mask) I get AttributeError: module 'swiftsimio' has no attribute 'subset_writer'.

Visualisation — Projection

It would be good to include the images generated by the examples, preferably using the available dataset.

Rotation

• In the example, a catalogue file is required. Can this be provided?

Lower-level API

• It would be nice if the example was executable.

Visualisation — Slices

Example

• In the second example, should "project_gas" should be replaced by "slice_gas"?

Visualisation — Volume Rendering

Example

• In the first code block, should "slice_gas" be replaced by "render_gas"?
• It seems like there may be a memory leak, or at least my laptop doesn't have enough memory to run the second code block with the dataset provided. The code crashes IPython.

Command-line Utilities

• Running swiftsnap on the provided dataset gives me: AttributeError: 'str' object has no attribute 'decode'

Visualisation routines should have some way of returning an `extent`

Visualisation routines should tell you the co-ordinates in space at which they were created.

Segfault in projection

We segfault in the projection if there is a particle outside of the boundaries of [0, 1] and it has a smoothing length so small that it is determined to use the PIC method.

to_physical() method of particle arrays correcting a factors in wrong direction

Expansion factor corrections seem to be correcting values in the wrong direction, (i.e. at higher z physical distances are a factor larger by a factor 1/a rather than smaller by a). This is true in general for other units, i.e. a factor a^-n is applied rather than a^n. Related, the test in swiftsimio/tests/test_physical_conversion.py is upside down (yielding physical coords > comoving coords)

Test and document wrapped masking

If we use a spatial only mask, the box is wrapped correctly (it seems). This is undocumented behaviour, and is untested. It would be good to fix both of thsoe.

Remove hand-casting of units

At the moment, at various points in the code, we currently multiply by unyt units to convert numpy arrays as follows:

x = np.arange(1000)
x_with_units = x * unyt.Mpc

This has the unintended effect of actually multiplying each value by a value, instead of just
adding the symbolic units for an array. For instance:

x = np.arange(1000)
units = 1.0 * unyt.Mpc
units.convert_to_cgs()
print(x * units) >>> .... 1.1231232e22 cm

Much better behaviour is

print(unyt.unyt_array(x, units=units) >>> ... 999 [1.12312e22cm]

(numbers are wrong, but this makes the point).

There should be a cosmo_quantity analog to unyt_quantity

Many of the unyt_array functions can return a unyt_quantity in at least some circumstances. For a cosmo_array, this leads to the extra attributes being silently dropped, and then potential unexpected results or errors when trying to access those or use the resulting unyt_quantity in a calculation where the user expected a cosmo_array-like output.

Feature request: Document the cosmology module

Following questions from users it would be good to say a few words in the documentation about the way swiftsimio extracts the cosmo info from a snapshot and turns it into a astropy.cosmology blob.

Add particle type 6

Parallel visualisation memory usage

The parallel visualisation routines have memory usage that scales with the number of threads; we create a copy of the output array for each thread and sum them at the end. Surely there is a faster and more memory-lean way of doing this?

Statistics docs are broke

https://swiftsimio.readthedocs.io/en/latest/statistics/index.html

Performance Regression in Visualistion

Changing from swiftsimio 3.1.0 to 3.1.2 I get that the code runs significantly slower as before. This is at least a difference of a factor of 10. This was all run on cosma7.

In the isolated-galaxy-plotting repository I have a script that did very bad called: edge_on_gas_and_stars.py

Let me know if you want to know something else.

Need to mock smoothing lengths even with backend=histogram

When using backend=histogram in project_pixel_grid we still need to have a 'smoothing lengths' attribute even though it is not used.

Crashing code:

import swiftsimio as sw
import unyt

from swiftsimio.visualisation.projection import project_pixel_grid

import sys

filename = sys.argv[1]

mask = sw.mask(filename)
mask.constrain_spatial([None, None, [0.0 * unyt.Mpc, 50 * unyt.Mpc]])
data = sw.load(filename, mask)

img = project_pixel_grid(
    data=data.dark_matter,
    boxsize=data.metadata.boxsize,
    resolution=2048,
    parallel=True,
    backend="histogram",
)

Fixed by including:

data.dark_matter.smoothing_lengths = [0.0]

Reading element mass fractions is broken when loading particles from halo

MWE:

from velociraptor import load as load_catalogue
from velociraptor.particles import load_groups
from velociraptor.swift.swift import to_swiftsimio_dataset


catalogue_name = "velociraptor"
snapshot_name = "snapshot"

catalogue = load_catalogue(f"{catalogue_name}.properties")
groups = load_groups(f"{catalogue_name}.catalog_groups", catalogue=catalogue)

particles, unbound_particles = groups.extract_halo(halo_id=0)

data, mask = to_swiftsimio_dataset(
    particles,
    f"{snapshot_name}.hdf5",
    generate_extra_mask=True
)

data.gas.element_mass_fractions.hydrogen

Segmentation fault

element_mass_fractions for both gas and stars is broken. Returns Segmentation fault

Crash in a SWIFT makeIC script

I am running on cosma7 with the python/3.6.5 module. When running the makeIC.py in the ConstantCosmoTempEvolution directory, I get the following error:

Traceback (most recent call last):
  File "makeIC.py", line 21, in <module>
    from swiftsimio.units import cosmo_units
  File "/cosma/home/jlvc76/.local/lib/python3.6/site-packages/swiftsimio/units.py", line 15, in <module>
    "cosmological", unyt.Mpc, 1e10 * unyt.msun, unyt.s * unyt.Mpc / unyt.km
  File "/cosma/home/jlvc76/.local/lib/python3.6/site-packages/unyt/unit_systems.py", line 218, in __init__
    bu = _split_prefix(unit, default_lut)[1]
  File "/cosma/home/jlvc76/.local/lib/python3.6/site-packages/unyt/unit_systems.py", line 111, in _split_prefix
    possible_prefix = symbol_str[0]
TypeError: 'Unit' object does not support indexing

PyPi homepage

On the PyPi webpage, the home page for the project is cosma's gitlab and not github.

The issue while using the VELOCIraptor

Hi, i'm Zhen. Now I'm using swiftsimio for my research project. When i tried to extract a halo and make a plot, i got some error.
Here is the error.


Here is my script.

I dont know how to fix this issue. Im thinking about is it the some problems with system's numba or something wrong in the swiftsimio?

Thanks in advance!

Cell masking does not give correct results

Cell metadata masking is giving dodgy results. Testing on replicated EAGLE_12 boxes.

Here we have: orange - region we asked to read, blue shows all top level cells, red shows cells that we think were read (ha!), and the background shows the particles we read.

I think this has to be something to do with unyt's unit conversions...

Note that here we ask to read 7-10 mpc each time, but even the rectangle gets larger - wtf?

Sketchy-ass code to test this:

from swiftsimio import mask, load
from time import time
from unyt import Mpc
from swiftsimio.visualisation import project_gas_pixel_grid as project_gas
import matplotlib.pyplot as plt
from matplotlib.colors import LogNorm
from matplotlib.patches import Rectangle

repeats = []
timings = []
middle_timings = []
lens = []
maxs = []
projections = []

read = [[7.0 * Mpc, 10.0 * Mpc]] * 3
print(read)

for repeat in [1, 2, 3, 4, 5]:
    filename = f"{repeat**3}/eagle_0000.hdf5"

    middle = time()

    initial = time()
    

    try:
        hello = read.copy()[0]
        rect = Rectangle(
                xy=[hello[0].value, hello[0].value],
                width=(hello[1] - hello[0]).value,
                height=(hello[1] - hello[0]).value,
                fill=False,ec="orange")
        this_mask = mask(filename)
        this_mask.constrain_spatial(restrict=read.copy())
        cells_read = this_mask.centers[this_mask._generate_cell_mask(read.copy())]
        data = load(filename, mask=this_mask)
        proj = project_gas(data, resolution=256)
        projections.append(proj)

        print(this_mask.cell_size.to("Mpc"))

        plt.scatter(this_mask.centers[:, 0].to("Mpc").value, this_mask.centers[:, 1].to("Mpc").value, s=1)
        plt.scatter(cells_read[:, 0].to("Mpc").value, cells_read[:, 1].to("Mpc").value, s=1, color="red")
        plt.xlim(0, this_mask.metadata.boxsize[0].to("Mpc").value)
        plt.ylim(0, this_mask.metadata.boxsize[1].to("Mpc").value)
        plt.imshow(proj.T, origin="lower", extent=[0, this_mask.metadata.boxsize[0].to("Mpc").value, 0, this_mask.metadata.boxsize[1].to("Mpc").value], norm=LogNorm())

        plt.gca().add_artist(rect)
        plt.savefig(f"{repeat}_cells.png")
        plt.cla()
        plt.clf()
        plt.close()

        middle = time()
        maxs.append(data.gas.coordinates.max())
        lens.append(len(data.gas.coordinates))
    except (FileNotFoundError, IOError):
        print(f"Failed {filename}")
        pass

    final = time()
    
    timings.append(final - initial)
    middle_timings.append(final - middle)
    repeats.append(repeat)

Provide an astropy.cosmology object for the relevant cosmology

Generic support for fields

At the moment, every time a new field is added, we need to add it to the metadata. We should be able to surface this information from the HDF5 files (including cosmology factors and units) as SWIFT outputs these.

Python3.8 warnings

We get the following warnings from python3.8. We should fix these, but only after merging @aborissov's latest changes to the writer.

swiftsimio/writer.py:126
  /Users/mphf18/Documents/GitHub/swiftsimio/swiftsimio/writer.py:126: SyntaxWarning: "is" with a literal. Did you mean "=="?
    if name is "particle_ids":

swiftsimio/writer.py:245
  /Users/mphf18/Documents/GitHub/swiftsimio/swiftsimio/writer.py:245: SyntaxWarning: "is not" with a literal. Did you mean "!="?
    if dimensions is not 1:

-- Docs: https://docs.pytest.org/en/latest/warnings.html

Visualisation suggestions

We recieved a lot of feedback for visualisation. Current issues:

• Fix Sylvia's breaking bug with an EAGLE_6 COLIBRE run
• Add angle and position centering for images
• Add non-square image generation
• Add a better API for visualising derived fields?

Gas gamma warning

Could not find gas gamma, assuming 5./3. is printed every mask or load if it's missing from a snapshot, yet isn't needed for generic equations of state.

Add new time base quantities

[JOSS review] Some fixes and suggestions for the documentation

JOSS review issue: openjournals/joss-reviews#2430

Summary

This is a really great code. There is a clearly demonstrated need and the authors of the code have found creative ways to allow the user to analyze large simulation data in a memory efficient way. The code is intuitive to use, as well. From a practical standpoint, I would have liked to see the code be able to interface more seamlessly with halo catalogs, but I do not think that is a huge issue. The paper is well written, as is the very detailed documentation. Below I outline a few suggestions and edits to be made to the documentation. In particular, I had issues performing some functions and had to update a package (numba).

Documentation Comments

1. Given that the typical user will likely be interacting with a halo catalog, it would be nice to have a section devoted to such examples. Currently, direct use of velociraptor catalogs is deferred to the section on rotating projections. A clear example of how one could use his code in conjunction with a halo catalog will be useful. Making a separate section would make this information easier to find. At the start of this section would also be a useful place to link to the velociraptor code repository as well. This would likely be a good fit near the beginning of the Masking section.

2. In the Projections:Backends sub-section, when discussing sub-sampled projections the example given uses the star formation rate but for this particular example dataset this value is all 0. It might be nice to present an example that can actually produce a meaningful image.

3. In the Projection:Rotation subsection, there is a typo. The variable vector passed as an argument to rotation_matrix_from_vector() should be angular_momentum_vector I think.

4. Also in the Rotation subsection, I was initially unable to run the line un_rotated = project_gas_pixel_grid(**common_arguments) and instead got the traceback below. The problem was solved by updating numba from 0.48.0 to 0.50.1 (I had to update my entire anaconda environment to do this, but based on the traceback I think numba was the key). It should be made clear that there is a version dependence on numba.

In [27]: un_rotated = project_gas_pixel_grid(**common_arguments) 
    ...:                                                                                                                                                   
---------------------------------------------------------------------------
TypingError                               Traceback (most recent call last)
<ipython-input-27-73ddb1693c8e> in <module>
----> 1 un_rotated = project_gas_pixel_grid(**common_arguments)

~/anaconda3/lib/python3.6/site-packages/swiftsimio/visualisation/projection.py in project_gas_pixel_grid(data, resolution, project, region, mask, rotation_matrix, rotation_center, parallel, backend)
    303         rotation_matrix=rotation_matrix,
    304         rotation_center=rotation_center,
--> 305         backend=backend,
    306     )
    307 

~/anaconda3/lib/python3.6/site-packages/swiftsimio/visualisation/projection.py in project_pixel_grid(data, boxsize, resolution, project, region, mask, rotation_matrix, rotation_center, parallel, backend)
    139             )
    140 
--> 141     number_of_particles = data.coordinates.shape[0]
    142 
    143     if project is None:

~/anaconda3/lib/python3.6/site-packages/swiftsimio/reader.py in getter(self)
    835                                     output_shape=output_shape,
    836                                     output_type=output_type,
--> 837                                     columns=columns,
    838                                 ),
    839                                 unit,

~/anaconda3/lib/python3.6/site-packages/swiftsimio/accelerated.py in read_ranges_from_file(handle, ranges, output_shape, output_type, columns)
    495     )
    496 
--> 497     return read_ranges(handle, ranges, output_shape, output_type, columns)

~/anaconda3/lib/python3.6/site-packages/swiftsimio/accelerated.py in read_ranges_from_file_chunked(handle, ranges, output_shape, output_type, columns)
    397 
    398     # Make array of chunk ranges
--> 399     chunk_ranges = get_chunk_ranges(ranges, chunk_size, handle.shape[0])
    400     chunk_range_size = np.diff(chunk_ranges).sum()
    401 

~/anaconda3/lib/python3.6/site-packages/numba/dispatcher.py in _compile_for_args(self, *args, **kws)
    399                 e.patch_message(msg)
    400 
--> 401             error_rewrite(e, 'typing')
    402         except errors.UnsupportedError as e:
    403             # Something unsupported is present in the user code, add help info

~/anaconda3/lib/python3.6/site-packages/numba/dispatcher.py in error_rewrite(e, issue_type)
    342                 raise e
    343             else:
--> 344                 reraise(type(e), e, None)
    345 
    346         argtypes = []

~/anaconda3/lib/python3.6/site-packages/numba/six.py in reraise(tp, value, tb)
    666             value = tp()
    667         if value.__traceback__ is not tb:
--> 668             raise value.with_traceback(tb)
    669         raise value
    670 

TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Direct iteration is not supported for arrays with dimension > 1. Try using indexing instead.
[1] During: typing of intrinsic-call at /home/michael/anaconda3/lib/python3.6/site-packages/swiftsimio/accelerated.py (237)

File "anaconda3/lib/python3.6/site-packages/swiftsimio/accelerated.py", line 237:
def get_chunk_ranges(
    <source elided>
    n_ranges = len(ranges)
    for bounds in ranges:
    ^

5. In the Py-SPHViewer section, I believe there is a typo. When you call SPHViewer() in this code snippet I think you want to call SPHViewerWrapper().

6. Also in the Py-SPHViewer section, just for completeness include an import matplotlib.pyplot as plt in the code snippet.

`to_physical()` requires data to be read in

To use the cosmology transforms we must have already accessed the data - this is unecessary and crashes in a nasty way if it is not the case.

Cells that do not contain any particles of a given type break read_ranges_from_file

When trying to read a masked dataset (i.e. using SWIFTMask), if we read cells that contain no particles of a given type (usually stars), this crashes read_ranges_from_file.

Example:

x.mask.stars = array([[1171805, 1171815],
       [1171816, 1171817],
       [1171818, 1171819],
       [1303295, 1303308],
       [1303309, 1322907],
       [1322908, 1335147],
       [1335881, 1335880],
       [1335881, 1344208],
       [1344209, 1344459],
       [1394689, 1394689], # bad cell
       [1394690, 1566551],
       [1566552, 1566784],
       [1568805, 1568804],
       [1568805, 1569776],
       [1569777, 1569779],
       [1661180, 1661179],
       [1661180, 1661901],
       [1661902, 1661906]])

x.stars.particle_ids

output:

---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
<ipython-input-13-c6e9e6e099e1> in <module>()
----> 1 x.stars.particle_ids

/cosma7/data/dp004/dc-borr1/pysimba/lib/python3.6/site-packages/swiftsimio/reader.py in getter(self)
    611                                     mask,
    612                                     output_shape=output_shape,
--> 613                                     output_type=output_type,
    614                                 ),
    615                                 unit,

/cosma7/data/dp004/dc-borr1/pysimba/lib/python3.6/site-packages/swiftsimio/accelerated.py in read_ranges_from_file(handle, ranges, output_shape, output_type)
     99         output_dest_sel = np.s_[already_read : size_of_range + already_read]
    100 
--> 101         handle.read_direct(output, source_sel=hdf5_read_sel, dest_sel=output_dest_sel)
    102 
    103         already_read += size_of_range

/cosma7/data/dp004/dc-borr1/pysimba/lib/python3.6/site-packages/h5py/_hl/dataset.py in read_direct(self, dest, source_sel, dest_sel)
    730                 dest_sel = sel.select(dest.shape, dest_sel, self.id)
    731 
--> 732             for mspace in dest_sel.broadcast(source_sel.mshape):
    733                 self.id.read(mspace, fspace, dest, dxpl=self._dxpl)
    734 

/cosma7/data/dp004/dc-borr1/pysimba/lib/python3.6/site-packages/h5py/_hl/selections.py in broadcast(self, target_shape)
    307         tshape = tuple(tshape)
    308 
--> 309         chunks = tuple(x//y for x, y in zip(count, tshape))
    310         nchunks = int(np.product(chunks))
    311 

/cosma7/data/dp004/dc-borr1/pysimba/lib/python3.6/site-packages/h5py/_hl/selections.py in <genexpr>(.0)
    307         tshape = tuple(tshape)
    308 
--> 309         chunks = tuple(x//y for x, y in zip(count, tshape))
    310         nchunks = int(np.product(chunks))
    311 

ZeroDivisionError: integer division or modulo by zero

Simple fix, just need to check for empty ranges.

Adding material ID attribute

Hi,
I'm running a planetary simulation, how do I add material IDs property?

thanks

Visualisation rotations

The projection visualisation routines have the ability to provide arbritary rotations. This should be true for slices and volumetric rendering too.

Their documentation also needs to be updated and be consistent with the numpy style.

Cosmological scale factor conversion

We used to have a method to_physical() on datasets; this had to be removed because it was really buggy.

We currently deal with this in the following way: https://swiftsimio.readthedocs.io/en/latest/loading_data/index.html#non-unyt-properties but it would be nice to have a cleaner solution.

Numba error with VELOCIraptor integration

Hello,
I'm a student working with Matthieu Schaller and after following the example code proposed here https://swiftsimio.readthedocs.io/en/latest/velociraptor/index.html (I just replaced the data from the FLAMINGO simulation instead of the toy model) I got a very long error related to Numba with this final message:

---

LoweringError: Failed in nopython mode pipeline (step: nopython mode backend)
iterating over 2D array

File "../.local/lib/python3.6/site-packages/swiftsimio/accelerated.py", line 247:
def get_chunk_ranges(

n_ranges = len(ranges)
for bounds in ranges:
^

[1] During: lowering "$18.2 = iternext(value=$phi18.1)" at /home/filipello/.local/lib/python3.6/site-packages/swiftsimio/accelerated.py (247)

This should not have happened, a problem has occurred in Numba's internals.

Please report the error message and traceback, along with a minimal reproducer
at: https://github.com/numba/numba/issues/new

If more help is needed please feel free to speak to the Numba core developers
directly at: https://gitter.im/numba/numba

Thanks in advance for your help in improving Numba!

Is this strictly related to the SWIFTsimIO package or should I ask on the Numba Gituhub?

Thanks in advance

Datetime does not work with CEST timezone

%Z does not recognise CEST as a valid timezone for some reason. Need to find a workaround.

Large spatial masks are very slow

Seems that reading spatial masked data is very slow on COSMA-7 (100mb/s).

JH is unable to reproduce - could be to do with version of h5py?

COLIBRE-6 quarter-box visualisation crashes

SP has reported a crashing bug.

/cosma7/data/dp004/dc-ploe1/SIMULATION_RUNS/2019_11_small_cosmo_volume_tests/L06N188_CDM/data/snaps/snap_0137.hdf5

crashes when creating a [0.0, 0.25] image.

Masked subset writing not working

Writing a masked subset of the snapshot doesn't seem to work.
Below is the error I receive on both cosma and Leiden systems and on several different snapshots and with different python versions.

>>> mask = sw.mask(filename)
>>> # load_region is a 3x2 list [[left, right], [bottom, top], [front, back]]
... load_region = [[0, 1], [0, 1], [0, 1]]
>>>
>>> # Constrain the mask
... mask.constrain_spatial(load_region * Mpc)
>>> sw.subset_writer.write_subset('test_subset.hdf5', mask)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/cosma/home/dp004/dc-bras1/local_env/lib64/python3.6/site-packages/swiftsimio/subset_writer.py", line 326, in write_subset
    write_datasubset(infile, outfile, mask, find_datasets(infile), list_of_links)
  File "/cosma/home/dp004/dc-bras1/local_env/lib64/python3.6/site-packages/swiftsimio/subset_writer.py", line 268, in write_datasubset
    mask_size = get_dataset_mask(mask, name, suffix="_size")
  File "/cosma/home/dp004/dc-bras1/local_env/lib64/python3.6/site-packages/swiftsimio/subset_writer.py", line 69, in get_dataset_mask
    part_type = [int(x) for x in filter(str.isdigit, dataset_name)][0]
IndexError: list index out of range```

Write a snapshot from the cell meta-data info

A feature that could be useful to have is that once a SWIFT snapshots has been open and the user specified a region of interest, the library could somehow write a mini stand-alone valid snapshot with only the particles in the region selected but copying over all the fields and meta-data.
That could be useful for people that only want to analyse one halo/region out of a big run. We can then provide a simple tool to extract things once only and then have a much more manageable data size to provide them.

This could also be used a service in the long run. For instance linked to some DB. A user identifies a halo of interest and then requests a mini-snapshot around that region.

For a rainy day I guess...

Check h5py load calls

Check all h5py load calls to see if they do not have a 'r' argument. They should explicitly state this as the future API is changing this default.

Use named fields instead of parttype0

SWIFT now outputs named fields, such as GasParticles, as well as PartType{X}. We can use this to get rid of even more metadata!

Multiplying by cosmo_factor is very slow

Multiplying an unyt array with a substituted cosmo factor is very slow.

density_factor = data.gas.densities.cosmo_factor.a_factor
number_density = (data.gas.densities * (density_factor / mh)).to(cm**-3)

is much slower than

density_factor = float(data.gas.densities.cosmo_factor.a_factor)
number_density = (data.gas.densities * (density_factor / mh)).to(cm**-3)

Import of smoothing_length_generation does not work as expected

importing visualisation as:
import swiftsimio.visualisation as swvis
and than generating the smoothing lengths as:
swvis.smoothing_length_generation.generate_smoothing_lengths()
does not find the function while:
from swiftsimio.visualisation.smoothing_length_generation import generate_smoothing_lengths
does work.

load_statsitics can have ( in variable names

e.g. from the sfr.

you can get sfr_(total), which you can access with getattr(stats, "sfr_(total)") but not directly.

Numba does not like partial types

Using the dscale filters, etc, causes issues when running with numba. It does not like the partial types (won't compile when using them) and if you force it with x.dtype = np.float32 or similar you segfault as it tries to read off the end of the array.

Cell Metadata

Looks like the cell metadata is broken at the moment.

Box-wrapping

Currently swiftsimio does not natively support box-wrapping. Would be nice to be able to request [0.9, 1.1] * boxsize and get it to give you [0.0, 0.1] and [0.9, 1.0].

Can no longer read ICs created by swiftsimio

It appears that we can no longer read our own files. This shouldn't happen, as it should be covered by one of the unit tests.

Could be something to do with weird particle types.

Segfault while doing simple swiftsimio operations

Sometimes I get a segfault with swiftsimio while I try to load a files. I saw this a few times before but I thought it was something I did, because it rarely occurred at lower resolution/smaller files. It does not seem to be always the same snapshot that is the problem either, so therefore it is a bit hard to track down, if I do things with several snapshots is does not really show the bug at the same one. Maybe it is two separate bugs, I sometimes get a segfault and sometimes a abort signal. The program I use that gives this bug, loads the files, converts arrays to other units and makes a few surface density plots.

The first error I get is:
61%|██████████████████████████████████████████████████████████████▌ | 31/51 [09:25<06:01, 18.07s/it] Program received signal SIGSEGV, Segmentation fault. 0x00002aaaab8b0b9c in free () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 expat-2.1.0-12.el7.x86_64 fontconfig-2.13.0-4.3.el7.x86_64 freetype-2.8-14.el7_9.1.x86_64 glibc-2.17-324.el7_9.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-50.el7.x86_64 libX11-1.6.7-3.el7_9.x86_64 libXau-1.0.8-2.1.el7.x86_64 libXcursor-1.1.15-1.el7.x86_64 libXfixes-5.0.3-1.el7.x86_64 libXft-2.3.2-2.el7.x86_64 libXrender-0.9.10-1.el7.x86_64 libcom_err-1.42.9-19.el7.x86_64 libffi-3.0.13-19.el7.x86_64 libpng-1.5.13-8.el7.x86_64 libselinux-2.5-15.el7.x86_64 libuuid-2.23.2-65.el7_9.1.x86_64 libxcb-1.13-1.el7.x86_64 nss-softokn-freebl-3.53.1-6.el7_9.x86_64 openssl-libs-1.0.2k-21.el7_9.x86_64 pcre-8.32-17.el7.x86_64 sssd-client-1.16.5-10.el7_9.7.x86_64 tcl-8.5.13-8.el7.x86_64 tk-8.5.13-6.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-19.el7_9.x86_64

And the second error I sometimes get is:
91%|█████████████████████████████████████████████████████████████████████████████████████████████▋ | 10/11 [02:24<00:14, 14.31s/it]*** Error in/cosma/local/Python/3.9.1-cosma7/bin/python3': free(): invalid next size (fast): 0x0000000003c66fe0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81329)[0x2aaaab8ac329]
/cosma/home/durham/dc-nobe1/.local/lib/python3.9/site-packages/h5py/../h5py.libs/libhdf5-4a4ec477.so.200.0.0(H5MM_xfree+0xe)[0x2aaafe7420ee]
/cosma/home/durham/dc-nobe1/.local/lib/python3.9/site-packages/h5py/../h5py.libs/libhdf5-4a4ec477.so.200.0.0(H5A__shared_free+0x38)[0x2aaafe61cae8]
/cosma/home/durham/dc-nobe1/.local/lib/python3.9/site-packages/h5py/../h5py.libs/libhdf5-4a4ec477.so.200.0.0(H5A__close+0xe0)[0x2aaafe61cdc0]

Many more lines like this ....

2aaaaacac000-2aaaaacb2000 r--s 00000000 fd:00 424286 /usr/lib/fontconfig/cache/41105eee-da7f-4edf-a2d0-7e0a63c9006a-le64.cache-7
Program received signal SIGABRT, Aborted.
0x00002aaaab861387 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 expat-2.1.0-12.el7.x86_64 fontconfig-2.13.0-4.3.el7.x86_64 freetype-2.8-14.el7_9.1.x86_64 glibc-2.17-324.el7_9.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-50.el7.x86_64 libX11-1.6.7-3.el7_9.x86_64 libXau-1.0.8-2.1.el7.x86_64 libXcursor-1.1.15-1.el7.x86_64 libXfixes-5.0.3-1.el7.x86_64 libXft-2.3.2-2.el7.x86_64 libXrender-0.9.10-1.el7.x86_64 libcom_err-1.42.9-19.el7.x86_64 libffi-3.0.13-19.el7.x86_64 libpng-1.5.13-8.el7.x86_64 libselinux-2.5-15.el7.x86_64 libuuid-2.23.2-65.el7_9.1.x86_64 libxcb-1.13-1.el7.x86_64 nss-softokn-freebl-3.53.1-6.el7_9.x86_64 openssl-libs-1.0.2k-21.el7_9.x86_64 pcre-8.32-17.el7.x86_64 sssd-client-1.16.5-10.el7_9.7.x86_64 tcl-8.5.13-8.el7.x86_64 tk-8.5.13-6.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-19.el7_9.x86_64

`

Let me know if I can help by providing more information.

Exception when loading test volume

I noticed that when loading the test volume (http://virgodb.cosma.dur.ac.uk/swift-webstorage/IOExamples/cosmo_volume_example.hdf5) using data = load("cosmo_volume_example.hdf5") an IndexError is thrown when creating a MassTable in SWIFTMetadata.postprocess_header().

I think this might be due to the most recent version having 7 particle types while the test volume only has 6. As a workaround for now I've removed the sixth entries of the three arrays in particle_types.py:

import swiftsimio.metadata.particle as swp

swp.particle_name_underscores.pop(6)
swp.particle_name_class.pop(6)
swp.particle_name_text.pop(6)

This seemed to work as I could then load the snapshot as expected. I'm still just trying to figure out the SWIFT basics but hopefully this is helpful!

Cheers, F