asdf-format / asdf-compression

ASDF support for various compression algorithms

This packages includes a plugin for the Python library asdf to add support for reading and writing various compression algorithms including:

• Blosc
• LZ4 Frame
• Zstandard

Installation

This plugin is not yet stable and released on PyPi.

To install all compression algorithms supported by asdf-compression, install using the all optional parameters.

$ pip install "asdf-compression[all] @ git+https://github.com/asdf-format/asdf-compression"

Or alternatively by cloning and installing this repository.

$ git clone https://github.com/asdf-format/asdf-compression
$ cd asdf-compression
$ pip install ".[all]"

If only a specific algorithm is needed, just that algorithm can be installed.

Or a subset can be selected

Usage

When installed asdf-compression will register any supported and available compression algorithms with asdf using the Compressor interface.

The following example shows saving an array with zstandard compression provided via asdf-compression.

import asdf, numpy as np

af = asdf.AsdfFile({"arr": np.arange(42)})
af.set_array_compression(af["arr"], "zstd")
af.write_to("test.asdf")

Testing

pytest is used for testing. Tests can be run (from the source checkout of this repository):

$ pytest

Contributing

We welcome feedback and contributions to this project.

asdf-compression's Issues

Update URLs, publish to URLs

Currently:

extension_uri = "asdf://stsci.edu/example-project/tags/zstd-1.0.0"

and

[project.urls]
'documentation' = 'https://asdf-zstd.readthedocs.io/en/latest'
'repository' = 'https://github.com/braingram/asdf-zstd'
'tracker' = 'https://github.com/braingram/asdf-zstd/issues'

Not sure what all these URLs should ultimately be, or how to get docs to build and publish.

Python version requirement

@braingram I gave this extension a try, works perfectly, thanks! Only thing is I had to force pip to ignore the Python version requirement with --ignore-requires-python, since I'm running Python 3.8 on Ubuntu 20.04. Is there any particular reason why a given plugin requires e.g. Python 3.9 vs 3.8? I know the latest asdf core library in git main has been bumped to Python 3.9, but this plugin can presumably work with older versions of asdf, can it not? Or is the idea that it's best to restrict new extensions to whatever version of asdf is the current one?

Also, is there any distinction between "extension" and "plugin"? The README.rst uses the term "plugin" :)

Improve blosc decompression performance

As pointed out in this comment the blosc decompression if possible should use the decompress_ptr.

Blosc compression and compatibility with existing files

There are 2 known examples of blosc compression support for asdf files:

• https://github.com/abacusorg/abacusutils/blob/main/abacusnbody/data/asdf.py
• https://github.com/eschnett/asdf-cxx/blob/579539046f6ef44cde7093ca95bdaab60d9dd4b6/ndarray.cpp#L215C23-L215C28

Investigate if these are compatible and if they are compatible with asdf-compression.

asdf-cxx blosc

So far, a simple test attempting to read the example file produced for the asdf-cxx example:
https://github.com/eschnett/asdf-cxx/blob/main/demo-compression.cpp
in asdf (with asdf-compression installed) succeeds and all arrays equal the non-compressed array.
I was not able to test blosc2 as I don't have it installed (and I didn't immediately see it available through homebrew).

abacusutils blosc

Testing with the abacus test file:
https://github.com/abacusorg/abacusutils/blob/main/tests/Mini_N64_L32/halos/z0.000/field_rv_A/field_rv_A_000.asdf
and the asdf-compression code at 2cc262d
I get the following error:

error: Error 19295 : not a Blosc buffer or header info is corrupted

This is possibly due to how the abacus compressor splits up the contents into blocks:
https://github.com/abacusorg/abacusutils/blob/b76af8f302f45b7064d7929f3c1bcd9960329788/abacusnbody/data/asdf.py#L82-L88