blosc / c-blosc2

A fast, compressed, persistent binary data store library for C.

Blosc: A blocking, shuffling and lossless compression library

Author	Contact	URL
Blosc Development Team	[email protected]	https://www.blosc.org

Gitter	GH Actions	NumFOCUS	Code of Conduct

What is it?

Blosc is a high performance compressor optimized for binary data. It has been designed to transmit data to the processor cache faster than the traditional, non-compressed, direct memory fetch approach via a memcpy() OS call. Blosc is the first compressor (that I'm aware of) that is meant not only to reduce the size of large datasets on-disk or in-memory, but also to accelerate memory-bound computations.

It uses the blocking technique so as to reduce activity in the memory bus as much as possible. In short, this technique works by dividing datasets in blocks that are small enough to fit in caches of modern processors and perform compression / decompression there. It also leverages, if available, SIMD instructions (SSE2, AVX2) and multi-threading capabilities of CPUs, in order to accelerate the compression / decompression process to a maximum.

See some benchmarks about Blosc performance.

Blosc is distributed using the BSD license, see LICENSE.txt for details.

Meta-compression and other differences over existing compressors

C-Blosc is not like other compressors: it should rather be called a meta-compressor. This is so because it can use different compressors and filters (programs that generally improve compression ratio). At any rate, it can also be called a compressor because it happens that it already comes with several compressor and filters, so it can actually work like a regular codec.

Currently C-Blosc comes with support of BloscLZ, a compressor heavily based on FastLZ (https://ariya.github.io/FastLZ/), LZ4 and LZ4HC (http://www.lz4.org/), Snappy (https://google.github.io/snappy/), Zlib (https://zlib.net/) and Zstandard (https://facebook.github.io/zstd/).

C-Blosc also comes with highly optimized (they can use SSE2 or AVX2 instructions, if available) shuffle and bitshuffle filters (for info on how and why shuffling works see here). However, additional compressors or filters may be added in the future.

Blosc is in charge of coordinating the different compressor and filters so that they can leverage the blocking technique as well as multi-threaded execution (if several cores are available) automatically. That makes that every codec and filter will work at very high speeds, even if it was not initially designed for doing blocking or multi-threading.

Finally, C-Blosc is specially suited to deal with binary data because it can take advantage of the type size meta-information for improved compression ratio by using the integrated shuffle and bitshuffle filters.

When taken together, all these features set Blosc apart from other compression libraries.

Compiling the Blosc library

Blosc can be built, tested and installed using CMake_. The following procedure describes the "out of source" build.

  $ cd c-blosc
  $ mkdir build
  $ cd build

Now run CMake configuration and optionally specify the installation directory (e.g. '/usr' or '/usr/local'):

  $ cmake -DCMAKE_INSTALL_PREFIX=your_install_prefix_directory ..

CMake allows to configure Blosc in many different ways, like preferring internal or external sources for compressors or enabling/disabling them. Please note that configuration can also be performed using UI tools provided by CMake (ccmake or cmake-gui):

  $ ccmake ..      # run a curses-based interface
  $ cmake-gui ..   # run a graphical interface

Build, test and install Blosc:

  $ cmake --build .
  $ ctest
  $ cmake --build . --target install

The static and dynamic version of the Blosc library, together with header files, will be installed into the specified CMAKE_INSTALL_PREFIX.

Codec support with CMake

C-Blosc comes with full sources for LZ4, LZ4HC, Snappy, Zlib and Zstd and in general, you should not worry about not having (or CMake not finding) the libraries in your system because by default the included sources will be automatically compiled and included in the C-Blosc library. This effectively means that you can be confident in having a complete support for all the codecs in all the Blosc deployments (unless you are explicitly excluding support for some of them).

But in case you want to force Blosc to use external codec libraries instead of the included sources, you can do that:

  $ cmake -DPREFER_EXTERNAL_ZSTD=ON ..

You can also disable support for some compression libraries:

  $ cmake -DDEACTIVATE_SNAPPY=ON ..  # in case you don't have a C++ compiler

Examples

In the examples/ directory you can find hints on how to use Blosc inside your app.

Supported platforms

Blosc is meant to support all platforms where a C89 compliant C compiler can be found. The ones that are mostly tested are Intel (Linux, Mac OSX and Windows) and ARM (Linux), but exotic ones as IBM Blue Gene Q embedded "A2" processor are reported to work too.

Mac OSX troubleshooting

If you run into compilation troubles when using Mac OSX, please make sure that you have installed the command line developer tools. You can always install them with:

  $ xcode-select --install

Wrapper for Python

Blosc has an official wrapper for Python. See:

https://github.com/Blosc/python-blosc

Command line interface and serialization format for Blosc

Blosc can be used from command line by using Bloscpack. See:

https://github.com/Blosc/bloscpack

Filter for HDF5

For those who want to use Blosc as a filter in the HDF5 library, there is a sample implementation in the hdf5-blosc project in:

https://github.com/Blosc/hdf5-blosc

Mailing list

There is an official mailing list for Blosc at:

[email protected] https://groups.google.com/g/blosc

Acknowledgments

See THANKS.rst.

---

Enjoy data!

c-blosc2's Issues

Changing the number of threads during appends blocks.

The program below shows the problem. When it starts with a number of threads larger than 1 it can not go back to a another number of threads larger than 1. However it can cycle correctly between the original number of threads and 1, and back.

`/*
Copyright (C) 2015 Francesc Alted
http://blosc.org
License: BSD (see LICENSE.txt)

Example program demonstrating use of the delta filter from C code.

To compile this program:

$ gcc -O delta_schunk_ex.c -o delta_schunk_ex -lblosc

To run:

$ ./delta_schunk_ex
Blosc version info: 2.0.0a4.dev ($Date:: 2016-08-04 #$)
Compression ratio: 762.9 MB -> 7.6 MB (100.7x)
Compression time: 0.222 s, 3437.4 MB/s
Decompression time: 0.162 s, 4714.4 MB/s
Successful roundtrip!

*/

#include <stdio.h>
#include <assert.h>
#include <context.h>
#include "blosc.h"

#define KB 1024.
#define MB (1024KB)
#define GB (1024MB)

#define CHUNKSIZE (200 * 1000)
#define NCHUNKS 500
#define NTHREADS 4

int main() {
static int64_t data[CHUNKSIZE];
static int64_t data_dest[CHUNKSIZE];
const size_t isize = CHUNKSIZE * sizeof(int64_t);
int dsize = 0;
int64_t nbytes, cbytes;
blosc2_cparams cparams = BLOSC_CPARAMS_DEFAULTS;
blosc2_dparams dparams = BLOSC_DPARAMS_DEFAULTS;
blosc2_schunk* schunk;
int i;
int j = 0;
int nchunk;
size_t nchunks;
blosc_timestamp_t last, current;
double ttotal;

printf("Blosc version info: %s (%s)\n",
BLOSC_VERSION_STRING, BLOSC_VERSION_DATE);

/* Initialize the Blosc compressor */
blosc_init();

/* Create a super-chunk container */
cparams.typesize = 8;
cparams.filters[0] = BLOSC_DELTA;
cparams.compcode = BLOSC_BLOSCLZ;
cparams.clevel = 9;
cparams.nthreads = NTHREADS;
dparams.nthreads = NTHREADS;
schunk = blosc2_new_schunk(cparams, dparams);

struct blosc2_context_s * cctx = schunk->cctx;
blosc_set_timestamp(&last);
for (nchunk = 1; nchunk <= NCHUNKS; nchunk++) {
for (i = 0; i < CHUNKSIZE; i++) {
data[i] = i * (int64_t)nchunk;
}
if (j % 2 != 0) {
cctx->nthreads = 1; // if different from 1 the program blocks
} else {
cctx->nthreads = NTHREADS;
}
nchunks = blosc2_append_buffer(schunk, isize, data);
printf("Compressed");
j++;
assert(nchunks == nchunk);
}
/* Gather some info */
nbytes = schunk->nbytes;
cbytes = schunk->cbytes;
blosc_set_timestamp(&current);
ttotal = blosc_elapsed_secs(last, current);
printf("Compression ratio: %.1f MB -> %.1f MB (%.1fx)\n",
nbytes / MB, cbytes / MB, (1. * nbytes) / cbytes);
printf("Compression time: %.3g s, %.1f MB/s\n",
ttotal, nbytes / (ttotal * MB));

/* Retrieve and decompress the chunks (0-based count) */
blosc_set_timestamp(&last);
for (nchunk = NCHUNKS-1; nchunk >= 0; nchunk--) {
dsize = blosc2_decompress_chunk(schunk, (size_t)nchunk,
(void *)data_dest, isize);
}
if (dsize < 0) {
printf("Decompression error. Error code: %d\n", dsize);
return dsize;
}
blosc_set_timestamp(&current);
ttotal = blosc_elapsed_secs(last, current);
printf("Decompression time: %.3g s, %.1f MB/s\n",
ttotal, nbytes / (ttotal * MB));

/* Check integrity of the first chunk */
for (i = 0; i < CHUNKSIZE; i++) {
if (data_dest[i] != (uint64_t)i) {
printf("Decompressed data differs from original %d, %zd!\n",
i, data_dest[i]);
return -1;
}
}

printf("Successful roundtrip!\n");

/* Free resources */
blosc2_destroy_schunk(schunk);

return 0;
}`

Does not work for 64bit buffer

FAQ claims blosc2 is meant to support 64bit buffer, but I note that blosc_compress returns an int type.

Prefilter failure

The test_prefilter is failing on Windows. The main reason is this line: https://github.com/Blosc/c-blosc2/blob/master/blosc/blosc.c#L2753

We must make sure that pparams has been allocated prior to deallocate. The best would be to always alloc on creation and copy the pparams passed in e.g. stack.

Add lz4 to the minimum set of codecs

Right now, only the blosclz codec is mandatory, and all the other codecs are optionals. It would be nice to add at least lz4 (and lz4hc, because they are the same library) to the list of mandatory codecs.

Add a blosc2_schunk_getitem()

This will allow to retrieve different items from inside a super-chunk. The API blosc2_schunk_getitem() is not written in stone.

Compress/Decompress routines should return Errnos

When an error happens, the API leaves the user wondering why. I suggest that blosc_compress, blosc2_compress_ctx, blosc_decompress, blosc2_decompress_ctx, blosc_getitem, and blosc2_getitem_ctx should all return -1 on error and set errno. Alternatively, Blosc could define its own error codes with negative values.

This would be especially useful for the decompress routines, which can return an error either because the destination buffer is too small, the data is corrupted, or the decompression library is unavailable.

Add checksums to chunks

For safety and for early discovery of data corruption, one might be interesting to add (optional) checksums for chunks (and maybe for frames too).

Libraries like xxHash or an SSSE3 version of adler32 might be good fits for this.

Add blosc2_decompress_safe()

Following the discussion in #40, it could be handy to have a new call for safer decompression that might have this signature:

int blosc2_decompress_safe(blosc2_context* context, const void* src, size_t *srcsize, void* dest, size_t *destsize);

On input, srcsize and destsize will have maximum number of bytes for reading (srcsize) or writing (destsize), whereas on output, these will contain the actual number of bytes read (srcsize) and written (destsize).

Essentially, the implementation would be something like:

int blosc2_decompress_safe(blosc2_context* context, const void* src, size_t* srcsize, void* dest, size_t* destsize);
  int result;
  size_t nbytes, cbytes, blocksize;

  blosc_cbuffer_sizes(src, &nbytes,  &cbytes, &blocksize);
  if (*srcsize < cbytes) {
    *srcsize = BLOSC_EXTENDED_HEADER_LENGTH;
    *destsize = 0;
    return -11;  // means that srcsize is too short
   }
  if (*destsize < nbytes) {
    *srcsize = BLOSC_EXTENDED_HEADER_LENGTH;
    *destsize = 0;
    return -12;  // means that destsize is too short
   }

  result = blosc2_decompress_ctx(context, src, dest, destsize);
  if (result > 0) {
    *srcsize = cbytes;
    *destsize = nbytes;
  }
  return result;
}

missing timestamp.h included by blosc.h in the INSTALLed version

As the title says. I would suggest that that #include be relocated from blosc.h to blosc.c since it seems to be used internally only.

blosc_getitem return value seems inconsistent (DeepState fuzzing)

Whether I assert that it is the #bytes of the #items, some tests seem to violate the claim. Is this known/expected?

	      unsigned start_item = DeepState_UIntInRange(0, num_elements-1);
	      unsigned num_items = DeepState_UIntInRange(0, num_elements-start_item);
	      LOG(TRACE) << "Getting " << num_items << " from " << start_item;
	      int get_result = blosc_getitem(intermediate, start_item, num_items, items);
	      //ASSERT_EQ(get_result, num_items * type_size) <<
	      //"Getting " << num_items << " from " << start_item << " expected: " << num_items * type_size << ": got " << get_result;

If I allow either:

             ASSERT((get_result == num_items) || (get_result == (num_items * type_size))) <<
                "Getting " << num_items << " from " << start_item << " with size " << type_size << ": got " << get_result;

it never fails, but that's not a consistent API. I can minimize a failing example, if you can clarify which is the expected return value.

Optimize the number of reallocs in frame to schunk

By using chunksize instead of BLOSC_MIN_HEADER_LENGTH in

c-blosc2/blosc/frame.c

Line 699 in cc66023

data_chunk = malloc((size_t)prev_alloc);

we could minimize the number of buffer reallocations. However, chunksize might be zero in some cases (variable length chunks in schunks), so be carefull when evaluating a solution here.

Support for thread-safe operation

Currently, all the operation in super-chunks are making use of blosc_compress() and blosc_decompress(). For applications that are multithreaded, one must rather use blosc_compress_ctx() and blosc_decompress_ctx().

An idea is to pass a new field to the blosc2_sparams struct. The new field could be tentatively called thread_safety. The default could be 1 (i.e. thread safe, so the _ctx() API is going to be used), but the user can select a 0 for speed.

Add support for Intel compiler (icc)

The support existed in Blosc1, but somehow it got broken in Blosc2.

Support ALTIVEC vector instruction on IBM POWER.

This issue the the continuation of:
Blosc/c-blosc#267

As stated there, SSE2 instructions from Intel can be transparently mapped into ALTIVEC starting with GCC-8. I made a quick demonstrator which is 40 to 50% faster (bitshuffle-lz4) compared to the master on a power9. The roadmap is the following:

• Add the -DNO_WARN_X86_INTRINSICS compiler flag
• Check the compiler version and processor version
• Update the tests to validate this path like all the others

Consider creating a package with Spack

Spack looks like a good fit for creating and distributing binaries for c-blosc2. Worth a try.

Add `blosc_compress_ctx()` and `blosc_decompress_ctx()` for Blosc1 compatibility

The initial plan was to deprecate blosc_compress_ctx() and blosc_decompress_ctx() in favor of the newer blosc2_compress_ctx() and blosc2_decompress_ctx(). However, as there is no deprecation warning in docs of the original blosc_compress_ctx() and blosc_decompress_ctx() for Blosc1, we should provide an implementation for these (probably as thin wrappers of blosc2_compress_ctx() and blosc2_decompress_ctx() ) for Blosc2.

Metainfo type size not correct? (DeepState fuzzing)

I just started fuzzing C-Blosc2 with DeepState. The harness is here: https://github.com/agroce/deepstate-c-blosc2

A few odd things are coming up immediately. No round-trip failures yet, but oddities, maybe API misunderstandings.

    size_t typesize;
    int flags;
    blosc_cbuffer_metainfo(intermediate, &typesize, &flags);
    LOG(TRACE) << "type size = " << typesize;
    //ASSERT_EQ(typesize, type_size) << "type size = " << type_size << " but meta claims " << typesize;

The returned typesize seems to always be 1 in runs, whatever it was when the call to compress was made.

Out-of-order appends

Right now, the order of appending chunks in super-chunks sets the order of reading chunks. It would be nice if a new:
blosc2_schunk_set_buffer(blosc2_schunk *schunk, void *src, size_t nbytes, int pos)

call with the pos parameter specifying the order for reading. A pos=-1 would mean 'at the end', that is, equivalent to:

blosc2_schunk_append_buffer(blosc2_schunk *schunk, void *src, size_t nbytes) .

This would imply to create a new metalayer with this new order information.

Add a blosc2_frame_append_buffer

This should mimic the blosc2_schunk_append_buffer() for super-chunks.

Problem while building cblosc2 on ARM64

Hi,
I am testing blosc2 on ARM64, actually the Pine64. Not a super powerful board but it is 64bits and has neon SIMD and this is what I am interested in.
https://www.pine64.org/devices/single-board-computers/pine-a64/
The operating system is debian10 hence kernel 4.19 and gcc8.3

Here is the error message, it is likely the expectations for following the standard is slightly different from one architecture to another ... but I am not that fluent in C:

kieffer@pine64:/data/workspace/c-blosc2$ cd build/
kieffer@pine64:/data/workspace/c-blosc2/build$ rm -rf *    
kieffer@pine64:/data/workspace/c-blosc2/build$ cmake ..
-- The C compiler identification is GNU 8.3.0
-- The CXX compiler identification is GNU 8.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
Configuring for Blosc version: 2.0.0-beta.4.dev
-- Using LZ4 internal sources.
-- Using LIZARD internal sources.
-- Using MINIZ internal sources for ZLIB support.
-- Using ZSTD internal sources.
-- No IPP libraries found.
-- Not using IPP accelerated compression.
-- No build type specified. Defaulting to 'Release'.
-- Building for system processor aarch64
-- Adding run-time support for NEON
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
Skipping test_shuffle_roundtrip_altivec on non-ALTIVEC builds
Skipping test_shuffle_roundtrip_avx2 on non-AVX2 builds
Skipping test_shuffle_roundtrip_sse2 on non-SSE2 builds
-- Configuring done
CMake Warning (dev) at blosc/CMakeLists.txt:258 (add_library):
  Policy CMP0063 is not set: Honor visibility properties for all target
  types.  Run "cmake --help-policy CMP0063" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  Target "blosc2_static" of type "STATIC_LIBRARY" has the following
  visibility properties set for C:

    C_VISIBILITY_PRESET

  For compatibility CMake is not honoring them for this target.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Generating done
-- Build files have been written to: /data/workspace/c-blosc2/build
kieffer@pine64:/data/workspace/c-blosc2/build$ make
Scanning dependencies of target blosc_shared_testing
[  1%] Building C object blosc/CMakeFiles/blosc_shared_testing.dir/blosc2.c.o
In file included from /data/workspace/c-blosc2/blosc/blosc2.c:18:
/data/workspace/c-blosc2/blosc/blosc2.h:1162:34: warning: 'struct timespec' declared inside parameter list will not be visible outside of this definition or declaration
 #define blosc_timestamp_t struct timespec
                                  ^~~~~~~~
/data/workspace/c-blosc2/blosc/blosc2.h:1168:39: note: in expansion of macro 'blosc_timestamp_t'
 BLOSC_EXPORT void blosc_set_timestamp(blosc_timestamp_t* timestamp);
                                       ^~~~~~~~~~~~~~~~~
/data/workspace/c-blosc2/blosc/blosc2.h:1162:34: warning: 'struct timespec' declared inside parameter list will not be visible outside of this definition or declaration
 #define blosc_timestamp_t struct timespec
                                  ^~~~~~~~
/data/workspace/c-blosc2/blosc/blosc2.h:1173:41: note: in expansion of macro 'blosc_timestamp_t'
 BLOSC_EXPORT double blosc_elapsed_nsecs(blosc_timestamp_t start_time,
                                         ^~~~~~~~~~~~~~~~~
/data/workspace/c-blosc2/blosc/blosc2.h:1162:34: warning: 'struct timespec' declared inside parameter list will not be visible outside of this definition or declaration
 #define blosc_timestamp_t struct timespec
                                  ^~~~~~~~
/data/workspace/c-blosc2/blosc/blosc2.h:1179:40: note: in expansion of macro 'blosc_timestamp_t'
 BLOSC_EXPORT double blosc_elapsed_secs(blosc_timestamp_t start_time,
                                        ^~~~~~~~~~~~~~~~~
In file included from /data/workspace/c-blosc2/blosc/blosc2.c:51:
/data/workspace/c-blosc2/internal-complibs/miniz-2.0.8/miniz.c:3031:9: note: #pragma message: Using fopen, ftello, fseeko, stat() etc. path for file I/O - this path may not support large files.
 #pragma message("Using fopen, ftello, fseeko, stat() etc. path for file I/O - this path may not support large files.")
         ^~~~~~~
/data/workspace/c-blosc2/blosc/blosc2.c: In function 'blosc_compress_context':
/data/workspace/c-blosc2/blosc/blosc2.c:1592:23: warning: passing argument 1 of 'blosc_set_timestamp' from incompatible pointer type [-Wincompatible-pointer-types]
   blosc_set_timestamp(&last);
                       ^~~~~
In file included from /data/workspace/c-blosc2/blosc/blosc2.c:18:
/data/workspace/c-blosc2/blosc/blosc2.h:1168:58: note: expected 'struct timespec *' but argument is of type 'struct timespec *'
 BLOSC_EXPORT void blosc_set_timestamp(blosc_timestamp_t* timestamp);
                                                          ^
/data/workspace/c-blosc2/blosc/blosc2.c:1644:25: warning: passing argument 1 of 'blosc_set_timestamp' from incompatible pointer type [-Wincompatible-pointer-types]
     blosc_set_timestamp(&current);
                         ^~~~~~~~
In file included from /data/workspace/c-blosc2/blosc/blosc2.c:18:
/data/workspace/c-blosc2/blosc/blosc2.h:1168:58: note: expected 'struct timespec *' but argument is of type 'struct timespec *'
 BLOSC_EXPORT void blosc_set_timestamp(blosc_timestamp_t* timestamp);
                                                          ^
/data/workspace/c-blosc2/blosc/blosc2.c:1645:39: error: type of formal parameter 1 is incomplete
     double ctime = blosc_elapsed_secs(last, current);
                                       ^~~~
/data/workspace/c-blosc2/blosc/blosc2.c:1645:45: error: type of formal parameter 2 is incomplete
     double ctime = blosc_elapsed_secs(last, current);
                                             ^~~~~~~
/data/workspace/c-blosc2/blosc/blosc2.c: In function 'blosc_get_complib_info':
/data/workspace/c-blosc2/blosc/blosc2.c:2592:14: warning: implicit declaration of function 'strdup'; did you mean 'strcmp'? [-Wimplicit-function-declaration]
   *complib = strdup(clibname);
              ^~~~~~
              strcmp
/data/workspace/c-blosc2/blosc/blosc2.c:2592:12: warning: assignment to 'char *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
   *complib = strdup(clibname);
            ^
/data/workspace/c-blosc2/blosc/blosc2.c:2593:12: warning: assignment to 'char *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
   *version = strdup(clibversion);
            ^
make[2]: *** [blosc/CMakeFiles/blosc_shared_testing.dir/build.make:63: blosc/CMakeFiles/blosc_shared_testing.dir/blosc2.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:127: blosc/CMakeFiles/blosc_shared_testing.dir/all] Error 2
make: *** [Makefile:163: all] Error 2

Add a system-meta layer

Right now, the only way to store variable length metadata in a blosc2 container is to use the usermeta section, but the user has full access to it, so it can mess with that information. It would nice to add another section (let's call it system-meta) where it can be used by applications on top of blosc2 to store critical information that should be protected (i.e. the user cannot delete/modify).

Rename `nspace`to `metalayer`

The intent of the current name-spaces is to add different layers for storing meta-information so, we would need to do a global renaming of the nspace -> metalayer.

Make author more comprehensive

There are many contributions to Blosc codebase, so we must replace "Francesc Alted" by "Blosc Development Team" all throughout the different Blosc projects.

blosc2_decompress_ctx can overrun src

blosc2_decompress_ctx does not take the size of the source buffer as an argument. Instead, it assumes that the buffer is complete and correctly formed. That can result in array overruns for buffers that are truncated, malicious or otherwise corrupt. It should take the source buffer length as an argument, and never read beyond that.

Rename sparse -> copy argument in blosc2_schunk_from_frame

copy is a much better name for users in this case.

Pre-filter to add padding to typesizes that are not multiple of accelerated ones

This way, one can leverage the SIMD accelerated codepaths for the shuffle filter. Based on a suggestion by @krzsysFR: https://github.com/Cyan4973/zstd/issues/256#issuecomment-235586023

Add support for LGTM

LGTM is a CI-friendly analizer for security that might be useful.

Error with the file blosc-common.h

When Blosc is installed on the system, it seems that the blosc-common.h file needs to be installed with the following files:

c-blosc2/blosc/CMakeLists.txt

Line 248 in edcc44e

install(FILES blosc.h blosc-export.h DESTINATION include COMPONENT DEV)

The library and header need to be 'blosc2', no just 'blosc'

It turns out that there are many distributions that are carrying versions of libblosc.so and blosc.h of the blosc1 library, and even though blosc2 is compatible to these, for new blosc2 deployments, it is nearly impossible to override all the possible places where the blosc1 header and library are. We definitely need new names like libblosc2.so and blosc2.h.

Check simple codecs: wflz and pithy

wflz and pithy are very simple codecs that could be added to Blosc2 without too much overhead. They are also very fast decompressing (possibly the most important parameter in Blosc).

Add support for post-filters

This is a user-provided function and would mimic the current pre-filter implementation. This will be interesting for supporting e.g. encryption (via pre-filters) and decryption (post-filters), or on-the-flight computation based on data in the same chunk, or generic inputs that can be populated by the user in the postfilter struct that will be passed to the post-filter.

Implement filters on ppc64le with actual Altivec-SIMD instructions

For now, the code is just auto-translated from SSE2. Here are some hints on how to do it properly:

For byte-shuffle it looks straights forwards:
http://mirror.informatimago.com/next/developer.apple.com/hardware/ve/algorithms.html#byte_swap

For bit-shuffle, Initial explanation in:
https://mischasan.wordpress.com/2011/07/24/what-is-sse-good-for-transposing-a-bit-matrix/
But I did not find (yet) an Altivec implementation

Any examples for implementing command line interface that accept file or stream as input/output?

For example, the original Zstd compressor doesn't meet my requirement regarding the speed, and I need a "Blosc-accelerated" Zstd which works the same way as the original Zstd does, i.e., accepting files for compression or decompression.

Support for zfp codec

This seems to be a nice lossy floating point codec: https://github.com/LLNL/zfp

Support for `file` command line utility for disk-based frames

It would be nice to provide support for the file command line utility in Unix platforms. As an example, it is cool to see the amount of metainfo that is provided for .jpg files:

$ file ~/WWW.YTS.AG.jpg
/home/francesc/WWW.YTS.AG.jpg: JPEG image data, JFIF standard 1.01, resolution (DPI), density 96x96, segment length 16, Exif Standard: [TIFF image data, big-endian, direntries=4, xresolution=62, yresolution=70, resolutionunit=2, software=paint.net 4.0.6], baseline, precision 8, 350x500, frames 3

Add support for buldkite.com

Adding support for buildkite as another CI would be handy because it allows to use on-premise machines, potentially speeding-up the time to comple builds, but also to setup pipelines with more complex dependencies and analyzers.

Add a BLOSC_MAX_CODECS in blosc.h

This will specify the maximum number of codecs that can be available in C-Blosc2 simultaneously.

DOSHUFFLE bit in metainfo seems wrong (DeepState fuzzing)

DOSHUFFLE metainfo bit seems wrong:

RACE: Initialized test input buffer with data from `doshuffle.fail`
TRACE: Running: CBlosc2_RoundTrip from TestCBlosc2.cpp(34)
TRACE: TestCBlosc2.cpp(45): Performing 1 round trips.
TRACE: TestCBlosc2.cpp(48): *******************************   Starting run #0   *******************************
TRACE: TestCBlosc2.cpp(50): Type size = 1
TRACE: TestCBlosc2.cpp(52): Number of elements = 1
TRACE: TestCBlosc2.cpp(54): Buffer alignment = 32
TRACE: TestCBlosc2.cpp(56): Compression level = 0
TRACE: TestCBlosc2.cpp(58): Do shuffle = 1
TRACE: TestCBlosc2.cpp(61): Setting compressor to blosclz
TRACE: TestCBlosc2.cpp(65): Setting delta to 0
TRACE: TestCBlosc2.cpp(69): Buffer size = 1
TRACE: TestCBlosc2.cpp(89): # uncompressed bytes = 1
TRACE: TestCBlosc2.cpp(91): # compressed bytes = 33
TRACE: TestCBlosc2.cpp(92): block size = 1
TRACE: TestCBlosc2.cpp(97): type size = 1
TRACE: TestCBlosc2.cpp(99): DOSHUFFLE: 0
CRITICAL: TestCBlosc2.cpp(100): do shuffle = 0 but set to 1
ERROR: Failed: CBlosc2_RoundTrip
ERROR: Test case doshuffle.fail failed

Memory overwrite in test_maxout

This is what valgrind reports:

$ valgrind tests/test_maxout
==363== Memcheck, a memory error detector
==363== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==363== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==363== Command: tests/test_maxout
==363== 
==363== Invalid write of size 8
==363==    at 0x4C326CB: memcpy@@GLIBC_2.14 (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==363==    by 0x4E52427: memcpy (string3.h:53)
==363==    by 0x4E52427: blosc_compress_context (blosc.c:1524)
==363==    by 0x4E541F5: blosc_compress (blosc.c:1784)
==363==    by 0x400FE9: test_maxout_equal (test_maxout.c:41)
==363==    by 0x400FE9: all_tests (test_maxout.c:70)
==363==    by 0x400FE9: main (test_maxout.c:100)
==363==  Address 0x56df458 is 0 bytes after a block of size 1,016 alloc'd
==363==    at 0x4C2FFC6: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==363==    by 0x4C300D1: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==363==    by 0x401277: blosc_test_malloc.constprop.1 (test_common.h:73)
==363==    by 0x400D4D: main (test_maxout.c:91)

Support for endianness

So far (c-blosc2 beta.3), all the data stored in chunks does not provide information about its endianness.

PR #84 allows to store this information, but making use of it for restoring the correct endianness (e.g. when writing a frame on a big-endian machine and read it on a little-endian) has still to be implemented yet. In fact, this is not trivial, as the Blosc2 layer does not have info about the types (it is an int of 4 bytes or a str of 4 bytes?), so probably endianness restoring should be implemented on a different layer that has info about the types.

[STY] Come with a convention for creation of objects

Now there is different calls with create, new or init names on them. Come with a convention for that.

Use msgpack for the chunks section in frame format

Packing the chunks section of the frame format would bring more encapsulation and better maneagibility in the end. With that, fields like uncompressed_size and compressed_size and blocksize can be moved into the new chunks section.

Suggestion: add a chunks_size field to the new chunks so that computing the size of the chunk for offsets would be easier, and specially, faster (e.g. remove the need for get_trailer_offsets() in frame_get_chunk()).

After doing this, I am not sure on whether we should add another msgpck field at the front of the header for specifying that three other msgpack sections come later (header, chunks and trailer). Probably this is not a good idea because then utilities like msgpack2json may want to read the whole file, and this can be too much for large files.

cmake -DPREFER_EXTERNAL_ZSTD=ON does not work

See for example: facebook/zstd#975 (comment)

Error using metalayers

@FrancescAlted, When I tried to use the metalayers in blosc I get an error.

The example below consists of:

• A schunk based on a frame is created.
• A metalayer is added.
• The schunk is filled using blosc2_schunk_append_buffer.

If the metalayers lines are commented it works well. But if the metalayer is added, when it tries to append a buffer, blosc fails.

#include <blosc2.h>

int main() {

    blosc2_frame *frame = blosc2_new_frame("metalayer.blosc2");

    blosc2_schunk *sc = blosc2_new_schunk(BLOSC_CPARAMS_DEFAULTS, BLOSC_DPARAMS_DEFAULTS, frame);

    char name[] = "metalayer";
    uint64_t content = 1234;
    uint32_t content_size = sizeof(uint64_t);
    blosc2_frame_add_metalayer(sc->frame, name, (uint8_t *) &content, content_size);

    uint64_t *content2;
    uint32_t content_size2;
    blosc2_frame_get_metalayer(sc->frame, name, (uint8_t **) &content2, &content_size2);

    printf("Content %llu\n", *content2);

    size_t part_size = 100 * 100 * sc->typesize;

    uint8_t *part = malloc(part_size);

    for (int i = 0; i < 2; ++i) {
        int err = blosc2_schunk_append_buffer(sc, part, part_size);
        printf("Error %d\n", err);
    }

    free(part);

    return 0;
}

Allow the metalayers to be in-memory frames

The metalayers are currently backed by plain chunks, and that means a limitation of 31-bit in the size of the metalayers. By using an in-memory frame we would also get a contiguous buffer to be stored (similar to a chunk), and we would get rid of 1) the 31-bit limit and 2) the need to have large data chunks in-memory prior to be compressed into a chunk.

Wrong data returned from getitem (DeepState fuzzing)

This is a minimal test case; making any value smaller causes it to pass.

TRACE: TestCBlosc2.cpp(37): Performing 1 round trips.
TRACE: TestCBlosc2.cpp(40): *******************************   Starting run #0   *******************************
TRACE: TestCBlosc2.cpp(55): Compression task: type_size: 131 ; num_elements: 1 ; buffer_alignment: 32; compression_level: 6; do_shuffle: 0; compressor: blosclz; delta: 1; buffer_size: 131
TRACE: TestCBlosc2.cpp(69): original[0] = 0
TRACE: TestCBlosc2.cpp(69): original[1] = 0
TRACE: TestCBlosc2.cpp(69): original[2] = 0
TRACE: TestCBlosc2.cpp(69): original[3] = 0
TRACE: TestCBlosc2.cpp(69): original[4] = 0
TRACE: TestCBlosc2.cpp(69): original[5] = 0
TRACE: TestCBlosc2.cpp(69): original[6] = 0
TRACE: TestCBlosc2.cpp(69): original[7] = 0
TRACE: TestCBlosc2.cpp(69): original[8] = 0
TRACE: TestCBlosc2.cpp(69): original[9] = 0
TRACE: TestCBlosc2.cpp(69): original[10] = 0
TRACE: TestCBlosc2.cpp(69): original[11] = 0
TRACE: TestCBlosc2.cpp(69): original[12] = 0
TRACE: TestCBlosc2.cpp(69): original[13] = 0
TRACE: TestCBlosc2.cpp(69): original[14] = 0
TRACE: TestCBlosc2.cpp(69): original[15] = 0
TRACE: TestCBlosc2.cpp(69): original[16] = 0
TRACE: TestCBlosc2.cpp(69): original[17] = 0
TRACE: TestCBlosc2.cpp(69): original[18] = 0
TRACE: TestCBlosc2.cpp(69): original[19] = 0
TRACE: TestCBlosc2.cpp(69): original[20] = 0
TRACE: TestCBlosc2.cpp(69): original[21] = 0
TRACE: TestCBlosc2.cpp(69): original[22] = 0
TRACE: TestCBlosc2.cpp(69): original[23] = 0
TRACE: TestCBlosc2.cpp(69): original[24] = 0
TRACE: TestCBlosc2.cpp(69): original[25] = 0
TRACE: TestCBlosc2.cpp(69): original[26] = 0
TRACE: TestCBlosc2.cpp(69): original[27] = 0
TRACE: TestCBlosc2.cpp(69): original[28] = 0
TRACE: TestCBlosc2.cpp(69): original[29] = 0
TRACE: TestCBlosc2.cpp(69): original[30] = 0
TRACE: TestCBlosc2.cpp(69): original[31] = 0
TRACE: TestCBlosc2.cpp(69): original[32] = 0
TRACE: TestCBlosc2.cpp(69): original[33] = 0
TRACE: TestCBlosc2.cpp(69): original[34] = 0
TRACE: TestCBlosc2.cpp(69): original[35] = 0
TRACE: TestCBlosc2.cpp(69): original[36] = 0
TRACE: TestCBlosc2.cpp(69): original[37] = 0
TRACE: TestCBlosc2.cpp(69): original[38] = 0
TRACE: TestCBlosc2.cpp(69): original[39] = 0
TRACE: TestCBlosc2.cpp(69): original[40] = 0
TRACE: TestCBlosc2.cpp(69): original[41] = 0
TRACE: TestCBlosc2.cpp(69): original[42] = 0
TRACE: TestCBlosc2.cpp(69): original[43] = 0
TRACE: TestCBlosc2.cpp(69): original[44] = 0
TRACE: TestCBlosc2.cpp(69): original[45] = 0
TRACE: TestCBlosc2.cpp(69): original[46] = 0
TRACE: TestCBlosc2.cpp(69): original[47] = 0
TRACE: TestCBlosc2.cpp(69): original[48] = 0
TRACE: TestCBlosc2.cpp(69): original[49] = 0
TRACE: TestCBlosc2.cpp(69): original[50] = 0
TRACE: TestCBlosc2.cpp(69): original[51] = 0
TRACE: TestCBlosc2.cpp(69): original[52] = 0
TRACE: TestCBlosc2.cpp(69): original[53] = 0
TRACE: TestCBlosc2.cpp(69): original[54] = 0
TRACE: TestCBlosc2.cpp(69): original[55] = 0
TRACE: TestCBlosc2.cpp(69): original[56] = 0
TRACE: TestCBlosc2.cpp(69): original[57] = 0
TRACE: TestCBlosc2.cpp(69): original[58] = 0
TRACE: TestCBlosc2.cpp(69): original[59] = 0
TRACE: TestCBlosc2.cpp(69): original[60] = 0
TRACE: TestCBlosc2.cpp(69): original[61] = 0
TRACE: TestCBlosc2.cpp(69): original[62] = 0
TRACE: TestCBlosc2.cpp(69): original[63] = 0
TRACE: TestCBlosc2.cpp(69): original[64] = 0
TRACE: TestCBlosc2.cpp(69): original[65] = 0
TRACE: TestCBlosc2.cpp(69): original[66] = 0
TRACE: TestCBlosc2.cpp(69): original[67] = 0
TRACE: TestCBlosc2.cpp(69): original[68] = 0
TRACE: TestCBlosc2.cpp(69): original[69] = 0
TRACE: TestCBlosc2.cpp(69): original[70] = 0
TRACE: TestCBlosc2.cpp(69): original[71] = 0
TRACE: TestCBlosc2.cpp(69): original[72] = 0
TRACE: TestCBlosc2.cpp(69): original[73] = 0
TRACE: TestCBlosc2.cpp(69): original[74] = 0
TRACE: TestCBlosc2.cpp(69): original[75] = 0
TRACE: TestCBlosc2.cpp(69): original[76] = 0
TRACE: TestCBlosc2.cpp(69): original[77] = 0
TRACE: TestCBlosc2.cpp(69): original[78] = 0
TRACE: TestCBlosc2.cpp(69): original[79] = 0
TRACE: TestCBlosc2.cpp(69): original[80] = 0
TRACE: TestCBlosc2.cpp(69): original[81] = 0
TRACE: TestCBlosc2.cpp(69): original[82] = 0
TRACE: TestCBlosc2.cpp(69): original[83] = 0
TRACE: TestCBlosc2.cpp(69): original[84] = 0
TRACE: TestCBlosc2.cpp(69): original[85] = 0
TRACE: TestCBlosc2.cpp(69): original[86] = 0
TRACE: TestCBlosc2.cpp(69): original[87] = 0
TRACE: TestCBlosc2.cpp(69): original[88] = 0
TRACE: TestCBlosc2.cpp(69): original[89] = 0
TRACE: TestCBlosc2.cpp(69): original[90] = 0
TRACE: TestCBlosc2.cpp(69): original[91] = 0
TRACE: TestCBlosc2.cpp(69): original[92] = 0
TRACE: TestCBlosc2.cpp(69): original[93] = 0
TRACE: TestCBlosc2.cpp(69): original[94] = 0
TRACE: TestCBlosc2.cpp(69): original[95] = 0
TRACE: TestCBlosc2.cpp(69): original[96] = 0
TRACE: TestCBlosc2.cpp(69): original[97] = 0
TRACE: TestCBlosc2.cpp(69): original[98] = 0
TRACE: TestCBlosc2.cpp(69): original[99] = 0
TRACE: TestCBlosc2.cpp(69): original[100] = 0
TRACE: TestCBlosc2.cpp(69): original[101] = 0
TRACE: TestCBlosc2.cpp(69): original[102] = 0
TRACE: TestCBlosc2.cpp(69): original[103] = 0
TRACE: TestCBlosc2.cpp(69): original[104] = 0
TRACE: TestCBlosc2.cpp(69): original[105] = 0
TRACE: TestCBlosc2.cpp(69): original[106] = 0
TRACE: TestCBlosc2.cpp(69): original[107] = 0
TRACE: TestCBlosc2.cpp(69): original[108] = 0
TRACE: TestCBlosc2.cpp(69): original[109] = 0
TRACE: TestCBlosc2.cpp(69): original[110] = 0
TRACE: TestCBlosc2.cpp(69): original[111] = 0
TRACE: TestCBlosc2.cpp(69): original[112] = 0
TRACE: TestCBlosc2.cpp(69): original[113] = 0
TRACE: TestCBlosc2.cpp(69): original[114] = 0
TRACE: TestCBlosc2.cpp(69): original[115] = 0
TRACE: TestCBlosc2.cpp(69): original[116] = 0
TRACE: TestCBlosc2.cpp(69): original[117] = 0
TRACE: TestCBlosc2.cpp(69): original[118] = 0
TRACE: TestCBlosc2.cpp(69): original[119] = 0
TRACE: TestCBlosc2.cpp(69): original[120] = 0
TRACE: TestCBlosc2.cpp(69): original[121] = 0
TRACE: TestCBlosc2.cpp(69): original[122] = 0
TRACE: TestCBlosc2.cpp(69): original[123] = 0
TRACE: TestCBlosc2.cpp(69): original[124] = 0
TRACE: TestCBlosc2.cpp(69): original[125] = 0
TRACE: TestCBlosc2.cpp(69): original[126] = 0
TRACE: TestCBlosc2.cpp(69): original[127] = 0
TRACE: TestCBlosc2.cpp(69): original[128] = 1
TRACE: TestCBlosc2.cpp(69): original[129] = 0
TRACE: TestCBlosc2.cpp(69): original[130] = 0
TRACE: TestCBlosc2.cpp(81): # uncompressed bytes = 131
TRACE: TestCBlosc2.cpp(83): # compressed bytes = 34
TRACE: TestCBlosc2.cpp(84): block size = 131
TRACE: TestCBlosc2.cpp(90): typesize: 131; DOSHUFFLE: 0; DOBITSHUFFLE: 0; DODELTA: 8; MEMCPYED: 0
TRACE: TestCBlosc2.cpp(98): compressor: BloscLZ
TRACE: TestCBlosc2.cpp(111): Performing 1 non-buffer-changing actions.
TRACE: TestCBlosc2.cpp(128): Getting 1 from 0
TRACE: TestCBlosc2.cpp(139): [0][0]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][1]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][2]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][3]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][4]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][5]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][6]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][7]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][8]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][9]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][10]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][11]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][12]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][13]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][14]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][15]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][16]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][17]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][18]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][19]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][20]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][21]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][22]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][23]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][24]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][25]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][26]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][27]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][28]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][29]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][30]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][31]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][32]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][33]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][34]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][35]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][36]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][37]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][38]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][39]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][40]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][41]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][42]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][43]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][44]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][45]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][46]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][47]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][48]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][49]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][50]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][51]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][52]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][53]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][54]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][55]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][56]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][57]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][58]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][59]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][60]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][61]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][62]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][63]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][64]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][65]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][66]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][67]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][68]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][69]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][70]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][71]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][72]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][73]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][74]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][75]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][76]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][77]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][78]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][79]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][80]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][81]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][82]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][83]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][84]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][85]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][86]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][87]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][88]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][89]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][90]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][91]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][92]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][93]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][94]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][95]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][96]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][97]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][98]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][99]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][100]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][101]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][102]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][103]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][104]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][105]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][106]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][107]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][108]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][109]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][110]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][111]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][112]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][113]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][114]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][115]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][116]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][117]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][118]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][119]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][120]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][121]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][122]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][123]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][124]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][125]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][126]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][127]: original: 0; items: 0
TRACE: TestCBlosc2.cpp(139): [0][128]: original: 1; items: 1
TRACE: TestCBlosc2.cpp(137): !!! MISMATCH !!!:
TRACE: TestCBlosc2.cpp(139): [0][129]: original: 0; items: 1
TRACE: TestCBlosc2.cpp(139): [0][130]: original: 0; items: 0
CRITICAL: TestCBlosc2.cpp(141): Get returned wrong data for item 0 (getting 1 from 0)
ERROR: Failed: CBlosc2_RoundTrip
ERROR: Test case wrongdata.fail failed

is blosc2 supporting 64bit buffer?

In blosc we ask for 64bit buffer support Blosc/c-blosc#67 and @FrancescAlted pointed out that it will be supported in blosc2. Is this the case or do we still have the 32bit limitation?

C-Blosc Build Gives memcpy() errors

When trying to build c-blosc in VS2015 Developer cmd prmpt, I get memcpy errors. (NOTE: I get the same error with c-blosc and c-blosc2)

cmake .. works fine, but cmake --build . gives (at end):

"C:\blosc\build\ALL_BUILD.vcxproj" (default target) (1) ->
"C:\blosc\build\bench\bench.vcxproj" (default target) (3) ->
"C:\blosc\build\blosc\blosc_shared.vcxproj" (default target) (4) ->
(ClCompile target) ->
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\vcruntime_stri
ng.h(44): error C2040: 'memcpy': 'void *(void *,const void *,std::size_t)' diff
ers in levels of indirection from 'void *(void *,const void *,std::size_t)' [C:
\blosc\build\blosc\blosc_shared.vcxproj]

"C:\blosc\build\ALL_BUILD.vcxproj" (default target) (1) ->
"C:\blosc\build\blosc\blosc_shared_testing.vcxproj" (default target) (5) ->
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\vcruntime_stri
ng.h(44): error C2040: 'memcpy': 'void *(void *,const void *,std::size_t)' diff
ers in levels of indirection from 'void *(void *,const void *,std::size_t)' [C:
\blosc\build\blosc\blosc_shared_testing.vcxproj]

"C:\blosc\build\ALL_BUILD.vcxproj" (default target) (1) ->
"C:\blosc\build\blosc\blosc_static.vcxproj" (default target) (6) ->
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\vcruntime_stri
ng.h(44): error C2040: 'memcpy': 'void *(void *,const void *,std::size_t)' diff
ers in levels of indirection from 'void *(void *,const void *,std::size_t)' [C:
\blosc\build\blosc\blosc_static.vcxproj]

132 Warning(s)
3 Error(s)

Time Elapsed 00:00:30.01

Enumerated for multiples of sizes in blosc.h

It would be handy to have an enumerated for BLOSC_KB, BLOSC_MB, BLOSC_GB... right in blosc.h because this way the user can make use of them.

Disable C++ compiler detection

Ditto in Blosc/c-blosc#284