Snappy doesn't release the GIL about fastparquet

[martindurant]

I agree, although at some point, snappy should be released and new conda packages built.

[esc]

No, last I checked this was not the case, unfortunately. Might be worth asking @FrancescAlted though.

[FrancescAlted]

Yes, Robert McLeod recently contributed a way to release the GIL during Blosc operation. We still need to do a new public release though.

[mrocklin]

To be clear the question is also asking "Can I use python-blosc to snappy-compress data, such that other people with only snappy (not blosc) can decompress it?"

[martindurant]

Another useful feature not provided by python-snappy is a counterpart to GzipFile, so that larger data-blocks could be decompressed in a streaming fashion, without materializing the before and after bytestrings.

[FrancescAlted]

@mrocklin The answer for the question about Snappy being able to decompress a Blosc frame with Snappy codec is no. The reason is that Blosc adds some meta-information at the beginning of every frame, and hence the pure codecs will not be able to interpret that.

[mrocklin]

Yeah, my question isn't whether or not Snappy can decompress Blosc, but rather if the Blosc Python library provides hooks to call Snappy directly, without invoking the Blosc machinery.

[FrancescAlted]

That could be possible, but not implemented yet.

…

Sent from my phone El 24 des 2016, a les 16:01, Matthew Rocklin <[email protected]> va escriure:

Yeah, my question isn't whether or not Snappy can decompress Blosc, but rather if the Blosc Python library provides hooks to call Snappy directly, without invoking the Blosc machinery. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[martindurant]

There seems to be an open PR to release the GIL: intake/python-snappy#38

[makmanalp]

This can perhaps be closed now that the above PR is merged.