sixlabors / zlibstream Goto Github PK

Hello there,

I am trying to use this library, as SIMD enabled managed zlib implementation is the way I want to go for a hobby project.

So I am trying a roundtrip with an input text file containing 200 'a' characters.

With this code:

    private static void Main(string[] args)
    {
        {
            using FileStream inStream = File.OpenRead(@"D:\test\CrashTest\in.txt");
            using FileStream deflateOutStream = File.OpenWrite(@"D:\test\CrashTest\out.comp");

            ZlibOutputStream zlib = new ZlibOutputStream(deflateOutStream, CompressionLevel.Level6);
            inStream.CopyTo(zlib);
        }

        {
            using FileStream inStream = File.OpenRead(@"D:\test\CrashTest\out.comp");
            using FileStream inflateOutStream = File.OpenWrite(@"D:\test\CrashTest\out2.txt");

            ZlibInputStream zlib = new ZlibInputStream(inStream);
            zlib.CopyTo(inflateOutStream);
        }
    }

the Inflate is always running into Exception thrown here:

with various errors/ZStream states: ZBUFERROR with message "inflating: " for the above file
for another longer file I get ZDATAERROR with message "inflating: incomplete literal/length tree"

Am I using the library correctly?

By the way, I am using Net 5.0 on Windows 10.

Compression result for the Canterbury Corpus are disappointing. We always appear to be a few percentage points off the other libraries once we hit level 6, except for kennedy.xls which compresses far better than the alternatives. For lower compression levels we compare very well.

https://github.com/SixLabors/ZlibStream/blob/196c4730ba637a445e840ed7cfe67297e77b47af/benchmarks.md

According to the Squash Benchmark

• Zlib-ng achieved a compression ratio of 3.09 for cp.html at level 6, we only achieve 2.99.
• Zlib-ng achieved a compression ratio of 5.05 for kennedy.xls at level 6, we achieve 5.5.

I had a go at porting the deflate_slow method from there but that dramatically reduced compression in our sparse benchmarks to levels matching compression level 3. I haven't ported across deflate_medium yet to experiment (deflate_quick is currently broken and disabled via compiler conditionals).

@nmoinvaz I'd love to have your insight as to the cause of the difference if you have time.

APIs that take Span/Memory would be immensely useful.
They'd likely look like the System UTF8 methods, which both take a source and target, reporting how many bytes were written and if there is any left. This could work especially well with a separate method finding the minimum / maximum amount of bytes that could be produced based on the amount of source bytes. (I have no real knowledge about ZLib so excuse if this simply is impossible or the estimates would be so off, it'd be useless.)

I don't know when it happened as the coverage wasn't good enough to catch it but the round-trip tests now fail due to error introduced during optimization.

I'm not sure whether the ZlibInputStream has two constructors with the name "output" and one with the name "input" as the first parameter intentionally or whether this is a mistake, but it's very unclear when to use ZlibInputStream and when to use ZlibOutputStream, and when to use what constructor.
Some clarification in the documentation would be very helpful 😄