cSize overflow in ZSTD_compressSequences about zstd

[Cyan4973]

Thanks for reporting. I'll look into it.

From what I understand, (2^64 - 1) is not cSize as "the compressed size", it's an error code which is valid.
Now it remains to understand what could trigger this error code ....

[BohuTANG]

@Cyan4973
In ZSTD_compressSequences:

{
        size_t cSize;
        size_t litSize = op_lit - op_lit_start;
        if (litSize <= LITERAL_NOENTROPY) cSize = ZSTD_noCompressBlock (op, maxDstSize, op_lit_start, litSize);
        else
        {
            cSize = ZSTD_compressLiterals(op, maxDstSize, op_lit_start, litSize);
            if (cSize == 1) cSize = ZSTD_compressRle (op, maxDstSize, op_lit_start, litSize);
            else if (cSize == 0)
            {
                if (litSize >= maxLSize) return 0;   /* block not compressible enough */
                cSize = ZSTD_noCompressBlock (op, maxDstSize, op_lit_start, litSize);
            }
        }
        if (ZSTD_isError(cSize)) return cSize;
        op += cSize;
    }

[Cyan4973]

Yes, you are right, the name of the variable is cSize.

But it's first and foremost the return value of either function ZSTD_compressLiterals(), or ZSTD_compressRle(), or ZSTD_noCompressBlock(). This return value is either a compressed length or an error code. That's why it must always be checked with ZSTD_isError().

In this case, it's an error code. Which means the called function detected a problem, and reported it.

So now, bug hunting is about knowing which function was called, and then why it detected and reported a problem (which exact test triggered it). To do that, I'll probably need a sample.

[BohuTANG]

Yes, maybe we should add error enum to measure which phase is unexpected, not the 2^64 -1.

[Cyan4973]

hi @BohuTANG

Would it be possible to get a sample which produces the error ?
Also : could you try latest update in the "dev" branch with the sample which produced the problem ?

Thanks

[BohuTANG]

I will try the dev branch.
But I am interested in why caused this error?

On Monday, July 6, 2015, Yann Collet [email protected] wrote:

hi @BohuTANG https://github.com/BohuTANG

Would it be possible to get a sample which produces the error ?
Also : could you try latest update in the "dev" branch with the sample
which produced the problem ?

Thanks

—
Reply to this email directly or view it on GitHub
https://github.com/Cyan4973/zstd/issues/37#issuecomment-118671337.

BohuTANG

"The great artist is the simplifier."--- Vincent Van Gogh

[Cyan4973]

Without access to a sample to reproduce the issue, I'm not able to tell.

[Cyan4973]

latest updates have been pushed to master, so you can also test this version if you wish

[BohuTANG]

Sure, I will have a try.

On Wed, Jul 8, 2015 at 10:50 AM, Yann Collet [email protected]
wrote:

latest updates have been pushed to master, so you can also test this
version if you wish

—
Reply to this email directly or view it on GitHub
https://github.com/Cyan4973/zstd/issues/37#issuecomment-119404894.

BohuTANG

"The great artist is the simplifier."--- Vincent Van Gogh