Stack buffer overflow in v043 about zstd

[Cyan4973]

Thanks for reporting @drach. I'll look into it.

There is also a chance that the problem lies into fullbench rather than zstd.
Since you provide the sample to reproduce the issue, investigation should find it.

[Cyan4973]

I believe to have fixed this issue within latest update of "dev" branch 695a6cb.

The issue was within fullbench, zstd library looks unaffected (by this bug).
fullbench improperly allocated space for the ZSTD_decodeSeqHeaders test, which was extended by one cell in recent versions of FSE.

Since the issue is not specific to your sample, I'm surprised it has not show up earlier within Continuous Integration tests, as fullbench is tested with asan enabled. I suspect FSE dynamically downsizes reference sample statistics as they are generated from a small sample. Side effect : it doesn't test the limits of internal buffers.

If you witness some instability issues outside of fullbench, it may be related to something else.

[drach]

Indeed, fullbench is fixed in "dev", thank you.

If you witness some instability issues outside of fullbench, it may be related to something else.

Agree, will check it closer.

[drach]

In an existing quite a big iOS app the invocation of ZSTD_decompress (exactly this function, not its internals) may just crash after several dozens of successful calls, it looks like a problem in a corrupted stack.

I was wrong. We call ZSTD_decompress in different threads, some of them are set up with a 128Kb stack size that turns to be small for a ZSTD context. I was careless to notice it at first because the same code and data worked normally with LZ4 and Zlib's DEFLATE.

[Cyan4973]

Thanks for feedback @drach.

Which reminds me : the decompression context is quite large currently, and it's part of my todo list to tame it.

In the meantime, should you need to keep your stack at 128K per thread, I suggest creating a context per thread with ZSTD_createDCtx(), and use ZSTD_decompressDCtx() (which is not yet exposed unfortunately, but present within zstd_decompress.c). It will avoid usage of the stack.