zstd crashes on decoding invalid archives about zstd

[Cyan4973]

Correct. (and excellent debugging by the way)
Zstd is not yet protected against invalid input.
This is an objective, obviously.

Part of the work is done, since FSE seems to properly handle invalid input.
Now, Zstd proper must receive the same attention.

[geeknik]

I found a variation of this crash that is most likely exploitable by people with malicious intent. Let me know where I can send the details, or I can publish the details here.

[Cyan4973]

Hi Brian
It's just a bit too soon.
Zstd is not yet protected against such scenario, it's that simple.
Investigating specific situations will be useful later on, when protection code will be settled.
I'll keep this issue opened to track progresses.

[Cyan4973]

#22 contains some interesting testing code initiated by @luben

[Cyan4973]

Latest version of Zstd is now supposed to be safe against invalid input.

[magv]

Not completely though. Here's another example file:

00000000  fd 2f b5 1e 00 00 31 40  00 20 31 32 32 32 32 32
00000010  32 31 33 31 32 33 31 31  31 32 32 32 32 32 33 31
00000020  32 32 31 31 32 32 32 32  32 0a 03 00 20 00 00 00
00000030  00 a1 00 10 00 08 42 10  c0 00 00

If you'll run zstd -d on it (as of commit 79d3b1e), it will segfault with this backtrace:

#0  0x000000000040c9ac in FSE_read64 (memPtr=0x70101103b) at fse.c:202
#1  0x000000000040c8f5 in FSE_readLE64 (memPtr=0x70101103b) at fse.c:277
#2  0x000000000040474b in FSE_readLEST (memPtr=0x70101103b) at fse.c:311
#3  0x0000000000404a1e in FSE_reloadDStream (bitD=0x7fffffffe470) at fse.c:1594
#4  0x0000000000404a7c in FSE_initDState (DStatePtr=0x7fffffffe490, bitD=0x7fffffffe470, dt=0x801006000) at fse.c:1604
#5  0x000000000040a5d9 in ZSTD_decompressSequences (ctx=0x801006000, dst=0x801032000, maxDstSize=524288, seqStart=0x801011023, seqSize=14, litStart=0x801011003 "1222222131231112222231221122222\n\003", litSize=32) at ../lib/zstd.c:1543
#6  0x000000000040a387 in ZSTD_decompressBlock (ctx=0x801006000, dst=0x801032000, maxDstSize=524288, src=0x801011000, srcSize=14) at ../lib/zstd.c:1591
#7  0x000000000040a1df in ZSTD_decompressContinue (dctx=0x801006000, dst=0x801032000, maxDstSize=524288, src=0x801011000, srcSize=49) at ../lib/zstd.c:1734
#8  0x00000000004167d5 in FIO_decompressFilename (output_filename=0x4181d6 "-", input_filename=0x7fffffffec50 "--file-path-here--") at fileio.c:362
#9  0x00000000004174ee in main (argc=3, argv=0x7fffffffe998) at zstdcli.c:334

BTW, I'm using this fuzzer to find these crashes; I fully recommend you to use it too.

[Cyan4973]

Thanks @magv, this is exactly the kind of feedback I'm looking for right now.

Thanks also for the link, I did not know this tool, and I'll certainly have a look at it !

[Cyan4973]

This specific error fixed into fee8e24

[Cyan4973]

Thanks for the link. AFL is really an excellent fuzzer tool.
It managed to find 2 more bugs that internal fuzzer tools did not triggered yet.
What's more, it did it without knowing the internal data format, just "guessing" the edges, and progressing along promising test cases. This is truly impressive.

After a few hours, it did not found any new bug, so I guess we are starting to see a relatively robust release ...

[magv]

Yeah, AFL is magical. High yield with low effort? Yes, please.

I also see no crashes in the most recent commit, which is great, but I'd like to leave the fuzzer running for a while longer (a few more days would be good, I think).