out of bounds stack read in function HUF_readStats on malformed input about zstd

[Cyan4973]

Thanks Hanno, this is exactly the kind of report which is needed right now. This issue requires a pretty impressive set of conditions to trigger (it must successfully pass a lot of prior checks to get there).

I'm a bit surprised because I ran afl for almost 24h before release, without finding any issue.
I initially thought it was a matter of randomness. Digging deeper though, I note that when introducing your test case into the initial sef of afl, afl just runs normally, it doesn't find any crash.

Impressively enough, I also note that this bug pass completely unnoticed by valgrind memory tester, which was my tool of choice up to now to grab memory issues. Only clang + address sanitizer can catch it. So I'll reconsider its usage into my CI environment.

I suspect this might be because the faulty operation happens a few bytes beyond the limit of the intended table, but still within the limit of the local stack (there are some other tables after this one).

Anyway, bottom line : only address sanitizer seems able to catch this issue.
I therefore tried to combine afl with address sanitizer (which comes with its own set of difficulties, due to virtual addressing), but afl stops during initialization phase, while trying to decode zstd-oob-stack-HUF_readStats. So it doesn't even start fuzzing.

So the question is : how did you find this issue ?

[hannob]

Okay, a bit of background:
It doesn't surprise me that valgrind can't find these issues, because that is virtually impossible, it's just a valid access to another valid memory address. A stack overflow can't be found by any runtime tool, because the bare assembly code has no knowledge of what is valid and what not, a tool needs to change the memory layout at compile time to be able to detect these issues.

The "secret" of running afl+asan combined is that asan requires lots (terabytes) of virtual memory, so this only works if you disable the afl memory limit (-m none). Also see my tutorial:
https://fuzzing-project.org/tutorial3.html
There are more advanced tricks to make afl+asan work together using cgroups, but I never bothered to test them because disabling the memory limit works just fine for me (although this carries a small risk of making your system unstable if the fuzzer finds inputs that will make your tested app run wild and allocate lots of memory).

[Cyan4973]

Thanks for the link, this is a great read.

Indeed, using -m none is what I attempted to make afl work with asan, but another condition is that none of the initial seed files should make asan crash, otherwise fuzzing doesn't even start.

After that condition is fulfilled, fuzzing does proceed accordingly, just quite a bit slower due to asan checks. I'm just curious about what happens when asan detects a problem later on : does that make afl quit immediately, or does it just add the sample to the crashes directory as usual ?

[Cyan4973]

The issue should be fixed into latest update of the "dev" branch

[hannob]

afl doesn't treat asan-crashes different from other crashes. So it will just collect crashes in the [outdir]/crashes directory. You'll usually get a couple of duplicates for the issues it finds.

[hannob]

also can confirm fix works, now testing with the new dev branch to see if it finds anything more.