senier / cobrafuzz

Coverage guided fuzz testing for cobra^H^H^H^H^Hpython

CobraFuzz: Parallel coverage-guided fuzz testing for Python

CobraFuzz is a parallel coverage-guided fuzzer for testing Python programs. It is a fork of pythonfuzz.

Fuzzing for safe languages like Python is a powerful strategy for finding bugs like unhandled exceptions, logic bugs and security bugs. It makes a good addition to classic unit-testing.

Usage

Fuzz Target

The first step is to implement a function to be fuzz-tested, also called a fuzz target. Here is an example of a simple fuzz target for the built-in html module

from cobrafuzz.main import CobraFuzz


@CobraFuzz
def fuzz(buf: bytes) -> None:
    from html.parser import HTMLParser

    try:
        parser = HTMLParser()
        parser.feed(buf.decode("ascii"))
    except UnicodeDecodeError:
        pass


if __name__ == "__main__":
    fuzz()

Features of a fuzz target:

• fuzz() will call the fuzz target in an infinite loop with random data generated by the coverage-guided algorithm. The data is passed to the target in a separate process through the buf parameter.
• The function must catch and ignore any expected exceptions that arise when passing invalid input to the tested program.
• The fuzz target must call the test function / library with the passed buffer or a transformation on the test buffer if the structure is different or of different type.
• Fuzz functions can also implement application level checks to catch application or logical bugs. For example: decode the buffer with the library under test, encode it again, and check that both results are equal. To communicate the result, the fuzz target should throw an exception.
• CobraFuzz will report any unhandled exceptions as crashes.

Running

The next step is to download CobraFuzz and then run the fuzzer:

$ pip install cobrafuzz
[...]

$ python examples/fuzz_htmlparser/fuzz.py --crash-dir results fuzz --max-crashes 1
#0 READ units: 1 workers: 7 seeds: 1
#1 NEW     cov: 279 corp: 1 exec/s: 89 crashes: 0
#5 NEW     cov: 366 corp: 2 exec/s: 299 crashes: 0
#7 NEW     cov: 401 corp: 3 exec/s: 3124 crashes: 0
[...]
#154382 NEW     cov: 1086 corp: 50 exec/s: 22835 crashes: 0
#156521 NEW     cov: 1092 corp: 51 exec/s: 2797 crashes: 0
Crash dir created (results)
sample was written to results/crash-c76ab601df7d8513560ad3c1a38f43c715122b342637a4517a32d13dea03d3ce
sample = 3c215b2119
Found 1 crashes, stopping.

By default, CobraFuzz will use all CPUs available in the system (8 in this example: 1 coordination process and 7 independent workers). Use the -j / --num-workers command line parameter to override the default. In this example an unhandled exception is found in a few seconds.

Corpus

CobraFuzz will generate and test various inputs in an infinite loop.

Seeds can be provided as a mix of files and directories by the seeds parameter. If one of those seeds parameters is a directory, all regular files therein are used as seeds. CobraFuzz can also start with an empty seed corpus, though some valid test-cases in the seed corpus may speed up the fuzzing substantially.

More fuzz target examples (for real and popular libraries) can be found in the examples directory.

Credits & Acknowledgments

CobraFuzz is a fork of pythonfuzz. pythonfuzz is a port of fuzzitdev/jsfuzz. jsfuzz is in turn heavily based on go-fuzz originally developed by Dmitry Vyukov. go-fuzz is in turn heavily based on Michal Zalewski's AFL.

Contributions

Contributions are welcome, feel free to open issues and pull requests on GitHub.

Trophies

• python built-in HTMLParser - unhandled exception (pythonfuzz), twice
• beautifulsoup (pythonfuzz)

Feel free to add bugs that you found with CobraFuzz to this list via pull requests.

cobrafuzz's Issues

Fix example in README

The example in the README leads to an exception:

  File "/tmp/.venv/lib/python3.11/site-packages/cobrafuzz/fuzzer.py", line 187, in _worker_run                                                                
    target(bytes(data))                                                                                                                                                     
  File "/tmp/test.py", line 10, in fuzz                                                                                                                       
    parser = HTMLParser()                                                                                                                                                   
             ^^^^^^^^^^                                                                                                                                                     
NameError: name 'HTMLParser' is not defined

Here is a fixed version:

from cobrafuzz.main import CobraFuzz


@CobraFuzz
def fuzz(buf: bytes) -> None:
    from html.parser import HTMLParser

    try:
        string = buf.decode("ascii")
        parser = HTMLParser()
        parser.feed(string)
    except UnicodeDecodeError:
        pass


if __name__ == "__main__":
    fuzz()

Apart from moving the import inside the function, I have also added type hints. That makes it more obvious what the type of buf is.

Fuzzing with timeout

Allow for setting a timeout after which to give up the fuzzing attempt.

Improve console output

• Make hexdump output on found crashes more obvious

Improve simplification

Currently only a single line is removed during simplification. As a result, in situations where a whole block could be removed, but removal of a single line makes such a block invalid, simplification is not as effective as it could be.

Change line removal such that a range of lines is removed.

Continue fuzzing when issue was found

The fuzzer currently exits once a crash was found. Add a mode of operation where it continues fuzzing.

Re-add memory limit and deadlock detection

Idea: Create a shared memory area for each worker (shared with the coordination process). Before each execution, the worker stores the current binary into that shared memory area. The coordination tasks periodically check the liveness of each worker. When a configurable amount of time has passed without a status update, the worker is killed, the binary stored in the respective shared memory area stored as timeout and the worker is restarted.

TODO: Find a way to detect a memory exhaustion and handle it in a similar way (assuming the process gets killed when memory is exhausted). E.g. analyzing the return code for fatal error signals might be a solution.

Handle crash results according crash location

Currently, when an example in the corpus exists that will likely trigger a crash, it will be modified, stored back into the corpus and likely trigger another crash at the same location. The more examples of the same type of issue end up in the corpus, the more likely it is to produce the same issue over and over again.

Solutions:

• do not store subsequent occurrences in the corpus
• alternatively: store them, but make selecting every different kind of examples equally likely
• calculate a stable identifier from the crash location and put all examples under a corresponding directory in the crash directory

Update: May be as easy as checking the coverage for error in the coordinator process and only add / distribute a crash artifact iff the corresponding path has not been found in the coverage database.

Remove use of internal random functions

cobrafuzz/pythonfuzz/corpus.py

Line 10 in d0bb6b7

from random import _randbelow

Review README, add CI banner

Reduce found crash input

Try to shrink the input leading to a crash:

1. Get shortest successful crash (if multiple)
2. Apply a subset of mutations (those reducing size)
3. If mutation is shorter and the same crash still happens, use the mutated input, otherwise pick new mutation

Strategies for mutations:

• Remove a random line
• Remove random sequence of characters
• Tokenize input and consistently simplify identical tokens

Number of runs calculation seems off

The number of runs calculation on a 256 core systems seems much lower than expected, while the fuzzing results seem in accordance with the number of cores. Investigate.

Parallel simplification

Run simplification in parallel.

Parallel fuzzing

Utilize multiple CPU cores for fuzzing.

Parallel fuzzing requires refactoring of coverage collection. Right now, coverage is collected in the child process and the new coverage is sent over a pipe shared between parent and child. The parent then compares the new coverage with the coverage previously stored (by the parent) and in case it increased, stores the binary returned by the child process.

The current approach has a number of limitations:

• Coverage is recorded in the child process only. In case of multiple child processes, a sample may be considered to lead to new paths, while it just lead to a new path in a specific child process.
• Binaries are needlessly sent back from child to parent, while the parent needs to keep the sample anyways to handle cases where the child process hangs or times out
• The information returned is either a number (the current coverage) or the text of an exception. Ideally, the format should be more structured (e.g. a JSON document).

Design:

Child processes

The tracer is changed such that it can be reset.

• Read the next job from the job queue
• Reset the tracer
• Put a status message into the result queue before execution the target
• Run the target
• If the target ran successfully, put report message into result queue
• If the target crashed, put an error reported into the result queue

Status report

class Status:
    worker: int
    job: int

Coverage report

class Report:
    worker: int
    job: int
    covered: list[tuple[str, str, int]]

Error report

class Error:
    worker: int
    job: int
    message: str

Parent process

• The parent spawns and starts a configurable number of child processes and passes their worker ID, a command queue and a result queue
• An object for each child process contains:
  • child process
• In a loop, the parent
  • generates a new binary and associate it with a unique job ID
  • puts it into the job database under that id with status submitted and no worker id
  • submit it to the shared job queue
  • checks for response in the response queue
    • if response is an error: retrieve binary with corresponding job ID and store it into crash folder
    • if response is a status: update job with worker and timestamp in database
    • if response is a report: if provided coverage information increases total coverage store binary in samples
    • delete job from database
  • Check time stamps in job database
    • if timestamp of a submitted job is older than timeout, kill and restart the corresponding worker

Job

class Job:
   id: int
   data: bytes

Implement power schedule

Power schedule can be applied to the corpus (which input is selected for mutations) and on the mutation functions used (which mutations have been used):

• length of seed
• execution time
• coverage increases

Pass steps parameter to simplifier

• Extend CLI
• Extend Simp class

Load crash coverage from crash dir on startup

To allow seamless restart of a fuzzing campaign, obtain coverage data for all samples found in the crash dir (if any) and add them to the coverage database without reporting them as a crash.

Refactor Fuzzer.__init__ to have fewer arguments

Detect ignored exceptions

Exceptions in the __del__ method are ignored. Detect the respective warning message on stdout and treat the sample as a crash.

Make crashes directory configurable

Currently, either an exact artifacts path (storing a single file) is configured or the default directory crashes is used for storing crash results. Rework this to allow for providing an alternative directory (also required for #11).

Use cryptographic randomness

Currently the fuzzer often creates bit-identical crashes. This should be next to impossible with proper randomness. Check all random sources to be truly random.

Additional stop criteria

• time of last new path
• time of last crash

Investigate more mutation strategies

• Splicing (#23)
• Power schedule (#24)
• Create binary/text mode with specific mutations
• Investigate other mutations strategies

Include py.typed marker in package

Mypy cannot currently analyze the library as it's not marked as containing type information.

Add subcommand to prune invalid crash files

Run over all files in crash dir and delete those that do not produce an error.

Support sys.monitoring

https://docs.python.org/3/library/sys.monitoring.html

Implement more examples

• https://github.com/kjd/idna
• https://github.com/dateutil/dateutil

Replace statically set multiprocessing method

Done here:

cobrafuzz/pythonfuzz/fuzzer.py

Line 10 in d0bb6b7

mp.set_start_method('fork')

This conflicts with fuzzing targets that use a different method. Investigate alternatives.

Implement splicing

Combing multiple parts of the corpus into a new test input.

Cleanup of directory and existing state handling

The fuzzer does not seem to load previous state from the fuzzing directory. Nor does it load state from previous crashes directory.

Clean up the directory handling such that

• An optional state directory file is passed to the fuzzer as separate argument
• Known paths are loaded from state file on startup
• If a state file was configured, the state is regularly saved to that file
• The crashes directory is passed to the fuzzer as separate argument

Ref. #13
Related: #25

Remove --artifact-name

I don't have an application for this parameter. Remove.