microsoft / onefuzz

A self-hosted Fuzzing-As-A-Service platform

❗ IMPORTANT NOTICE ❗

August 31, 2023.

Since September 2020 when OneFuzz was first open sourced, we’ve been on a journey to create a best-in-class orchestrator for running fuzzers, driving security and quality into our products.

Initially launched by a small group in MSR, OneFuzz has now become a significant internal platform within Microsoft. As such, we are regretfully archiving the project to focus our attention on becoming a more deeply integrated service within the company. Unfortunately, we aren’t a large enough team to live in both the open-source world and the internal Microsoft world with its own unique set of requirements.

Our current plan is to archive the project in the next few months. That means we’ll still be making updates for a little while. Of course, even after it’s archived, you’ll still be able to fork it and make the changes you need. Once we’ve decided on a specific date for archiving, we’ll update this readme.

Thanks for taking the journey with us.

The OneFuzz team.

---

Update: September 15 2023: Our current target to archive the project is September 30th, 2023.

---

A self-hosted Fuzzing-As-A-Service platform

Project OneFuzz enables continuous developer-driven fuzzing to proactively harden software prior to release. With a single command, which can be baked into CICD, developers can launch fuzz jobs from a few virtual machines to thousands of cores.

Features

• Composable fuzzing workflows: Open source allows users to onboard their own fuzzers, swap instrumentation, and manage seed inputs.
• Built-in ensemble fuzzing: By default, fuzzers work as a team to share strengths, swapping inputs of interest between fuzzing technologies.
• Programmatic triage and result de-duplication: It provides unique flaw cases that always reproduce.
• On-demand live-debugging of found crashes: It lets you summon a live debugging session on-demand or from your build system.
• Observable and Debug-able: Transparent design allows introspection into every stage.
• Fuzz on Windows and Linux: Multi-platform by design. Fuzz using your own OS build, kernel, or nested hypervisor.
• Crash reporting notification callbacks: Including Azure DevOps Work Items and Microsoft Teams messages

For information, check out some of our guides:

• Terminology
• Getting Started
• Supported Platforms
• More documentation

Are you a Microsoft employee interested in fuzzing? Join us on Teams at Fuzzing @ Microsoft.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repositories using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Data Collection

The software may collect information about you and your use of the software and send it to Microsoft. Microsoft may use this information to provide services and improve our products and services. You may turn off the telemetry as described in the repository. There are also some features in the software that may enable you and Microsoft to collect data from users of your applications. If you use these features, you must comply with applicable law, including providing appropriate notices to users of your applications together with a copy of Microsoft's privacy statement. Our privacy statement is located at https://go.microsoft.com/fwlink/?LinkID=824704. You can learn more about data collection and use in the help documentation and our privacy statement. Your use of the software operates as your consent to these practices.

For more information:

• Onefuzz Telemetry Details

Reporting Security Issues

Security issues and bugs should be reported privately to the Microsoft Security Response Center (MSRC). For more information, please see SECURITY.md.

onefuzz's Issues

Instance wide max duration for tasks

It would be great to set an instance-wide maximum duration for tasks.

For example, on our playground instance, it would be useful to set the maximum duration of any task to 1 hour. This would reduce contention as we have more internal users playing with our playground without increasing our need fund their fuzzing efforts.

Basically: "Yes, you can get a taste, but use your own budget for actual fuzzing."

AB#39994813

Retry managed identity token acquisition attempts per docs

Managed pool VMSS nodes use a system-assigned managed identity to acquire access tokens for requests to the service.

The docs for the REST API used to acquire these tokens has specific guidance around when and how to retry (with backoff). Implement this in the token acquisition code.

Sample 'azure function' that consumes the SDK

Multiple people have discussed building a middleware from their CICD pipelines into Onefuzz using Azure Functions.

It would be useful to show an example using the Onefuzz SDK via Azure Functions.

Non-ASAN_LOG based crashes do not generate crash reports

Information

• Onefuzz version: 1.2.0
• OS: linux

Provide detailed reproduction steps (if any)

1. Deploy a go-fuzz based libfuzzer target. (See: https://github.com/microsoft/onefuzz-samples/tree/main/examples/golang)

Expected result

Crashes are identified & crash reports are generated.

Actual result

Crashes are identified but crashes are not.

Example crash output that isn't turned into a crash:

INFO: Seed: 619442538
INFO: 65536 Extra Counters
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2      INITED ft: 3 corp: 1/1b exec/s: 0 rss: 33Mb
#203    NEW    ft: 13 corp: 2/6b lim: 6 exec/s: 0 rss: 33Mb L: 5/5 MS: 1 InsertRepeatedBytes-
#641    NEW    ft: 14 corp: 3/14b lim: 8 exec/s: 0 rss: 33Mb L: 8/8 MS: 3 ShuffleBytes-CrossOver-ChangeByte-
#643    REDUCE ft: 14 corp: 3/13b lim: 8 exec/s: 0 rss: 33Mb L: 7/7 MS: 2 ChangeBit-EraseBytes-
#715    REDUCE ft: 14 corp: 3/12b lim: 8 exec/s: 0 rss: 33Mb L: 6/6 MS: 2 InsertByte-CrossOver-
#828    REDUCE ft: 15 corp: 4/20b lim: 8 exec/s: 0 rss: 33Mb L: 8/8 MS: 3 InsertByte-CrossOver-ShuffleBytes-
#829    REDUCE ft: 15 corp: 4/19b lim: 8 exec/s: 0 rss: 33Mb L: 7/7 MS: 1 CrossOver-
#940    REDUCE ft: 16 corp: 5/26b lim: 8 exec/s: 0 rss: 33Mb L: 7/7 MS: 1 ChangeBit-
#956    REDUCE ft: 16 corp: 5/24b lim: 8 exec/s: 0 rss: 33Mb L: 5/7 MS: 1 EraseBytes-
#1008   REDUCE ft: 16 corp: 5/23b lim: 8 exec/s: 0 rss: 33Mb L: 5/7 MS: 2 EraseBytes-CopyPart-
#1381   REDUCE ft: 16 corp: 5/22b lim: 11 exec/s: 0 rss: 33Mb L: 6/6 MS: 3 InsertRepeatedBytes-ChangeByte-EraseBytes-
#1444   REDUCE ft: 16 corp: 5/21b lim: 11 exec/s: 0 rss: 33Mb L: 5/5 MS: 3 ChangeByte-ChangeByte-EraseBytes-
#22884  REDUCE ft: 18 corp: 6/240b lim: 219 exec/s: 0 rss: 33Mb L: 219/219 MS: 5 CopyPart-InsertRepeatedBytes-CMP-InsertByte-CrossOver- DE: "\xff\xff\xff\xff\xff\xff\xff\xff"-
#22917  REDUCE ft: 18 corp: 6/220b lim: 219 exec/s: 0 rss: 33Mb L: 199/199 MS: 3 CopyPart-ChangeBinInt-EraseBytes-
#22934  REDUCE ft: 18 corp: 6/148b lim: 219 exec/s: 0 rss: 33Mb L: 127/127 MS: 2 CopyPart-EraseBytes-
#22950  REDUCE ft: 18 corp: 6/121b lim: 219 exec/s: 0 rss: 33Mb L: 100/100 MS: 1 EraseBytes-
#22963  REDUCE ft: 18 corp: 6/59b lim: 219 exec/s: 0 rss: 33Mb L: 38/38 MS: 3 CMP-CrossOver-CrossOver- DE: "\xff\xff\xff\xff\xff\xff\xff\xff"-
#23037  REDUCE ft: 18 corp: 6/57b lim: 219 exec/s: 0 rss: 33Mb L: 36/36 MS: 4 InsertRepeatedBytes-InsertByte-CopyPart-EraseBytes-
panic: runtime error: index out of range [50000] with length 36

goroutine 17 [running, locked to thread]:
_/home/bcaswell/projects/onefuzz/onefuzz-samples/examples/golang.MyFunc(0x1df62b0, 0x24, 0x24, 0x7fffb7208c40)
        /home/bcaswell/projects/onefuzz/onefuzz-samples/examples/golang/example.go:34 +0x371
_/home/bcaswell/projects/onefuzz/onefuzz-samples/examples/golang.Fuzz(...)
        /home/bcaswell/projects/onefuzz/onefuzz-samples/examples/golang/example.go:49
main.LLVMFuzzerTestOneInput(0x1df62b0, 0x24, 0x5bd3f8)
        _/home/bcaswell/projects/onefuzz/onefuzz-samples/examples/golang/go.fuzz.main/main.go:35 +0x84
main._cgoexpwrap_bd8ddfad68ed_LLVMFuzzerTestOneInput(0x1df62b0, 0x24, 0x1df4b10)
        _cgo_gotypes.go:64 +0x37
==883== ERROR: libFuzzer: deadly signal
    #0 0x4adfa0 in __sanitizer_print_stack_trace (/home/bcaswell/projects/onefuzz/onefuzz-samples/examples/golang/fuzz.exe+0x4adfa0)
    #1 0x45a2a8 in fuzzer::PrintStackTrace() (/home/bcaswell/projects/onefuzz/onefuzz-samples/examples/golang/fuzz.exe+0x45a2a8)
    #2 0x43f3f3 in fuzzer::Fuzzer::CrashCallback() (/home/bcaswell/projects/onefuzz/onefuzz-samples/examples/golang/fuzz.exe+0x43f3f3)
    #3 0x7fb02d85a3bf  (/lib/x86_64-linux-gnu/libpthread.so.0+0x153bf)
    #4 0x501730 in runtime.raise runtime/sys_linux_amd64.s:149

NOTE: libFuzzer has rudimentary signal handlers.
      Combine libFuzzer with AddressSanitizer or similar for better crash reports.
SUMMARY: libFuzzer: deadly signal
MS: 2 ChangeBinInt-ChangeByte-; base unit: 9ebbf1108c612f9b8de6f55528cc5b3b71543e2d
0x78,0x79,0x7a,0x30,0x0,0x0,0x0,0x0,0x0,0x0,0xfb,0xfb,0xfb,0x78,0x7d,0x7a,0xea,0xea,0x79,0xa,0x7a,0xea,0xa,0xfb,0xfb,0xfb,0xfb,0xfb,0xfb,0x5,0x4,0x5,0xea,0x79,0xa,0xfb,
xyz0\x00\x00\x00\x00\x00\x00\xfb\xfb\xfbx}z\xea\xeay\x0az\xea\x0a\xfb\xfb\xfb\xfb\xfb\xfb\x05\x04\x05\xeay\x0a\xfb
artifact_prefix='./'; Test unit written to ./crash-32880edfd1f4fa04a10b9689116008cb01620164
Base64: eHl6MAAAAAAAAPv7+3h9eurqeQp66gr7+/v7+/sFBAXqeQr7

Add a heartbeat to the supervisor agent

The supervisor should periodically send a heartbeat to the service
The service should periodically check that the agent is responsive based on the heartbeat data.
The service should automatically restart the agent when it becomes unresponsive

Deploy Fuzzing for Java Spring Boot and Python Scripts

Hi contributors, I am working at a company which mainly focuses on building Java Spring Boot applications and Python scripts. We are very interested to add fuzzing to our CICD pipelines using OneFuzz - ADO pipeline integration.

Given that, I would like to ask there are plans for adding these types of applications to OneFuzz, or is it already capable of running these types of fuzz targets?

Since I am currently new to fuzzing, kindly correct me. But, it is to my understanding the current templates (afl, libfuzz, ossfuzz, radamsa) are mainly for CLANG and do not support these types of applications yet.

Looking forward to your response.

Thank you very much.

AB#39994814

Shut down agents running tasks before deleting task queues

When tasks are stopped, the related queues are deleted immediately. This causes any agents currently using the queue to generate failures such as

[2020-10-13T22:06:54Z ERROR onefuzz_supervisor] error running supervisor agent: error sending request for url (https://HOST/messages?SAS_URL): connection error: Connection reset by peer (os error 104)
Error: error sending request for url (https://HOST/messages?SAS_URL): connection error: Connection reset by peer (os error 104)

Caused by:
    0: connection error: Connection reset by peer (os error 104)
    1: Connection reset by peer (os error 104)
[2020-10-13T22:06:54Z ERROR onefuzz_downloader] error running downloader: supervisor failed: Some(1)
Error: supervisor failed: Some(1)

While these errors are silenced at the service end as the task is in the stopping state, these look concerning to the user.

Provide the ability to create a new cli application registration

To support the scenario where the client application is required to have a specific role. We need a create a set of default app roles when we deploy the application.
We also need to be able to create a new client application registration and assign it to one of these app role.

why can't find unique reports

Information

• Onefuzz version: 1.0
• OS: ubuntu20.04

Provide detailed reproduction steps (if any)

Expected result

it should existed unique_report.json

Actual result

i can not find the unique_report.json , it just a empty array

Implement github issue management

We should support managing Github Issue lifecycles, similar to Azure Devops Work Items.

(For MSFT FTEs: see c2542533d864401f524727233a932c0e13421f66 on the pre-release main branch for proof-of-concept)

Code and syntax highlighting

How about adding code and syntax highlighting? following is an example
https://zen-austin-0c80be.netlify.app/posts/basic-markdown-syntax/#code-and-syntax-highlighting

Service level webhooks for integration

It would be useful to allow users to subscribe to events in Onefuzz. Following a model such as Github webhooks, Onefuzz could provide user-customized integrations based on any event in the system.

log analytics cost exceeds the cost of the VMs

Information

• Onefuzz version: 1.3.1
• OS: Ubuntu 20.04 docker container on macOS

Details

I followed the instructions in the Getting Started guide and it turns out that the log analytics cost exceeds the cost of the VMs after running for 24 hours as shown below:

I am not sure if this is expected, but I don't think it is great having to pay for logs much more than the computational cost itself. My rough understanding is that the agent is reporting everything in realtime which would produce a huge amount of logs to process. Would it be possible to somehow reduce the logging cost by introducing some delays between reports?

To me, realtime stats are good when we're fuzzing something that requires monitoring to improve our harnesses and corpus. However, there are cases that we need to run it for days if not weeks where we don't expect any crash to occur, just use it as some kind of a baseline. Logging too much would create some unnecessary cost.

Deployment's requirements.txt have incompatible versions

Information

• Onefuzz version: 1.3.1
• OS: Ubuntu 20.04 docker container on macOS

Provide detailed reproduction steps

1. Download and unzip https://github.com/microsoft/onefuzz/releases/download/1.3.1/onefuzz-deployment-1.3.1.zip
2. Try to install the dependencies from requirements.txt

Expected result

Install the dependencies without ignoring the incompatibilities.

Actual result

with python3 -m pip install requirements.txt

ERROR: azure-cli-core 2.13.0 has requirement azure-mgmt-resource==10.2.0, but you'll have azure-mgmt-resource 9.0.0 which is incompatible.
ERROR: azure-cli 2.10.1 has requirement azure-cli-core==2.10.1.*, but you'll have azure-cli-core 2.13.0 which is incompatible.
ERROR: azure-cli 2.10.1 has requirement azure-graphrbac~=0.60.0, but you'll have azure-graphrbac 0.61.1 which is incompatible.
ERROR: azure-cli 2.10.1 has requirement azure-mgmt-eventgrid==3.0.0rc7, but you'll have azure-mgmt-eventgrid 2.2.0 which is incompatible.
ERROR: azure-cli 2.10.1 has requirement azure-mgmt-storage~=11.1.0, but you'll have azure-mgmt-storage 11.2.0 which is incompatible.
ERROR: azure-cli 2.10.1 has requirement azure-mgmt-web~=0.47.0, but you'll have azure-mgmt-web 0.48.0 which is incompatible.
ERROR: azure-cli 2.10.1 has requirement cryptography<3.0.0,>=2.3.1, but you'll have cryptography 3.1.1 which is incompatible.

with pipenv install

ERROR: Could not find a version that matches azure-graphrbac~=0.60.0,~=0.61.1 (from -r /tmp/pipenv9_iok9glrequirements/pipenv-bboxuyxq-constraints.txt (line 11))
Tried: 0.30.0, 0.30.0, 0.31.0, 0.31.0, 0.32.0, 0.32.0, 0.33.0, 0.33.0, 0.40.0, 0.40.0, 0.50.0, 0.50.0, 0.51.0, 0.51.0, 0.51.1, 0.51.1, 0.52.0, 0.52.0, 0.53.0, 0.53.0, 0.60.0, 0.60.0, 0.61.0, 0.61.0, 0.61.1, 0.61.1
Skipped pre-versions: 0.30.0a1, 0.30.0a1, 0.30.0rc1, 0.30.0rc1, 0.30.0rc2, 0.30.0rc2, 0.30.0rc3, 0.30.0rc3, 0.30.0rc4, 0.30.0rc4, 0.30.0rc5, 0.30.0rc5, 0.30.0rc6, 0.30.0rc6
There are incompatible versions in the resolved dependencies:
  azure-graphrbac~=0.61.1 (from -r /tmp/pipenv9_iok9glrequirements/pipenv-bboxuyxq-constraints.txt (line 11))
  azure-graphrbac~=0.60.0 (from azure-cli==2.10.1->-r /tmp/pipenv9_iok9glrequirements/pipenv-bboxuyxq-constraints.txt (line 9))

Race condition handling crashes at start of supervisor fuzzing task

Information

Due to how support for AFL's ownership of the working directory is implemented, there is a race condition at the immediate start of launching afl-fuzz where we won't record crashes.

• Onefuzz version: 1.0.0
• OS: linux

AB#39994763

Add task setup container path as an environment variable when launching setup scripts

It would be useful to know the path to the setup container in the setup scripts. While this can be calculated via BASEDIR=$(dirname $0) in bash or PSScriptRoot for powershell, passing it as an environment variable would be helpful.

Perhaps use ONEFUZZ_TASK_SETUP_PATH

Move to registered secrets in keyvault for 3rd-party integrations

Today, ADO PATs are stored in the notification config. This means that each developer must provide their own PAT for their tasks.

It would be good to move to Azure Keyvault for storage of these, as well as make it such that the secrets are referable, but not exportable by notification templates.

This is related to #48

AFLplusplus Support

Great project!
As development on afl itself doesn't seem to be high priority anymore, it would be nice to see out of the box support for additional afl-based fuzzers, such as afl++.

OSError: [Errno 8] Exec format error: 'tools/linux/azcopy'

Information

• Onefuzz version: 1.2.0
• OS: macOS Catalina 10.15.5
• Python: 3.9.0
• Pip 20.2.3

Provide detailed reproduction steps (if any)

Attempting to run the deployment scripts

1. brew reinstall [email protected]
2. echo 'export PATH="/usr/local/opt/[email protected]/bin:$PATH"' >> /Users/aaron/.bash_profile
3. close and re-open terminal
4. pip3 install -r requirements.txt
5. python3 ./deploy.py uksouth fuzzit fuzzit emailaddress

Expected result

Expected result is that the deployment completes and tools successfully upload.

Actual result

MacBook-Pro:onefuzz-deployment-1.2.0 aaron$ Python3 ./deploy.py uksouth fuzzit fuzzit emailaddress
INFO:deploy:checking if RBAC already exists
INFO:deploy:Could not find the default CLI application under the current subscription, creating a new one
DEBUG:msrest.async_paging:Paging async iterator protocol is not available for ApplicationPaged
DEBUG:msrest.async_paging:Paging async iterator protocol is not available for ApplicationPaged
DEBUG:msrest.async_paging:Paging async iterator protocol is not available for ApplicationPaged
INFO:deploy:deploying arm template: azuredeploy.json
INFO:deploy:creating eventgrid destination queue
INFO:deploy:creating eventgrid subscription
INFO:deploy:uploading tools from tools
Traceback (most recent call last):
  File "/Users/aaron/Downloads/onefuzz-deployment-1.2.0/./deploy.py", line 753, in <module>
    main()
  File "/Users/aaron/Downloads/onefuzz-deployment-1.2.0/./deploy.py", line 747, in main
    state[1](client)
  File "/Users/aaron/Downloads/onefuzz-deployment-1.2.0/./deploy.py", line 478, in upload_tools
    subprocess.check_output(
  File "/usr/local/Cellar/[email protected]/3.9.0/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 420, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/local/Cellar/[email protected]/3.9.0/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 501, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/local/Cellar/[email protected]/3.9.0/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 947, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/Cellar/[email protected]/3.9.0/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 1819, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: 'tools/linux/azcopy'

Manage Scalesets within pool

Currently client has to manage all the scaling up and down by creating scalesets and managing its size. Adding it in service side will automatically enable scaling for clients w/o managing for it. The pool will be able to manage scalesets and #instances inside scalesets.

Supervisor telemetry not flushed on error

Information

• Onefuzz version: 1.0
• OS: Any

Provide detailed reproduction steps (if any)

Create a job (or task) with a target setup script that exits nonzero.

Expected result

Error log telemetry appears in App Insights.

Actual result

It does not appear. This is because we early return in the error path, without flushing error telemetry.

Move job templates to the service backend

Right now, individual developers are required to provide all of the configuration for a job, as well as notification configuration for handling work item submission, closure, etc.

It would be useful to have these centrally managed, but exposed in the CLI/SDK

AB#36005

Nested exceptions during failure of azure function deployment

Information

• Onefuzz version: 1.3.0
• OS: Windows

Provide detailed reproduction steps (if any)

1. Execute a deployment using a poor internet connection or induce an exception (e.g. kill func.exe) while subprocess "func azure functionapp publish ..." is running.

Expected result

I expect to see only the Azure.Functions.Cli.Common.CliException exception or reason displayed and not a PermissionError.

Traceback (most recent call last):
File "deploy.py", line 789, in
main()
File "deploy.py", line 783, in main
state1
File "deploy.py", line 604, in deploy_app
cwd=tmpdirname,
File "C:\Program Files\Python37\lib\subprocess.py", line 411, in check_output
**kwargs).stdout
File "C:\Program Files\Python37\lib\subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['C:\ProgramData\chocolatey\bin\func.EXE', 'azure', 'functionapp', 'publish', '', '--python', '--no-build']' returned non-zero exit status 1.

Actual result

Azure.Functions.Cli.Common.CliException: Timed out waiting for SCM to update the Environment Settings
at Azure.Functions.Cli.Actions.AzureActions.PublishFunctionAppAction.WaitForAppSettingUpdateSCM(Site functionApp, IDictionary2 shouldHaveSettings, IDictionary2 shouldNotHaveSet
tings, Int32 timeOutSeconds) in D:\a\1\s\src\Azure.Functions.Cli\Actions\AzureActions\PublishFunctionAppAction.cs:line 580
at Azure.Functions.Cli.Actions.AzureActions.PublishFunctionAppAction.PublishRunFromPackageLocal(Site functionApp, Func1 zipFileFactory) in D:\a\1\s\src\Azure.Functions.Cli\Actio ns\AzureActions\PublishFunctionAppAction.cs:line 504 at Azure.Functions.Cli.Actions.AzureActions.PublishFunctionAppAction.HandleLinuxDedicatedPublish(Site functionApp, Func1 zipStreamFactory) in D:\a\1\s\src\Azure.Functions.Cli\Ac
tions\AzureActions\PublishFunctionAppAction.cs:line 411
at Azure.Functions.Cli.Actions.AzureActions.PublishFunctionAppAction.PublishFunctionApp(Site functionApp, GitIgnoreParser ignoreParser, IDictionary`2 additionalAppSettings) in D:
\a\1\s\src\Azure.Functions.Cli\Actions\AzureActions\PublishFunctionAppAction.cs:line 319
at Azure.Functions.Cli.Actions.AzureActions.PublishFunctionAppAction.RunAsync() in D:\a\1\s\src\Azure.Functions.Cli\Actions\AzureActions\PublishFunctionAppAction.cs:line 173
at Azure.Functions.Cli.ConsoleApp.RunAsync[T](String[] args, IContainer container) in D:\a\1\s\src\Azure.Functions.Cli\ConsoleApp.cs:line 66
Traceback (most recent call last):
File "deploy.py", line 456, in deploy_app
env=dict(os.environ, CLI_DEBUG="1"),
File "C:\Users\OneFuzzDeployer\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 411, in check_output
**kwargs).stdout
File "C:\Users\OneFuzzDeployer\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['C:\Program Files\Microsoft\Azure Functions Core Tools\func.EXE', 'azure', 'functionapp', 'publish', '', '--python',
'--no-build']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "deploy.py", line 581, in
main()
File "deploy.py", line 575, in main
state1
File "deploy.py", line 459, in deploy_app
os.chdir(current_dir)
File "C:\Users\OneFuzzDeployer\AppData\Local\Programs\Python\Python37\lib\tempfile.py", line 807, in exit
self.cleanup()
File "C:\Users\OneFuzzDeployer\AppData\Local\Programs\Python\Python37\lib\tempfile.py", line 811, in cleanup
_shutil.rmtree(self.name)
File "C:\Users\OneFuzzDeployer\AppData\Local\Programs\Python\Python37\lib\shutil.py", line 516, in rmtree
return _rmtree_unsafe(path, onerror)
File "C:\Users\OneFuzzDeployer\AppData\Local\Programs\Python\Python37\lib\shutil.py", line 404, in _rmtree_unsafe
onerror(os.rmdir, path, sys.exc_info())
File "C:\Users\OneFuzzDeployer\AppData\Local\Programs\Python\Python37\lib\shutil.py", line 402, in _rmtree_unsafe
os.rmdir(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\Users\OneFuzzDeployer\AppData\Local\Temp\tmp8bdhk01z'

Configurable base directory

The runtime scripts assume a base directory of c:\onefuzz on Windows and /onefuzz on Linux. This may not be acceptable for certain environments. Since this setting (and likely others) is hardcoded in the scripts, a longer and error-prone search/replace activity maybe necessary ramp-up before actually running the tool. Having those setting configurable via a properties file instead of requiring code edits may improve the general OOBE.

Add an option to migrate the azure table data in the deployment script

The deployment script should have a an option for migrating data from 0.9 to 1.0.
The script needs to add the OS column to the Task table

Remove MSR internal work items

Provide a description of requested docs changes

Now that OneFuzz is open, we should remove vestiges of the MSR work items.

supervisor connection to storage queue is not resilient

Supervisor failed and had to be restarted when the TCP connection to the storage queue HTTP server died.

error running supervisor agent: error sending request for url (REDACTED_QUEUE_URL): connection error: An existing connection was forcibly closed by the remote host. (os error 10054)

Information

• Onefuzz version: 1.0.0
• OS: windows

support libfuzzer-sys based rust targets

The libfuzzer-sys crate, commonly exposed via cargo-fuzz provides the ability to fuzz rust code.

While the binaries "just work" for fuzzing, crash reporting generally does not. As is, non-sanitizer crashes (such as anything that aborts in Rust code), are logged to stderr while the stack trace is logged to log_path.

Enable auto-export from appinsights

By enabling continuous export from app insights, they can use existing tools to parse logs including better import into Power BI.

Unfortunately, this isn't possible via ARM templates.

This will likely need to be added to deploy.py using azure.mgmt.applicationinsights

Configurable disk size

Seems like the disk size for fuzzing (/onefuzz) is fixed. (by OS image?)

Even if we change the vm size, the temporary storage is attached to other drive such as F:\ in Windows or /mnt in Linux.

onefuzz@node000009:/onefuzz$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           797M  708K  796M   1% /run
/dev/sda1        29G  3.6G   26G  13% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda15      105M  3.6M  101M   4% /boot/efi
/dev/sdb1        74G   53M   70G   1% /mnt
tmpfs           797M     0  797M   0% /run/user/998
tmpfs           797M     0  797M   0% /run/user/1000

#8 might be a solution. I'm not sure if there's an easier way to achieve this.

AB#39994760

scalesets, pools, and nodes are not defined in our terminology

Provide a description of requested docs changes

Scaleset, pool, and nodes should be defined.

Reset unresponsive nodes based on the heartbeat

We need to reset nodes that have not received any heartbeats after some amount of time.
This is continuation of the work done in #30

On ADO notification failure, the task associated in the report should be marked as failed

There are many reasons that ADO notifications can fail, such as assigning the reports to an invalid user. We get an exception when that happens, and log it. However, the logs don't help the users fix their tasks. We should take that exception log and mark the task as failed with that exception log.

Example reported event:
ADO report failed: The identity value '[email protected]' for field 'Assigned To' is an unknown identity.

Full path of the input file

In the command replacement, {input} provides a relative path of the input file such as generator_tmp/input. But, some binaries requires a full path of the file.

I think {input} needs to be replaced to the absolute path rather than a relative path. What do you think?

Execute commands automatically in repro VMs

It would be useful to automatically execute commands in the repro VMs.

On task setup script failure, tasks should be marked as "failed" and nodes should be re-imaged

Information

• Onefuzz version: 1.0.0
• OS: Ubuntu 18.04

Provide detailed reproduction steps (if any)

1. Launch a task with a custom setup script (setup.sh) with a failure (say, 'exit 1')

Expected result

1. Task state set to failed. Stderr/stdout from the setup script should be added to task.error
2. Node is appropriately reimaged (/onefuzz/<task_id> no longer exists)

Actual result

1. Task state is scheduled
2. Verify /onefuzz/<task_id> on the node the task was assigned

handle libfuzzer targets crashing without dropping an a crashing input

When a libfuzzer target crashes in a fashion that libfuzzer can't handle, such as raise(SIGKILL), the input isn't saved to disk.

We should report this behavior as a failure to the user, so they can fix their libfuzzer target to not crash in such a fashion.

Basic logic:

if task == libfuzzer_fuzz and exit_code != 0 and get_crash() == None {
   mark task as failed
}

expose iterations per second in 'status top'

It would be beneficial to expose the 'iterations per second' and '% of coverage' for fuzzing tasks in 'status top'

AB#39994787

Deploy problem with Azure Cloud Shell

While following the Getting Started guide to deploy onefuzz , the command pip install -r requirements.txt can not work right. No matching version pyfunctional was found.

ERROR: Could not find a version that satisfies the requirement pyfunctional~=1.4.1 (from -r requirements.txt (line 12)) (from versions: 0.6.0, 0.7.0, 0.7.1, 0.8.0, 1.0.0, 1.1.0, 1.1.2, 1.1.3, 1.2.0, 1.3.0)
ERROR: No matching distribution found for pyfunctional~=1.4.1 (from -r requirements.txt (line 12))

Change the the version of pyfunctional package will work, but executing the following deploy.py will also failed :

huang@Azure:~/onefuzz$ ./deploy.py $REGION $RESOURCE_GROUP_NAME $ONEFUZZ_INSTANCE_NAME $CONTACT_EMAIL_ADDRESS
Traceback (most recent call last):
  File "./deploy.py", line 18, in <module>
    from azure.common.client_factory import get_client_from_cli_profile
ImportError: No module named 'azure.common'
huang@Azure:~/onefuzz$ pip list | grep azure.common
azure-common                          1.1.25
WARNING: You are using pip version 20.1.1; however, version 20.2.3 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

what maybe wrong? how can I fix it?

debug ssh for windows scalesets is broken

In the re-architecture to used managed pools of nodes, SSH key support on Windows VMSSs was dropped.

onefuzz/src/api-service/__app__/onefuzzlib/extension.py

Line 20 in fa25823

# TODO: figure out how to create VM specific SSH keys for Windows.

Add the ability to limit job / task display

With large numbers of concurrent tasks, it would be useful to limit the jobs / tasks that are displayed.

Possible options:

• job_id
• job project
• job project/name
• job project/name/build

investigate replacing python function code with rust

The Azure SDK for Rust continues to make large strides forwards, making it a potential avenue moving forwards.

Many, but not all, of our prerequisites are available in rust.

Currently available

• azure.storage.blob
• azure.storage.table
• azure.mgmt.compute
• azure.mgmt.resource
• azure.mgmt.storage
• azure.identity
• azure.mgmt.operationalinsights (known as azure.mgmt.loganalytics in python)

Alternatives available

• jinja2: tera, askama
• opencensus: appinsights-rs
• pyjwt: jwt - NOTE: We don't need full token validation, that's done by App Services. We only need to extract the appid & OID
• requests: reqwests - used for Teams webhook integration
• github3.py: octocrab

Currently in progress:

• azure.storage.queue - The basic functionality for iterating over a queue exists, but managing queue creation does not yet.

Missing

• azure.mgmt.network - required for creating networks used by scalesets (implemented in autorust, but not available in the SDK yet: https://github.com/ctaggart/autorust/blob/master/codegen/examples/gen_mgmt.rs#L62 )
• azure.devops

Investigate removing

• azure-mgmt-cosmosdb: This doesn't appear to be imported
• azure.graphrbac: implements is_member_of, which is better served using App Roles (see #147 )
• azure.keyvault: This doesn't appear to be imported rust impl
• azure.servicebus: This doesn't appear to be imported
• azure.mgmt.cosmosdb - We could move to all tables are managed in the Arm template and we use azure.storage.table as the client.

AB#39994746

CLI unable to interact with deployed OneFuzz

Information

• Onefuzz version: 1.3.1
• OS: Ubuntu 20.04 docker container on macOS

Provide detailed reproduction steps (if any)

1. Deploy onefuzz by using deploy.py, following https://github.com/microsoft/onefuzz/blob/main/docs/getting-started.md
2. Config onefuzz cli with the endpoint, authority, and client id returned by deploy.py
3. Check the version using the following command

$ onefuzz versions check --exact -v

4. Open https://microsoft.com/devicelogin and enter the code to login

Expected result

Authenticated and be able to interact with onefuzz using the CLI.

Actual result

After logging in successfully, I received the following message from the login portal and the CLI doesn't return.

AADSTS500113: No reply address is registered for the application.

Windows call stack and ASAN log report are not symbolized in ADO notification

The stacks in the ADO notification are not symbolized, both in the call stack and the ASAN log. However, if running the fuzzer on the crash input on a onefuzz host directly (e.g. over ssh), the reported stacks are symbolized. Verified the relevant PDBs are in the uploaded setup container.

Relevant portion of ADO notification configuration:

"This is the call stack: <ul> {% for item in report.call_stack %} <li> {{ item }} </li> {% endfor %} </ul>ASan log:<br>{{ report.asan_log }}"

Actual notification:

This is the call stack:

    #0 0x12344321 (dll+0x12344321)
    #1 0x12344321 (dll+0x12344321)
...
ASan log:
...

==2500==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x12344321 at pc 0x12344321 bp 0x12344321 sp 0x12344321
READ of size N at 0x12344321 thread T0
    #0 0x12344321 (dll+0x12344321)
    #1 0x12344321 (dll+0x12344321)
...

0x12344321 is located 0 bytes to the right of N-byte region [0x12344321,0x12344321 )
allocated by thread T0 here:
    #1 0x12344321 (dll+0x12344321)
    #2 0x12344321 (dll+0x12344321)
...
==2500==ABORTING

Expected:

This is the call stack:

    #0 0x12344321 in FunctionA sourceA.cpp:123
    #1 0x12344321 in FunctionB sourceB.cpp:456
...
ASan log:
...

==2500==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x12344321 at pc 0x12344321 bp 0x12344321 sp 0x12344321
READ of size N at 0x12344321 thread T0
    #0 0x12344321 in FunctionA sourceA.cpp:123
    #1 0x12344321 in FunctionB sourceB.cpp:456
...

0x12344321 is located 0 bytes to the right of N-byte region [0x12344321,0x12344321 )
allocated by thread T0 here:
    #1 0x12344321 in FunctionC sourceC.cpp:234
    #2 0x12344321 in FunctionD sourceD.cpp:567
...
==2500==ABORTING

Mark tasks as `setting_up` when their claiming node enters `setting_up`

When a node state enters setting_up, it does so because it has claimed a work set, which encompasses 1 or more tasks that share a common target setup container.

Right now, when this happens, the task state is scheduled, even though it has been claimed by a node. We don't reflect this until a given task is actually running, which may not happen until after running a target setup script, rebooting, &c. We also don't link the node and task commitment until this point, even though it exists earlier.

Instead, include the precipitating work set (or at least its task IDs) in the Node setting_up event, and immediately (1) update the task state to setting_up (which will be a new TaskState), and (2) create the NodeTask value which makes it known that the given task has been claimed by the given node.

Unable to determine coverage for libfuzzer targets without sancov support

Information

• Onefuzz version: 1.2.0
• OS: linux

Provide detailed reproduction steps (if any)

1. Deploy a go-fuzz based libfuzzer target. (See: https://github.com/microsoft/onefuzz-samples/tree/main/examples/golang)

Expected result

New seeds are found, coverage is recorded.

Actual result

New seeds are found, coverage task fails with the following error:

[2020-10-08T21:27:29Z ERROR onefuzz_agent] error running task: no coverage files for input: task_readonly_inputs_2/5ba93c9db0cff93f52b521d7420e43f6eda2784f

handle quota issues during scaleset resize

Information

• Onefuzz version: 1.0.0
• OS: NA

Provide detailed reproduction steps (if any)

1. Create scaleset
2. Let stablize
3. onefuzz scaleset update $ID --size 99999

Expected result

Resize attempt is rejected

Actual result

Eventually an exception about quotas is generated in the service.

AB#39994767

If a supervisor fails after starting a task, it will pick up another task and execute it.

Information

• Onefuzz version: 1.0.0
• OS: linux

Provide detailed reproduction steps (if any)

1. Start task
2. Wait until it's running on a node
3. Login to node (debug ssh works here)
4. Update service to raise an Exception in the agent callback URL handlers
5. Wait until supervisor exits due to HTTP errors
6. Undo change in step 4.
7. Watch supervisor restart and pick up a new task.

Expected result

Supervisor picks up a new task for the node, even though it's already running one

Actual result

Supervisor is resilient to http comms issues and has multiple retry attempts with backoff logic.

Go language support

We have a huge code base using Go, we would like to know if we can use this tool or what changes are required to support Go.

notify users on job failure

If a job gets launched and something is wrong with it (example: the setup script has a syntax error), then the task is marked as failed and the error is recorded.

When jobs are launched via CICD, users have no way to know their jobs have failed. It would be good to provide a notification for these failures.

(Suggestion: Teams message)