Distributed Magma fuzzing benchmark
flowchart TD;
GeneralQueue[(General Queue)]
FuzzingQueue[(Fuzzing Queue)]
S3[\S3/]
subgraph general workers
gworker-0
gworker-n
end
subgraph fuzzing workers
fworker-0
fworker-n
end
Trigger((Trigger))-. start_campaign .->GeneralQueue
GeneralQueue-. start_campaign,reduce .-gworker-0 & gworker-n-. fuzzing-task .->FuzzingQueue
FuzzingQueue-. fuzzing-task .->fworker-0 & fworker-n-- tar -->S3
gworker-0 & gworker-n<-- tar,json -->S3
- Define a development environment
.env
in the same directory of thedocker-compose.yml
:SHARED_VOLUME=shared-workdir-volume SHARED_PATH=/tmp/shared-workdir BROKER_USER=guest BROKER_PASS=guest S3_ACCESS_KEY=access_key S3_SECRET_KEY=secret_key
- Bootstrap the required services:
docker compose up -d tasks-broker tasks-result tasks-monitor s3
- Install dependencies
# all tests and executions have to be done in the context of the "src" folder cd src pip install -r requirements.txt -r requirements.dev.txt
- Perform the tests
Just to check that everything works:
docker compose up -d tasks-broker tasks-result s3 fuzzing-worker general-worker
cd src
pytest
If you want more control:
cd src
# Test locally (on the host)
export SHARED_PATH=/tmp/shared-workdir
export SHARED_VOLUME=${SHARED_PATH}
# Clear created docker images and partially S3 storage after the tests
export CLEAR_CACHE=1
pytest -m "not celery"
Test the celery tasks:
- Start the fuzzing worker
# Test locally (on the host) export SHARED_PATH=/tmp/shared-workdir export SHARED_VOLUME=${SHARED_PATH} ./run-fuzzing-worker.sh -l INFO
- Start the general worker in the new terminal
./run-general-worker.sh -l INFO
- Run the tests in another terminal
pytest -m "celery"
- API
- Remove dependency from the same docker volume of the fuzzing workers