c0rv4x / project-black

Pentest/BugBounty progress control with scanning modules

Project Black

Scope control, scope scanner and progress tracker for easier working on a bug bounty or pentest project

Anatoly Ivanov of Positive Technologies

What is this tool for?

The tools encourages more methodical work on pentest/bugbounty, tracking the progress and general scans information.

It can launch

• masscan
• nmap
• dirsearch
• amass
• patator

against the scope you work on and store the data in a handy form. Perform useful filtering of the project's data, for instance:

• find me all hosts, which have open ports, but not 80
• find me all hosts, whose ips start with 82.
• find me hosts where dirsearch has found at least 1 file with 200 status code

Installation

Basic setup via docker-compose will run on any system which has docker and docker-compose dependency

If you don't have docker installed then

Docker for Ubuntu/Debian

sudo apt install docker.io

Tool installation

If you have docker set up, then for Ubuntu/Debian simply

sudo curl -L "https://github.com/docker/compose/releases/download/1.23.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
git clone https://github.com/c0rvax/project-black
cd project-black
sudo docker-compose up

If you see some SQL errors, try stopping docker-compose (Ctrl + C once and wait for nice shutdown) and run docker-compose up

This might take some time but that's it! Other distros should have very similar guidance.

Now head to http://localhost:5000, enter the credentials. They can be found in https://github.com/c0rvax/project-black/blob/master/config/config_docker.yml under application

For a more complex setup or something failed, see the wiki.

Resources notice

None of the docker containers restrict the amount of resources usage, you are on your own here, however, you can change the amount of parallel tasks for each worker separately. See the wiki for that

How to work?

After a setup, create a project and head to the respective page.

Now we will follow the basic steps which you can do within the application

Add scope

Let's say we are assessing hackerone.com and all it's subdomains. Write hackerone.com into the add scope field and press Add to scope

Entrypoint has been added.

There are other ways to add scope, see wiki

Quick note on working

All of the tasks can read parameters from the user, however, lauching with some options won't diplay any new result as it is pretty difficult to parse all possible outputs of a program. So to start, try working duplicating the options from this manual.

Available options can be found on this page

Start amass

Click the blue button Launch task.

A popup with parameters will appear.

It is recommended to click the All_top_level_domains check box and in argv enter -ip and click Fire! button.

This would launch amass -d hackerone.com -ip. Note that in this case we did not specify any domain. This is beacause the All_top_level_domains check box means looking into the scope which is stored in the database. So the program sees that hackerone.com was added to the scope and launches amass against it.

Upon finishing, the new data is automatically added to scope.

Start masscan and nmap

Now head to IPs tab. Click the already known button Launch task and choose masscan.

We will launch a quick scan, using the button Top N ports. This autocompletes the argv parameter. Press Fire!

Results are automatically downloaded from the database.

Now click Launch task and choose nmap only open. This will find all the open ports which exist in the database and run nmap only against them.

Click Banner and Fire.

Detected banner will automatically appear

Launching dirsearch

Launch dirsearch against all ips and all open ports (both HTTP and HTTPS would be tried)

On IPs tab click Launch task and select dirsearch. Fill in extenstions you want to try and click Fire!

You can launch dirseach agains hosts (not ips) on the Hosts tab.

Note on dirsearch

If there are no ports, dirsearch won't even start. So first, make sure you launched nmap or masscan to discover open ports.

Inspecting results

There are generally three ways to check the results:

• IPs/Hosts list
• IP/Host details
• Dirsearch list

IPs and Hosts list

Those are two tabs. They work the same way so we will stop on Hosts.

You can see a list of hosts, their ports and files. Also you can edit a comment for that host.

Important part here is filtering box.

You can aggregate different filters using the field shown above. Type the filter you want (there is a helper for that) and press Shift + Enter

IP/Host details

You can also view details on a specific host or ip. Press button with the glasses

There you will see dirsearch result for every open port on that host

Dirsearch list

Dirsearch list button will open a new window showing all found files for every dirsearch which was launched in this project.

Launching tasks against specific scope

IPs and Hosts Launch task are different! The button on IPs page will start against all ips within the current project, meanwhile the button on the Hosts page will launch against hosts.

To launch a task against some hosts, you should

1. Filter the hosts
2. Launch the task

Example:

Some filters have been applied. If we now launch dirsearch, it will be launched against hosts which correspond to the used filters.

project-black's Issues

When a page is loaded all active tasks should be sent

Now only server decides whether to send a task or not. This is implemented via some flag

When task status is updated, new task creation is subverted

After the page is refreshed, a queue of tasks is not shown

Dirsearch table displays data incorrectly

Dirsearch table should display data on all ips, then on all hosts. Now it does mix ips and hosts in some cases, this is pretty uncomfortable

Don't load semantic from CDN

Make semantic bundled

When a task is killed the resources seem not to be freed correctly

Lock scopes is a little big buggy

Firstly, the button does not work when a project was just created
Secondly, the lock is not obvious on what it does, should add some pictures

IPs table cannot filter by host

The bug occurs as we don't fetch hosts when selecting ips. Also, the problem can be due to unhandled filter on the server-side

Dirsearch lists filtering on files not working correctly

Add tasks chains

Add a possibility for a user to create a chains of tasks. E.x.: masscan -> nmap -> dirsearch

Editable element throws error on editing

The component says that fluid cannot be set to true

Stop/Cancel task

SSH patator should check whether it can authenticate with password only

Add a hint for tasks lauchers which will point all the target/ports against which the task will be run

Make tasks table toogling

Add scope: shift + Enter submits new scope

Generate /etc/hosts for resolved domains

Awesome resolver tries to query not only 8.8.8.8, but also requests NS server of one of the main domains. It would be nice to have a button which will display /etc/hosts format which you can put to your local machine in order to access the hosts the right way

Filters should be implemented with ILIKE instead of LIKE

Display a special message when there is no data

Add scope via file upload

Make task status update more granular for masscan

When a page is reloaded, masscan will send new status not immediately

Multiple problems in tasks notifications

• Failed tasks are displayed as a success

• Too many hosts are said to be updated

Dirsearch filtering working incorrectly when you are not on the first page

Switching to ips list before it was initially loaded, throws a server error

project_uuid = int(msg.get('project_uuid', None))
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

Project delete broken

Implement nmap progress

Make scopes comment use semantic's Comment, should look better

Select custom page number

Sometimes you know which page you want to proceed to, so it should be selectable

update_needed forces to reload the page too many times

Nmap only open ports broken after refactoring scopes_manager

Back to home button now working

Unicode dicts are created with error

Add support for filtering by comment

Changing page does not show a spinner (at some pages) and does not scroll to top

Dirsearch table is working incorrectly when a dirsearch is running

Lock scope

Scopes of most projects are specified by ips (sometime ips and hosts). It would be really nice to have a lock, which disallows adding new ips or/and hosts

Postgres work should be async

Resolve returns some ips on NXDomain

When external NSes are used, resolver can use that IPs as a query result which obviously is not intended

Comment is resetted when it is edited and any task puts new progress

Resolve should grab as many NSes as possible

First of all, user should be able to specify the NS against which the domains would be resolved.
Secondly, right now autoresolve grabs the main domain from the first name on the list. If you resolve

test.yandex.ru
mail.google.com

resolver will take only NS of yandex.ru, whereas it should also take NS of google.com

• custom NS
• autoresolve grabs all main domains

Dirsearch incremental load - incorrect "load more" button

If a target has two ports with some files there and we fully load one of the pages, the "load more" button on the second page will be inactive, thinking that there are no more files to be loaded

Serving huge bundle.js results in tcp (websocket) connection broken

Add button "google dork this"

It would be nice to have a special button on ipslist which would open a new window, requesting google dork of the current scope

Export currently filtered scopes

Add a button which would download the currently filtered scope as an attachement

Patator should accept user's dictionaries

Create a special component which can accept dictionaries from the user (upload them to a special directory), delete the dictionary, use it as a parameter to the suitable arguments

Start tasks against the targets which were not scanned

Delete working incorrectly

When delete happens, loader spinner is not updated

Tasks for IP or Host are displayed badly

The table with tasks has somehow moved to the top left corner

Deleting project failes on deleting association table

Dirsearch table is working incorrectly when a dirsearch is running