GithubHelp home page GithubHelp logo

sandialabs / dr_robot Goto Github PK

View Code? Open in Web Editor NEW
141.0 13.0 37.0 6.51 MB

This tool can be used to enumerate the subdomains associated with a company by aggregating the results of multiple OSINT (Open Source Intelligence) tools.

License: Other

Python 99.84% Makefile 0.16%
scr-2340

dr_robot's Introduction

.______  .______  .______  ._______  ._______ ._______  _____._
:_ _   \ : __   \ : __   \ : .___  \ : __   / : .___  \ \__ _:|
|   |   ||  \____||  \____|| :   |  ||  |>  \ | :   |  |  |  :|
| . |   ||   :  \ |   :  \ |     :  ||  |>   \|     :  |  |   |
|. ____/ |   |___\|   |___\ \_. ___/ |_______/ \_. ___/   |   |
 :/      |___|    |___|       :/                 :/       |___|
 :                            :                  :             

Dc27BadgeLicenseBuild StatusGitHub release (latest by date)

Copyright 2019 National Technology & Engineering Solutions of Sandia, LLC (NTESS). Under the terms of Contract DE-NA0003525 with NTESS, the U.S. Government retains certain rights in this software.

Introduction

Dr.ROBOT is a tool for Domain Reconnaissance and Enumeration. By utilizing containers to reduce the overhead of dealing with dependencies, inconsistencies across operating systems, and different languages, Dr.ROBOT is built to be highly portable and configurable.

Use Case: Gather as many public facing servers that an organization possesses. Querying DNS resources enables us to quickly develop a large list of possible targets that you can run further analysis on.

Note: Dr.ROBOT is not just a one trick pony. You can easily customize the tools that are used to gather information, so that you can enjoy the benefits of using latest and greatest along with your battle tested favorites.

List of current tools

  • Altdns
  • Amass
  • Anubis
  • Aquatone (Discover portion, when aquatone had multiple parts)
  • CT-Exposer
  • CTFR
  • Eyewitness
  • HTTPScreenshot
  • Knock
  • NMap Screenshot
  • NMap
  • Reconng
  • Subbrute
  • Subfinder
  • Sublist3r
  • Webscreenshot
  • GoWitness

Config Files

Dr.ROBOT adds config files, templates, logs, output files, and db files to your $HOME directory under .drrobot

The directory structure will look like this:

-rw-r--r--   1       0 Sep 16 12:15 ansible_inventory
drwxr-xr-x   5     160 Sep 16 12:18 ansible_plays
-rw-r--r--   1   13576 Sep 16 12:41 config.json
drwxr-xr-x   4     128 Sep 17 10:48 dbs
drwxr-xr-x  21     672 Sep 16 13:51 docker_buildfiles
drwxr-xr-x   4     128 Sep 16 15:38 logs
drwxr-xr-x   3      96 Sep 16 12:46 output

If you ever break your config beyond saving, you can delete the config.json file in your $HOME directory and rerun Dr.ROBOT, which will generate a new config file for you.

Installation (with pip)

git clone <URL>
cd gitrepo
pip install -r requirements.txt
pip install -e .
drrobot --help


usage: drrobot [-h] [--proxy PROXY] [--dns DNS] [--verbose] [--dbfile DBFILE]
               {gather,inspect,upload,rebuild,dumpdb,output,serve} ...

Docker DNS recon tool

positional arguments:
  {gather,inspect,upload,rebuild,dumpdb,output,serve}
    gather              Runs initial scanning phase where tools under the
                        webtools/scannerscategory will run and gather
                        information used in the following phases
    inspect             Run further tools against domain information gathered
                        from previous step.Note: you must either supply a file
                        which contains a list of IP/Hostnames orThe targeted
                        domain must have a db under the dbs folder
    upload              Upload recon data to Mattermost/Slack
    rebuild             Rebuild the database with additional files/all files
                        from previous runtime
    dumpdb              Dump contents of database (ip,hostname,banners) to a
                        text file with hostname for filename
    output              Generate output in specified format. Contains all
                        information from scans (images, headers, hostnames,
                        ips)
    serve               Serve database file in docker container using django

optional arguments:
  -h, --help            show this help message and exit
  --proxy PROXY         Proxy server URL to set DOCKER http_proxy too
  --dns DNS             DNS server to add to resolv.conf of DOCKER containers
  --verbose             Display verbose statements
  --dbfile DBFILE       Specify what db file to use for saving data too

Installation (pipenv)

git clone <URL>
cd gitrepo
pipenv sync
pipenv shell
drrobot --help


usage: drrobot [-h] [--proxy PROXY] [--dns DNS] [--verbose] [--dbfile DBFILE]
               {gather,inspect,upload,rebuild,dumpdb,output,serve} ...

Docker DNS recon tool

positional arguments:
  {gather,inspect,upload,rebuild,dumpdb,output,serve}
    gather              Runs initial scanning phase where tools under the
                        webtools/scannerscategory will run and gather
                        information used in the following phases
    inspect             Run further tools against domain information gathered
                        from previous step.Note: you must either supply a file
                        which contains a list of IP/Hostnames orThe targeted
                        domain must have a db under the dbs folder
    upload              Upload recon data to Mattermost/Slack
    rebuild             Rebuild the database with additional files/all files
                        from previous runtime
    dumpdb              Dump contents of database (ip,hostname,banners) to a
                        text file with hostname for filename
    output              Generate output in specified format. Contains all
                        information from scans (images, headers, hostnames,
                        ips)
    serve               Serve database file in docker container using django

optional arguments:
  -h, --help            show this help message and exit
  --proxy PROXY         Proxy server URL to set DOCKER http_proxy too
  --dns DNS             DNS server to add to resolv.conf of DOCKER containers
  --verbose             Display verbose statements
  --dbfile DBFILE       Specify what db file to use for saving data too

Certs

Running this behind a proxy was a pain. To make this less painful we create a certs directory under the $HOME/.drrobot/* where you can add your crt files. As part of the dockerfile build process we now generate tarfiles with the certificates so that applications, such as Amass, can run.

Minio

Included with Dr.ROBOT is a docker-compose.yml file. This file contains a simple compose file to serve up Minio and the files gathered during runtime.

To use:

cd /path/to/drrobot/
docker-compose up

Docker

This tool relies heavily on Docker.

See installation instructions here:

Ansible

You can make any module support Ansible.

See Installation guide for instructions.

  • If using a mac you will need to install gnu-tar for Ansible to unpack compressed files: brew install gnu-tar
  • If you have an encrypted ssh key that requires a password to use and would not like to enter their password for every command ran remotely look into using an ssh-agent
eval $(ssh-agent)
ssh-add /path/to/keyfile

Documentation

To add your own tool see the Configuration to get started.

For usage see Usage to get started.

dr_robot's People

Contributors

crimsonk1ng avatar l50 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dr_robot's Issues

Threading of Ansible

Currently Ansible modules are ran on the main thread due to the likely possibility of user input. However, the addition of a configuration option to allow a user to specify whether the Ansible module can be ran without user input would be nice.

Hostname aggregation failing

Refactor hostname aggregation piece to have more consistency when aggregating. Virtual hosting may cause an issue with the aggregation piece. Need to look at refactoring the following methods dealing with hostname aggregation, resolve ips and such.

Eyewitness Python3

Error with Eyewitness migration to Python3. Their current docker container does not come with python3 or pip updated to use python3. On top of this they currently have an issue with raw_input that they are patching. Should be resolved soon.

Python 3.6.8 can't start properly

Hi,
Had some wierd erorrs, please help
Ubuntu 18.04.1 fully up to date
Python 3.6.8

ubuntu@ubuntu:~/Projects/dr_robot$ pipenv install && pipenv shell Installing dependencies from Pipfile.lock (9fd82d)… 🐍 β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰ 32/32 β€” 00:00:09 To activate this project's virtualenv, run pipenv shell. Alternatively, run a command inside the virtualenv with pipenv run. Launching subshell in virtual environment… ubuntu@ubuntu:~/Projects/dr_robot$ . /home/ubuntu/.local/share/virtualenvs/dr_robot-wkMS6zjP/bin/activate (dr_robot) ubuntu@ubuntu:~/Projects/dr_robot$ dr driverless drrobot (dr_robot) ubuntu@ubuntu:~/Projects/dr_robot$ drrobot --help Traceback (most recent call last): File "/home/ubuntu/.local/share/virtualenvs/dr_robot-wkMS6zjP/bin/drrobot", line 11, in load_entry_point('drrobot', 'console_scripts', 'drrobot')() File "/home/ubuntu/.local/share/virtualenvs/dr_robot-wkMS6zjP/lib/python3.6/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point return get_distribution(dist).load_entry_point(group, name) File "/home/ubuntu/.local/share/virtualenvs/dr_robot-wkMS6zjP/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point return ep.load() File "/home/ubuntu/.local/share/virtualenvs/dr_robot-wkMS6zjP/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2443, in load return self.resolve() File "/home/ubuntu/.local/share/virtualenvs/dr_robot-wkMS6zjP/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2449, in resolve module = __import__(self.module_name, fromlist=['__name__'], level=0) File "/home/ubuntu/Projects/dr_robot/src/robot_api/cli.py", line 13, in from robot_api.robot import Robot File "/home/ubuntu/Projects/dr_robot/src/robot_api/robot.py", line 27, in from robot_api.api import Ansible, Docker, Aggregation File "/home/ubuntu/Projects/dr_robot/src/robot_api/api/__init__.py", line 4, in from robot_api.api.upload import Forum, Mattermost, Slack File "/home/ubuntu/Projects/dr_robot/src/robot_api/api/upload.py", line 15, in import slack File "/home/ubuntu/.local/share/virtualenvs/dr_robot-wkMS6zjP/lib/python3.6/site-packages/slack/__init__.py", line 4, in from slack.web.client import WebClient # noqa File "/home/ubuntu/.local/share/virtualenvs/dr_robot-wkMS6zjP/lib/python3.6/site-packages/slack/web/client.py", line 9, in from slack.web.base_client import BaseClient, SlackResponse File "/home/ubuntu/.local/share/virtualenvs/dr_robot-wkMS6zjP/lib/python3.6/site-packages/slack/web/base_client.py", line 15, in import aiohttp File "/home/ubuntu/.local/share/virtualenvs/dr_robot-wkMS6zjP/lib/python3.6/site-packages/aiohttp/__init__.py", line 6, in from .client import BaseConnector as BaseConnector File "/home/ubuntu/.local/share/virtualenvs/dr_robot-wkMS6zjP/lib/python3.6/site-packages/aiohttp/client.py", line 32, in from . import hdrs, http, payload File "/home/ubuntu/.local/share/virtualenvs/dr_robot-wkMS6zjP/lib/python3.6/site-packages/aiohttp/http.py", line 7, in from .http_parser import HeadersParser as HeadersParser File "/home/ubuntu.local/share/virtualenvs/dr_robot-wkMS6zjP/lib/python3.6/site-packages/aiohttp/http_parser.py", line 15, in from .helpers import NO_EXTENSIONS, BaseTimerContext File "/home/ubuntu/.local/share/virtualenvs/dr_robot-wkMS6zjP/lib/python3.6/site-packages/aiohttp/helpers.py", line 58, in import idna_ssl ModuleNotFoundError: No module named 'idna_ssl'

Logs location

When you run into an error, it states: [!] Value Error thrown. Check error log for details

However, I don't see a logs directory or any information about where to find them.

Duplicate HTTPScreenshot containers

I started by running python drrobot.py <domain> gather -sub -aqua, which appeared to be successful.

I followed this with python drrobot.py <domain> inspect -http -eye, which introduced the issue of standing up duplicate HTTPScreenshot containers.

Running using Python 3.6.5

Need to add user-created files and directories to .gitignore

In the normal course of using the program, various directories and files get created that contain information gathered by the end user. These files should never be committed or pushed to the repo, so they should be included in the .gitignore:

Β» git status
On branch master
Your branch is up to date with 'origin/master'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)

	configs/user_config.json
	dbs/
	docker_buildfiles/Dockerfile.Aquatone
	docker_buildfiles/Dockerfile.Sublist3r
	output/

nothing added to commit but untracked files present (use "git add" to track)

Issue when running with fresh clone

This error came up with a fresh clone:

drrobot gather -sub -aqua -shodan -sfinder -amass -arin TARGET

Beginning gather
[!] Value Error thrown. Check error log for details
[!] Value Error thrown. Check error log for details
[#] Docker image Sublist3r, building...: : 1it [00:02,  2.01s/it]
[#] Docker image Amass, building...: : 1it [00:02,  2.01s/it]
[#] Docker image Aquatone, building...: : 1it [00:02,  2.01s/it]]
[#] Docker image Subfinder, building...: : 1it [00:02,  2.01s/it]
[#] Docker container Sublist3r, running...: : 7it [00:14,  2.02s/it]
[#] Docker container Subfinder, running...: : 44it [01:28,  2.01s/it]
[#] Docker container Amass, running...: : 74it [02:28,  2.01s/it]it]]
[#] Docker container Aquatone, running...: : 266it [08:54,  2.01s/it]
[!] File shodan.txt does not exist, verify scan results
[!] File arin.txt does not exist, verify scan results
[!] can't pickle _thread.RLock objects

Unfortunately, the Value Errors are not easy to find out more information about. Additionally, no output is generated.

Verbosity

Add more verbosity for users to debug/visualize output

Sublist3r SSL issue

Sublist3r fails when certificates don't verify. There is currently no insecure option.

Tag images

All associated images should have something like drrobot- prefixed.

Commit config option?

Add configuration option to allow specifying the commit hash of a repo. This would centralize the location of commits to user config and remove the tedium of messing with dockerfiles.

Add CHANGELOG

So we can keep track of what changes on a release-by-release basis.

Install error

I can not install with python3.7. Does it only work on python3.6?

dr_robot# pipenv install && pipenv shell
Warning: Python 3.6 was not found on your system…
You can specify specific versions of Python with:
$ pipenv --python path/to/python

Slack Integration

Slack integration using the new Slack API would make for a very useful addition.

Things needed for configuration:

  • Oauth Token for requests
  • URL (if needed) for authorization and upload

User requirements

  • User would be required to add Scopes and Integration for their Slack application.

Aggregation doesn't work with latest version

Aggregation folder is empty after running gather:
python drrobot.py <target> gather -sub -aqua

However, upon running inspect it works. Is this intended functionality? I figured that the aggregation step should happen during the gather phase, but perhaps I'm mistaken.

Travis-ci

It would be nice to have travis-ci for unit tests, and also include various versions of python 3.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.