sa7mon / s3scanner

Scan for misconfigured S3 buckets across S3-compatible APIs!

S3Scanner

Features - Usage - Quick Start - Installation - Discuss

A tool to find open S3 buckets in AWS or other cloud providers:

• AWS
• DigitalOcean
• DreamHost
• GCP
• Linode
• Scaleway
• Custom

Features

• ⚡️ Multi-threaded scanning
• 🔭 Supports many built-in S3 storage providers or custom
• 🕵️‍♀️ Scans all bucket permissions to find misconfigurations
• 💾 Save results to Postgres database
• 🐇 Connect to RabbitMQ for automated scanning at scale
• 🐳 Docker support

Used By

Usage

INPUT: (1 required)
  -bucket        string  Name of bucket to check.
  -bucket-file   string  File of bucket names to check.
  -mq                    Connect to RabbitMQ to get buckets. Requires config file key "mq". Default: "false"

OUTPUT:
  -db       Save results to a Postgres database. Requires config file key "db.uri". Default: "false"
  -json     Print logs to stdout in JSON format instead of human-readable. Default: "false"

OPTIONS:
  -enumerate           Enumerate bucket objects (can be time-consuming). Default: "false"
  -provider    string  Object storage provider: aws, custom, digitalocean, dreamhost, gcp, linode, scaleway - custom requires config file. Default: "aws"
  -threads     int     Number of threads to scan with. Default: "4"

DEBUG:
  -verbose     Enable verbose logging. Default: "false"
  -version     Print version Default: "false"

If config file is required these locations will be searched for config.yml: "." "/etc/s3scanner/" "$HOME/.s3scanner/"

🚀 Support

If you've found this tool useful, please consider donating to support its development. You can find sponsor options on the side of this repo page or in FUNDING.yml

Huge thank you to tines for being an ongoing sponsor of this project.

Quick Start

Scan AWS for bucket names listed in a file, enumerate all objects

$ s3scanner -bucket-file names.txt -enumerate

Scan a bucket in GCP, enumerate all objects, and save results to database

$ s3scanner -provider gcp -db -bucket my-bucket -enumerate

Installation

Platform	Version	Steps
Homebrew (MacOS)		brew install s3scanner
Kali Linux		apt install s3scanner
Parrot OS		apt install s3scanner
BlackArch		pacman -S s3scanner
Docker		docker run ghcr.io/sa7mon/s3scanner
Winget (Windows)		winget install s3scanner
Go		go install -v github.com/sa7mon/s3scanner@latest
Other (Build from source)		git clone [email protected]:sa7mon/S3Scanner.git && cd S3Scanner && go build -o s3scanner .

Using

Input

s3scanner requires exactly one type of input: -bucket, -bucket-file, or -mq.

INPUT: (1 required)
  -bucket        string  Name of bucket to check.
  -bucket-file   string  File of bucket names to check.
  -mq                    Connect to RabbitMQ to get buckets. Requires config file key "mq". Default: "false"

-bucket

Scan a single bucket

s3scanner -bucket secret_uploads

-bucket-file

Scans every bucket name listed in file

s3scanner -bucket-file names.txt

where names.txt contains one bucket name per line

$ cat names.txt
bucket123
assets
image-uploads

Bucket names listed multiple times will only be scanned once.

-mq

Connects to a RabbitMQ server and consumes messages containing bucket names to scan.

s3scanner -mq

Messages should be JSON-encoded Bucket objects - refer to mqingest for a Golang publishing example.

-mq requires the mq.uri and mq.queue_name config file keys. See Config File section for example.

Output

OUTPUT:
  -db       Save results to a Postgres database. Requires config file key "db.uri". Default: "false"
  -json     Print logs to stdout in JSON format instead of human-readable. Default: "false"

-db

Saves all scan results to a PostgreSQL database

s3scanner -bucket images -db

• Requires the db.uri config file key. See Config File section for example.
• If using -db, results will also be printed to the console if using -json or the default human-readable output mode.
• s3scanner runs Gorm's Auto Migration feature each time it connects two the database. If the schema already has tables with names Gorm expects, it may change these tables' structure. It is recommended to create a Postgres schema dedicated to s3scanner results.

-json

Instead of outputting scan results to console in human-readable format, output machine-readable JSON.

s3scanner -bucket images -json

This will print one JSON object per line to the console, which can then be piped to jq or other tools that accept JSON input.

Example: Print bucket name and region for all buckets that exist

$ s3scanner -bucket-file names.txt -json | jq -r '. | select(.bucket.exists==1) | [.bucket.name, .bucket.region] | join(" - ")'       
10000 - eu-west-1
10000.pizza - ap-southeast-1
images_staging - us-west-2

Options

OPTIONS:
  -enumerate           Enumerate bucket objects (can be time-consuming). Default: "false"
  -provider    string  Object storage provider: aws, custom, digitalocean, dreamhost, gcp, linode, scaleway - custom requires config file. Default: "aws"
  -threads     int     Number of threads to scan with. Default: "4"

-enumerate

Enumerate all objects stored in bucket. By default, s3scanner will only check permissions of buckets.

s3scanner -bucket attachments -enumerate

• Note: This can take a long time if there are a large number of objects stored.
• When enumerating, s3scanner will request "pages" of 1,000 objects. If there are more than 5,000 pages of objects, it will skip the rest.

-provider

Name of storage provider to use when checking buckets.

s3scanner -bucket assets -provider gcp

• Use "custom" when targeting a currently unsupported or local network storage provider.
• "custom" provider requires config file keys under providers.custom listed in the Config File section.

-threads

Number of threads to scan with.

s3scanner -bucket secret_docs -threads 8

• Increasing threads will increase the number of buckets being scanned simultaneously, but will not speed up object enumeration. Enumeration is currently single-threaded per bucket.

Debug

DEBUG:
  -verbose     Enable verbose logging. Default: "false"
  -version     Print version Default: "false"

-verbose

Enables verbose logging of debug messages. This option will produce a lot of logs and is not recommended to use unless filing a bug report.

s3scanner -bucket spreadsheets -verbose

-version

Print the version info and exit.

s3scanner -version

• Will print dev if compiled from source.

Development

A docker compose file is included which creates 4 containers:

• rabbitmq
• postgres
• app
• mitm

2 profiles are configured:

• dev - Standard development environment
• dev-mitm - Environment configured with mitmproxy for easier observation of HTTP traffic when debugging or adding new providers.

To bring up the dev environment run make dev or make dev-mitm. Drop into the app container with docker exec -it -w /app app_dev sh, then go run . If using the dev-mitm profile, open http://127.0.0.1:8081 in a browser to view and manipulate HTTP calls being made from the app container.

Config File

If using flags that require config options, s3scanner will search for config.yml in:

• (current directory)
• /etc/s3scanner/
• $HOME/.s3scanner/

# Required by -db
db:
  uri: "postgresql://user:[email protected]:5432/schema_name"

# Required by -mq
mq:
  queue_name: "aws"
  uri: "amqp://user:pass@localhost:5672"

# providers.custom required by `-provider custom`
#   address_style - Addressing style used by endpoints.
#     type: string
#     values: "path" or "vhost"
#   endpoint_format - Format of endpoint URLs. Should contain '$REGION' as placeholder for region name
#     type: string
#   insecure - Ignore SSL errors
#     type: boolean
# regions must contain at least one option
providers:
  custom: 
    address_style: "path"
    endpoint_format: "https://$REGION.vultrobjects.com"
    insecure: false
    regions:
      - "ewr1"

When s3scanner parses the config file, it will take the endpoint_format and replace $REGION for all regions listed to create a list of endpoint URLs.

S3 compatible APIs

Note: S3Scanner currently only supports scanning for anonymous user permissions of non-AWS services

📚 More information on non-AWS APIs can be found in the project wiki.

Permissions

This tool will attempt to get all available information about a bucket, but it's up to you to interpret the results.

Possible permissions for buckets:

• Read - List and view all files
• Write - Write files to bucket
• Read ACP - Read all Access Control Policies attached to bucket
• Write ACP - Write Access Control Policies to bucket
• Full Control - All above permissions

Any or all of these permissions can be set for the 2 main user groups:

• Authenticated Users
• Public Users (those without AWS credentials set)
• Individual users/groups (out of scope of this tool)

What this means: Just because a bucket doesn't allow reading/writing ACLs doesn't mean you can't read/write files in the bucket. Conversely, you may be able to list ACLs but not read/write to the bucket

License

MIT

s3scanner's Issues

Test: Add scenarios to test_arguments()

• Scenario mainargs.2: --out-file
• Scenario mainargs.3: --list
• Scenario mainargs.4: --dump

Test: Add scenarios to test_checkBucketWithoutCreds()

• Scenario checkBucketwc.1 - Non-existent bucket
• Scenario checkBucketwc.2 - Good bucket
• Scenario checkBucketwc.3 - No public read perm

Add "default region" as a flag

To speed things up if the user has an idea of which S3 region the buckets are probably in

String formatting of logs not pretty

Seems to be only when a certain sequence of results happens. i.e. open, not found, closed

Add --try-upload option

Try to upload a file to see if write permissions are available. Have a default file that it tries to upload and possible re-create the file automatically if it doesn't exist.

Performance: Reduce number of calls to checkBucket()

Will need to do some initial testing to see how much time a bucket file of size, say, 200 takes. Then look at taking out the calls to checkBucket from inside the other functions. The way the program flow goes, they're always checked first so we can probably assume later that they exist.

Change output line prefix

Check if it's still outputting lines with " [found] :" as prefix. If so, remove that. Output should be awk-able on '|'.

Scanning stops after ~ 2k domains

Domain list used: alexa1m-resume.txt

Scanning just stops after 2,343 domains

Add --version output

Now that we're doing proper release versioning, add an arg to output it. Also useful for having people submit issues.

Test: Add scenarios to test_dumpBucket()

• Scenario dumpBucket.2 - Public read permission disabled
• Scenario dumpBucket.3 - Authenticated users read enabled, public users read disabled

Cancelling bucket dumping doesn't work

Need to pass the Ctrl+C signal to the sh process from the Python script

Seperate the "frontend" from the "backend"

Currently, both s3scanner and s3utils do string formatting on the results. Move all string formatting to s3scanner to simplify things and make the code more portable.

Add --no-listing option

Listing buckets is time consuming. Add an option to either skip it, or just check if we can list it instead of getting the size.

Omitting and including --list have the same effect

Both produce same output:
s3scanner.py sites.txt
s3scanner.py --list sites.txt

• Decide what default behavior should be. Probably just don't write anything to file if --list is omitted.
• Change the name to something more meaningful

Probably fix #61 at the same time

Enhancement Ideas

Finder

• Job creation - Job = Buckets to try
• Notifications with configurable triggers. i.e. Interesting file types, total bucket size, or any buckets found in job.

Dumper

• Set maximum file size

Add references to other tools:

• https://github.com/jordanpotti/AWSBucketDump
• https://www.virtuesecurity.com/blog/aws-penetration-testing-s3-buckets/

Grab owner displayName if possible

I think it's only visible if the bucket is list-able. Look into.

Move tests to tests folder

Figure out how to do this using pytest

Test: Add scenario to test_listBucket()

• Scenario listBucket.2 - Public read disabled

Remove test.txt

Not sure why it's in master. Make sure we really don't need it first.

Use Python's logging module

pprint was only supposed to be temporary. Fix that.

--list help text not clear

Buckets in bucket-stream form return 'not found'

Buckets from the log output of bucket-stream are in the form: mybucket.s3.amazonaws.com. Currently, these buckets return as non-existent.

This seems to be part of a bigger (seemingly incorrect) assumption I made about how S3 urls work. It looks like bucket.s3.amazonaws.com will always work while bucket.s3-region-1.amazonaws.com my not always return valid results.

Remedy

Probably switch to always checking bucket.s3.amazonaws.com. Otherwise, look at using boto3 or aws-cli to check for bucket existence.

Add single bucket option

Would also be cool if it could read from STDIN

Simplify bucket checking with FQDN

Instead of taking a FQDN, getting the DNS records, and then resolving them to see if they resolve to s3-website-region.amazonaws.com, we can simply GET websitename.com-any-s3-region.amazonaws.com and check for a 200 or 301 status.

Test: Add scenarios to test_checkBucketName()

• checkBucketName.4 - Blank name
• checkBucketName.5 - Good name

getBucketSize operation times out

Maybe related to #13.

Description: When a bucket with a large number of files is found and the script attempts to get the size of the bucket, the operation times out and the script exits with a 255 error return code.

Steps to reproduce: use letsintern(dot)com

Stack trace: (partial output)

raise exc
sh.ErrorReturnCode_255:

RAN: /Volumes/DATA/Projects/s3scanner/venv/bin/aws s3 ls --summarize --human-readable --recursive --no-sign-request s3://letsintern(dot)com

STDERR:
The read operation timed out

Test: Add scenarios to test_checkAcl()

• checkAcl.1 - Public ACL listing enabled
• checkAcl.4 - Bucket doesn't exist

Update README.md

The README.md file mentions in the first example "--include-closed", yet this option was deprecated in one of the recent commits. Maybe edit that one out and replace with another working example.

Bucket names with punctuation break scanning

bucketname: de'an

Need to filter out all invalid characters from input bucket names

Amazon docs

Test: Verify --out-file works correctly

Verify that when --out-file is set that:

• The supplied file is written to
• buckets.txt isn't written to

Verify that when --out-file isn't set that:

• buckets.txt gets written to

Cant Use s3scanner on Windows 10

s3scanner.py -h
Traceback (most recent call last):
File "C:\Python3\S3Scanner-master\s3scanner.py", line 12, in
import s3utils as s3
File "C:\Python3\S3Scanner-master\s3utils.py", line 1, in
import sh
File "C:\Python3\lib\site-packages\sh.py", line 36, in
support." % version)
ImportError: sh 1.12.14 is currently only supported on linux and osx. please install pbs 0.110 (http://pypi.python.org/pypi/pbs) for windows support.

#########################################################

pip3 install sh
Requirement already satisfied: sh in c:\python3\lib\site-packages (1.12.14)

pip3 install pbs
Requirement already satisfied: pbs in c:\python3\lib\site-packages (0.110)

pip3 install s3utils
Requirement already satisfied: s3utils in c:\python3\lib\site-packages (0.6.1)
Requirement already satisfied: boto>=2.39.0 in c:\python3\lib\site-packages (from s3utils) (2.49.0)

Add more domain name checking options

Blocked by #3

Users should have the ability to provide domain names or bucket names in the following ways:

• List of bucket names
• List of domain names
• List of full bucket domains from another tool i.e. BucketStream
• bucket:region

Combine both into single tool

Would be nice to have a single tool. Probably Python that can call a bash subprocess.

Make log file argument optional

Create a default value for the out file and make it optional.

Add --no-listing or --listing-timeout argument

Since listing is pretty time intensive, add an option to either not list at all, or to set the timeout to change the default from 8. A user who wants to quickly scan a big list can change the timeout to 1.

Public-ness is not accurate

S3 ACL's can apply to Authenticated Users (those with AWS credentials) or to Everyone (the general internet). ACL's can also apply to specific users, but we're not getting into that.

The problem with the code right now is that to check if a bucket is open, it does a GET request which is filtered by the Everyone ACL. The bug you found is because that bucket is not readable by the Everyone group, but is readable by Authenticated Users (Those who have the aws-cli tool installed and AWS credentials defined in ~/.aws/credentials).

Remedy

Probably use boto

Add ability to list buckets contents without dumping

Requested on Twitter

ImportError: No module named sh

~/Tools/S3Scanner$ python s3scanner.py
Traceback (most recent call last):
File "s3scanner.py", line 12, in
import s3utils as s3
File "/home/wayc0de/Tools/S3Scanner/s3utils.py", line 1, in
import sh
ImportError: No module named sh

Add test: checkAwsCreds

Dunno how we're going to tackle this one on Travis. Probably a way of storing secrets.

Performance: Speed up file listings

Currently, it takes forever for listings to be saved to file. Not sure if this can be fixed or if we'll just have to rely on multi-threading to help boost speed.

GetBucketSize() doesn't get the size of buckets

Since the getBucketSize() function uses the --no-sign-request, it functions as a member of the Everyone group. This means that it can't get the size of buckets when the Everyone group doesn't have read rights, but the Authenticated Users group does.

Unhandled status code

bucket: ed

Traceback (most recent call last):
  File "./s3scanner.py", line 109, in <module>
    result = s3.checkBucket(bucket, region)
  File "/Projects/s3scanner/s3utils.py", line 35, in checkBucket
    raise ValueError("Got an unhandled status code back: " + str(r.status_code) + " for site: " + bucketName + ":" + region)
ValueError: Got an unhandled status code back: 400 for site: ed:us-west-1

Test: All bucket input options

• Verify all bucket options are working. Need to add full S3 url test

Add to test matrix

Performance: Add multi-threading/multi-processing

Whichever is more appropriate in this situation.

Add test: listBucket

• Verify the list-buckets folder gets created
• Verify the file gets written to

Output file doesn't match specified format

Format of the lines in the output should be:
bucketname:region

Image for readme

No list/write permissions marked

👋
It seems that calling BucketAcl on bucket to which ACL you don't have read access is restricted. No read access to ACL is the default so s3scanner won't mark the buckets as listable or writable.

>>> s3.BucketAcl("rubyci").load()
...
ClientError: An error occurred (AccessDenied) when calling the GetBucketAcl operation: Access Denied

docker run s3scanner rubyci.s3.amazonaws.com
2019-03-17 15:35:55   Warning: AWS credentials not configured. Open buckets will be shown as closed. Run: `aws configure` to fix this.

2019-03-17 15:36:04       [found] : rubyci | Unknown Size - timeout | ACLs: unknown - no aws creds

With bash, you can see that bucket is in fact listable:

$ curl rubyci.s3.amazonaws.com
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>rubyci</Name>
...

Setup automated Docker builds

Setup Docker automated builds to update as the repo is updated

#72 (comment)

Test: Add scenarios to test_getBucketSize()

• Scenario getBucketSize.3 - Public read disabled
• Scenario getBucketSize.4 - Bucket doesn't exist