johannesebke / aws_list_all

List all your AWS resources, all regions, all services.

aws_list_all

List all resources in an AWS account, all regions, all services(*). Writes JSON files for further processing.

(*) No guarantees for completeness. Use billing alerts if you are worried about costs.

(Note that the build will break on some boto3 updates)

Usage

You need to have python (only python3 tested) as well as AWS credentials set up as usual.

Quick Start with virtualenv:

mkvirtualenv -p $(which python3) aws
pip install aws-list-all
aws-list-all query --region eu-west-1 --service ec2 --directory ./data/

Quick Start Output:

---------------8<--(snip)--8<-------------------
--- ec2 eu-west-1 DescribeVolumes Volumes
--- ec2 eu-west-1 DescribeVolumesModifications VolumesModifications
--- ec2 eu-west-1 DescribeVpcEndpointConnectionNotifications ConnectionNotificationSet
--- ec2 eu-west-1 DescribeVpcEndpointConnections VpcEndpointConnections
--- ec2 eu-west-1 DescribeVpcEndpointServiceConfigurations ServiceConfigurations
--- ec2 eu-west-1 DescribeVpcEndpoints VpcEndpoints
--- ec2 eu-west-1 DescribeVpcPeeringConnections VpcPeeringConnections
--- ec2 eu-west-1 DescribeVpcs Vpcs
--- ec2 eu-west-1 DescribeVpnConnections VpnConnections
--- ec2 eu-west-1 DescribeVpnGateways VpnGateways
+++ ec2 eu-west-1 DescribeKeyPairs KeyPairs
+++ ec2 eu-west-1 DescribeSecurityGroups SecurityGroups
+++ ec2 eu-west-1 DescribeTags Tags
!!! ec2 eu-west-1 DescribeClientVpnEndpoints ClientError('An error occurred (InternalError) when calling the DescribeClientVpnEndpoints operation (reached max retries: 4): An internal error has occurred')

Lines start with "---" if no resources of this type have been found, and start with "+++" if at least one resource has been found. ">:|" denotes an error due to missing permissions, other errors are prefixed with "!!!",

Currently, some default resources are still considered "user-created" and thus listed, this may change in the future.

Details about found resources are saved in json files named after the service, region, and operation used to find them. They can be dumped with:

aws-list-all show data/ec2_*
aws-list-all show --verbose data/ec2_DescribeSecurityGroups_eu-west-1.json

How do I really list everything?

Warning: As AWS has over 1024 API endpoints, aws-list-all tries to increase your allowed number of open files See #6

Restricting the region and service is optional, a simple query without arguments lists everything. It uses a thread pool to parallelize queries and randomizes the order to avoid hitting one endpoint in close succession. One run takes around two minutes for me.

More Examples

Add immediate, more verbose output to a query with --verbose. Use twice for even more verbosity:

aws-list-all query --region eu-west-1 --service ec2 --operation DescribeVpcs --directory data --verbose

Show resources for all returned queries:

aws-list-all show --verbose data/*

Show resources for all ec2 returned queries:

aws-list-all show --verbose data/ec2*

List available services to query:

aws-list-all introspect list-services

List available operations for a given service, do:

aws-list-all introspect list-operations --service ec2

List all resources in sequence to avoid throttling:

aws-list-all query --parallel 1

aws_list_all's Issues

Follow nextTokens

Currently, if a nextToken is found in a response, we mark it as "non-complete" and show "> N" resources. In some cases (inspector) this leads to "> 0" listings. While this is correct, it would be nice to have an (optional) "complete mode" that follows up all nextTokens.

CloudFormation error in eu-north-1 where stacksets cannot be queried

Error:

cloudformation eu-north-1 ListStackSets ClientError('An error occurred (ValidationError) when calling the ListStackSets operation: AWS CloudFormation StackSets is not supported in eu-north-1')

Not an issue, but a question

Hi Johannes,

Sorry to bother you, but I am curious how you generated the list of service and regions. I found the the links below in google, but It's a bit of challenging to scrape the data from them.

Do you have any secret souce?

Projects to Visualize the Infrastructure as a Diagram?

I really like this project, its good to do some specific queries when one is interested in dig deeper. But It can be easily automated to extract all resources which are running in AWS and eventually draw some infra diagram. Is there any tool which is compatible with output of this tool to visualize infra? I am not ask to develop it here, its more if someone is aware of such app.

Thanks.

Is it still up to date?

Are Regions and Services still up to date?

• Available Regions and Zones
• AWS RDS Regions, Availability Zones, and Local Zones

aws_list_all: error: argument COMMAND: invalid choice: 'eu-west-1' (choose from 'query', 'show', 'introspect', 'recreate-caches')

Hello,

I have a few issues:

1. Running
   aws-list-all --region eu-west-1 --service s3 --directory ./data/

Returns:
usage: aws_list_all [-h] COMMAND ... aws_list_all: error: argument COMMAND: invalid choice: 'eu-west-1' (choose from 'query', 'show', 'introspect', 'recreate-caches')

2. Is there a way I can pull all resources (all regardless of the service) with all theirs tags & IDs? This is to create a manifest which can then be used for bulk tagging.

Nothing is happening when "--service vpc"

is it a personal or general problem?

my goal here is to retrieve information about the "internet gateway" service

Hangs

When I run the example (Ubuntu 18, python 3.6) it just hangs forever. Whenever I ctrl-c the process, it shows the same output:

me:~ $ python3 -m aws_list_all query --service ec2 --region us-east-1
^CTraceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 746, in next
    item = self._items.popleft()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/dist-packages/aws_list_all/__main__.py", line 109, in <module>
    main()
  File "/usr/local/lib/python3.6/dist-packages/aws_list_all/__main__.py", line 87, in main
    do_query(services, args.region, args.operation)
  File "/usr/local/lib/python3.6/dist-packages/aws_list_all/query.py", line 91, in do_query
    for result in ThreadPool(32).imap_unordered(acquire_listing, to_run):
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 750, in next
    self._cond.wait(timeout)
  File "/usr/lib/python3.6/threading.py", line 295, in wait
    waiter.acquire()
KeyboardInterrupt
me:~ $ 

Direct aws CLI commands like aws s3 ls work fine.

Application stops at "Executing queries" and does nothing

I cannot execute any query against any service. This is always what happens:

aws-list-all query --region eu-west-1 --service ec2 --operation DescribeVpcs --verbose
Building set of queries to execute...
Service: ec2                          | Region: eu-west-1       | Operation: DescribeVpcs
...done. Executing queries...

Regardless of operation or service, it just hangs. I've let it run for hours with no progress.

List just all used services

This would be deadly useful for those of us who do pci compliance

Support for GovCloud Regions

It would be nice if the us-gov-west-1 and us-gov-east-1 regions were supported.

When running a query against a gov region:

$ aws-list-all query --directory aws_inventory --region us-gov-west-1 --service ec2 -v -v
Increasing the open connection limit "nofile" from 256 to 6000.

Building set of queries to execute...
...done. Executing queries...
...done

Expected results would be queries running against the region instead of no queries ran.

S3 Service Fails and Unable to Retrieve region agnostic services like Route53

No response for s3, route53 etc.

README is out of date?

I cannot run the "quick start" in the README.

$ aws-list-all --region eu-west-1 --service ec2 --directory ./data/

usage: aws_list_all [-h] COMMAND ...
aws_list_all: error: argument COMMAND: invalid choice: 'eu-west-1' (choose from 'query', 'show', 'introspect', 'recreate-caches')

Is this a regression, or is the README out of date?

AWS KMS ListKeys returns AWS-managed keys together with User-managed keys

Determining if a key is AWS-managed would require a second call per key.

Issue parsing cloudformation ListTypes in multiple regions

Traceback (most recent call last):
  File "/usr/local/bin/aws-list-all", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/aws_list_all/__main__.py", line 142, in main
    do_query(services, args.region, args.operation, verbose=args.verbose or 0, parallel=args.parallel)
  File "/usr/local/lib/python3.7/site-packages/aws_list_all/query.py", line 206, in do_query
    print('Service: {: <28} | Region: {:<15} | Operation: {}'.format(service, region, operation))
TypeError: unsupported format string passed to NoneType.__format__

Route53

One apparent omission seems to be route53. It would be useful to get a complete list of domains.

AWS Braket SearchQuantumTasks not recognized by the heuristic

The Braket SearchQuantumTasks and SearchDevices APIs are such that they require a list as "filters" argument, yet this list can be zero-length. Therefore, there is no equivalent "ListQuantumTask" as in other services.

Relevant API documentation:
https://docs.aws.amazon.com/braket/latest/APIReference/API_SearchQuantumTasks.html
https://docs.aws.amazon.com/braket/latest/APIReference/API_SearchDevices.html

Example code illustrating the problem even without any Quantum Tasks:

boto3.client("braket", region_name="us-east-1").search_devices(filters=[])

application-insights failure

When trying a simple query:
aws-list-all query

the following error is generated:

~\envs\aws\lib\site-packages\aws_list_all\introspection.py", line 343, in get_regions_for_service
regions = set(get_service_regions()[requested_service])
KeyError: 'application-insights'

Dos this mean that the application-insight services reported by AWS is not available in any region? Although this doesnt make sense I would expect this type of error to handled rather than a crash.

OR

What am I doing wrong?

Thanks for your help.

support for profile

Use Cloud Control API

It should be possible to use the new AWS Cloud Control API, both to simplify the methods used to find resources, and to find resources that aren't currently found (and potentially to make it so that new resource types work without any code changes).

List all the domains in route53

Is it possible to list all the route53 domains either based on the ResourceRecordSetCount in (route53_ListHostedZonesByName*) or any other possible methods ?

Using aws-list-all as a library

Hi I am new as well, thank you for creating this!

I am trying to use your module but am having trouble using it inside my python script... I need to list all services for each of my aws accounts but I don't know how I can use this module to do so.. I understand that it says it creates a json of data when running query/commands in the terminal but I cannot find any json produced. Please help/redirect thank you!

KeyError when running in us-east-2 region

I tried to list my AWS resources in us-east-2 and got a KeyError for at least one of the services:

$ aws-list-all query --region us-east-2 --directory ./data/                        

Building set of queries to execute...
Traceback (most recent call last):
  File "/home/ubuntu/.local/share/virtualenvs/acm-api-UVOmdwT4/bin/aws-list-all", line 10, in <module>
    sys.exit(main())
  File "/home/ubuntu/.local/share/virtualenvs/acm-api-UVOmdwT4/lib/python3.7/site-packages/aws_list_all/__main__.py", line 115, in main
    do_query(services, args.region, args.operation, verbose=args.verbose or 0)
  File "/home/ubuntu/.local/share/virtualenvs/acm-api-UVOmdwT4/lib/python3.7/site-packages/aws_list_all/query.py", line 195, in do_query
    for region in get_regions_for_service(service, selected_regions):
  File "/home/ubuntu/.local/share/virtualenvs/acm-api-UVOmdwT4/lib/python3.7/site-packages/aws_list_all/introspection.py", line 343, in get_re
gions_for_service
    regions = set(get_service_regions()[requested_service])
KeyError: 'managedblockchain'

I expect this is a service that does not exist in that region.

JSON Files to CSV

any tips on converting the json files to Excel? Tried pandas library, however the converted Excel files do not display the returned values in the right order of columns/rows.

Drop Python 2.7 as supported?

Let me know if you would like a PR for above.

Recommendation for Policies

Is there a recommendation for a policy with the least rights that can be used to successfully run the script ?

list all ARNs in an account - feature proposal

I'd like to simply get a list of all resources in my account, giving their ARN. As a JSON query this is non trivial and probably impossible to get right since AWS uses many different ways of giving the ARN (example, sometimes it's just a parameter on the resource ( ARN: "value"), sometimes it's a differently named parameter ( "DBClusterARN") and sometimes

I have a workaround; change into the json directory and run

cat *.json | tr '"' '\n' | grep '^arn:aws' | sort |  sed 's/:\*$//'  | uniq 

I don't see this as fully reliable though, for example if there are references to ARNs in a different account they will turn up which may or may not be a good thing, if there are references to old ARNs that no longer exist, the same.

I think the fix is to have this as an option to aws_list_all. This could start by gathering the values of all parameters which match ^(ARN|[^ ]*Arn) and then printing out the unique values but if AWS changed something might need updated.

Out of memory

Hi, trying to run the command to show all resources but the script goes out of memory.

Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
[...]
	File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
MemoryError

How much memory is required to "query" everything and how it could be optimized? Thanks a lot.

Not running in python3.4

Hi, Very nice work!

I tried aws_list_all in python 2.7 and it worked as expected without any problems. However when I moved to another system with a virtualenv using python 3.4, I get errors like the following.

--------------------------------------------------------------------------------
WARNING!
Your system limits the number of open files and network connections to 4096.
This may lead to failures during querying.
Please increase the hard limit of open files to at least 6000.
The configuration for hard limits is often found in /etc/security/limits.conf
--------------------------------------------------------------------------------

Increasing the open connection limit "nofile" from 3072 to 4096.

Building set of queries to execute...
Traceback (most recent call last):
  File "/home/user/scripts/aws/lib/python3.4/site-packages/app_json_file_cache/function_cache.py", line 17, in f
    return self.get(key)
  File "/home/user/scripts/aws/lib/python3.4/site-packages/app_json_file_cache/data_cache.py", line 27, in get
    return self._data[key_string]['data']
KeyError: '{"args": [], "kwargs": {}}'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/scripts/aws/bin/aws-list-all", line 10, in <module>
    sys.exit(main())
  File "/home/user/scripts/aws/lib/python3.4/site-packages/aws_list_all/__main__.py", line 142, in main
    do_query(services, args.region, args.operation, verbose=args.verbose or 0, parallel=args.parallel)
  File "/home/user/scripts/aws/lib/python3.4/site-packages/aws_list_all/query.py", line 203, in do_query
    for region in get_regions_for_service(service, selected_regions):
  File "/home/user/scripts/aws/lib/python3.4/site-packages/aws_list_all/introspection.py", line 373, in get_regions_for_service
    regions = set(get_service_regions().get(requested_service, []))
  File "/home/user/scripts/aws/lib/python3.4/site-packages/app_json_file_cache/function_cache.py", line 20, in f
    return self._cheap_default_func(*args, **kwargs)
  File "/home/user/scripts/aws/lib/python3.4/site-packages/aws_list_all/introspection.py", line 355, in packaged_service_regions
    return load(resource_stream(__package__, 'service_regions.json'))['data']
  File "/usr/lib64/python3.4/json/__init__.py", line 268, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/usr/lib64/python3.4/json/__init__.py", line 312, in loads
    s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'

I will try troubleshoot later and come back in case I find something.

Handle Too Many Open Files (Open Connections)

If run with a nlimit of 1024, aws-list-all runs into OSErrors due to too many open files (connections).

ModuleNotFoundError: No module named 'resource'

Newbie alert! Just getting into both AWS and Python.

Trying to run aws-list-all and I get the following error message.
Python 3.8.2
Windows 10.

(aws-list-all) C:\Program Files (x86)\Python38-32\aws-list-all\Scripts>aws-list-all query --region eu-west-1 --service ec2
Traceback (most recent call last):
  File "c:\program files (x86)\python38-32\lib\runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\program files (x86)\python38-32\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Program Files (x86)\Python38-32\aws-list-all\Scripts\aws-list-all.exe\__main__.py", line 4, in <module>
  File "c:\program files (x86)\python38-32\aws-list-all\lib\site-packages\aws_list_all\__main__.py", line 5, in <module>
    from resource import getrlimit, setrlimit, RLIMIT_NOFILE
ModuleNotFoundError: No module named 'resource'

Loop error?

aws-list-all query --service ec2 --operation DescribeVpcs --directory data --verbose
Building set of queries to execute...
Service: ec2 | Region: us-west-1 | Operation: DescribeVpcs
Service: ec2 | Region: ap-northeast-2 | Operation: DescribeVpcs
Service: ec2 | Region: us-west-2 | Operation: DescribeVpcs
Service: ec2 | Region: ca-central-1 | Operation: DescribeVpcs
Service: ec2 | Region: ap-south-1 | Operation: DescribeVpcs
Service: ec2 | Region: ap-southeast-1 | Operation: DescribeVpcs
Service: ec2 | Region: sa-east-1 | Operation: DescribeVpcs
Service: ec2 | Region: us-east-2 | Operation: DescribeVpcs
Service: ec2 | Region: eu-west-1 | Operation: DescribeVpcs
Service: ec2 | Region: us-east-1 | Operation: DescribeVpcs
Service: ec2 | Region: eu-central-1 | Operation: DescribeVpcs
Service: ec2 | Region: ap-southeast-2 | Operation: DescribeVpcs
Service: ec2 | Region: eu-west-2 | Operation: DescribeVpcs
Service: ec2 | Region: eu-west-3 | Operation: DescribeVpcs
Service: ec2 | Region: eu-north-1 | Operation: DescribeVpcs
Service: ec2 | Region: ap-northeast-1 | Operation: DescribeVpcs
...done. Executing queries...

The task "...done. Executing queries..." gets in loop and never finish, any sugestion?

Thanks in advance

current Travis build fails

https://travis-ci.org/github/JohannesEbke/aws_list_all/jobs/737406268

___________________________ test_get_service_regions ___________________________
    def test_get_service_regions():
        services = get_services()
        regions = get_service_regions()
>       assert set(services) - set(regions) == set()
E       AssertionError: assert {'s3outposts'...stream-write'} == set()
E         Extra items in the left set:
E         'timestream-query'
E         'timestream-write'
E         's3outposts'
E         Use -v to get the full diff

Support for Profile

Currently it only works with the [default] profile credentials. Please provide support for using a different profile from ~/.aws/credentials.

In general we have multiple profiles for different accounts for different purposes.