cloud-architects / cloudiscovery Goto Github PK

More services with Custom VPC.

• Amazon Neptune
• CloudHSM
• SageMaker Notebook
• SageMaker Training job
• SageMaker Models

• Directory Service
• Workspaces instances attached to a specific vpc
• Quicksight Data Source

It would be good to have a command to discover Alicloud network resources.

I got Lambda functions that are related only to two out of three private subnets. Yet, on the diagram this is not represented and the lambda function goes to an aggregated 3 subnets, which is incorrect.

That should be further adjusted, so this subgroup is represented on the diagram. That can be done to e.g. display sub-subnets on relations.

It would be good to have:

• ALB resources
• Internet Gateway
• NAT Gateway
• VPC/Service Endpoint
• Route tables
• Subnets
• Peered connections
• Security Groups
• NACLs
• Virtual Private Gateways
• VPN Connections
• Client VPN Endpoints

It's possible to refer VPC Endpoint [1] instead of VPC id when defining policies. It even seems like is the only option in case of SQS Policies [2].

The tool should correlate endpoints to VPCs when generating a report.

Refs
[1] https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourcevpce
[2] https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-api-permissions-reference.html

Boto3 sessions aren't thread safe [1], so the code must be refactored to ensure a session is not shared between threads.

[1] https://boto3.amazonaws.com/v1/documentation/api/latest/guide/session.html#multithreading-or-multiprocessing-with-sessions

Command: python clouddiscovery_init_.py aws-vpc -v vpc-XXXXX -d True -r us-east-1

Error message:

Collecting data from EFS Mount Targets...
Error running check EFS.get_resources. Error message 'Name'

There's a lot of duplicates in the diagram with multiple EC2 instances in ASGs. That should be aggregated, along with ECS instances

It's would be a good idea to list all vpcs/iots across all regions.

These resources use custom VPC.

• MediaConnect
• MediaLive Inputs
• MediaStore Container Policy

It should be possible to list account limits.

A base for the new command can be tool https://github.com/jantman/awslimitchecker

A code refactoring from # 5 generated a problem when region is not informed on profile file (~.aws/config). Region is not required when configure aws cli and if not informed region attribute doesn't exist on file.

When running with diagram parameter, it would be good to check if PATH contains Graphviz.

Currently a route table is reported to be directly associated with a VPC but in practice, the relationship is much more complex, with the tables being associated with subnets. The tool should detect that relationship and report it properly.
There is also a concept of a default table, that should be reported.

Hi,

I get the following error when it's scanning.
Command I ran:
clouddiscovery aws-vpc --region-name eu-west-1 --profile-name XXXX
Running:
python --version
Python 3.7.5
pip --version
pip 20.0.2 from /usr/local/lib/python3.7/site-packages/pip (python 3.7)
pip show clouddiscovery
Name: clouddiscovery
Version: 2.0.508
Summary: The tool to help you discover resources in the cloud environment
Home-page: https://github.com/Cloud-Architects/cloud-discovery
Author: Cloud Architects
Author-email: None
License: Apache License 2.0
Location: /usr/local/lib/python3.7/site-packages
Requires: diagrams, boto3, ipaddress
Required-by:
aws --version
aws-cli/2.0.18 Python/3.8.3 Darwin/19.3.0 botocore/2.0.0dev22

Collecting data from SQS Queue Policy...
Collecting data from Subnets...

Error running check SYNTHETICSCANARIES.get_resources. Error message Unknown service: 'synthetics'. Valid service names are: accessanalyzer, acm, acm-pca, alexaforbusiness, amplify, apigateway, apigatewaymanagementapi, apigatewayv2, appconfig, application-autoscaling, application-insights, appmesh, appstream, appsync, athena, autoscaling, autoscaling-plans, backup, batch, budgets, ce, chime, cloud9, clouddirectory, cloudformation, cloudfront, cloudhsm, cloudhsmv2, cloudsearch, cloudsearchdomain, cloudtrail, cloudwatch, codebuild, codecommit, codedeploy, codeguru-reviewer, codeguruprofiler, codepipeline, codestar, codestar-connections, codestar-notifications, cognito-identity, cognito-idp, cognito-sync, comprehend, comprehendmedical, compute-optimizer, config, connect, connectparticipant, cur, dataexchange, datapipeline, datasync, dax, detective, devicefarm, directconnect, discovery, dlm, dms, docdb, ds, dynamodb, dynamodbstreams, ebs, ec2, ec2-instance-connect, ecr, ecs, efs, eks, elastic-inference, elasticache, elasticbeanstalk, elastictranscoder, elb, elbv2, emr, es, events, firehose, fms, forecast, forecastquery, frauddetector, fsx, gamelift, glacier, globalaccelerator, glue, greengrass, groundstation, guardduty, health, iam, imagebuilder, importexport, inspector, iot, iot-data, iot-jobs-data, iot1click-devices, iot1click-projects, iotanalytics, iotevents, iotevents-data, iotsecuretunneling, iotthingsgraph, kafka, kendra, kinesis, kinesis-video-archived-media, kinesis-video-media, kinesis-video-signaling, kinesisanalytics, kinesisanalyticsv2, kinesisvideo, kms, lakeformation, lambda, lex-models, lex-runtime, license-manager, lightsail, logs, machinelearning, macie, managedblockchain, marketplace-catalog, marketplace-entitlement, marketplacecommerceanalytics, mediaconnect, mediaconvert, medialive, mediapackage, mediapackage-vod, mediastore, mediastore-data, mediatailor, meteringmarketplace, mgh, migrationhub-config, mobile, mq, mturk, neptune, networkmanager, opsworks, opsworkscm, organizations, outposts, personalize, personalize-events, personalize-runtime, pi, pinpoint, pinpoint-email, pinpoint-sms-voice, polly, pricing, qldb, qldb-session, quicksight, ram, rds, rds-data, redshift, rekognition, resource-groups, resourcegroupstaggingapi, robomaker, route53, route53domains, route53resolver, s3, s3control, sagemaker, sagemaker-a2i-runtime, sagemaker-runtime, savingsplans, schemas, sdb, secretsmanager, securityhub, serverlessrepo, service-quotas, servicecatalog, servicediscovery, ses, sesv2, shield, signer, sms, sms-voice, snowball, sns, sqs, ssm, sso, sso-oidc, stepfunctions, storagegateway, sts, support, swf, textract, transcribe, transfer, translate, waf, waf-regional, wafv2, workdocs, worklink, workmail, workmailmessageflow, workspaces, xray
Collecting data from VPC Endpoints...
Collecting data from VPC Peering...

Since the tool is able to detect VPC resources, there's a task to detect IoT resources, it would be possible to detect policy resources.

We can all agree IAM policies are complex, with a lot of moving parts and manual verification of evaluation logic is not easy. The tool can help to get an overview of policies within an account.

Similarly to #29, there can be a command to initiate policy reporting:

$ ./aws-network-discovery.py policy --vpc-id vpc-xxxxxxx --region-name xx-xxxx-xxx

When run script with region "all" and there is no default region in ~./aws/config, botocore raise a exception.

python clouddiscovery\__init__.py aws-vpc -v vpc-XXXX -d True -r all

raise NoRegionError()
botocore.exceptions.NoRegionError: You must specify a region.

Since we often have situations when network can get complex, it's often simpler to represent the network as a graph. A ready tool can be used for that purpose, which is cloudmapper.

Update readme to inform how to list all regions.

It should be possible by the tool to pass a single required parameter in supported listing operation.
The tool should rely on the first run on the methods that did not required any input.

Refactor code to implement new enhancements and fix problems:

1 - PEP compliance
2 - New and fast way to instance classes to run checks
3 - General method to parse and analyze most of checks

I think we should update setup.py to get required libraries from requirements.txt.

Right now the tool detects resources within VPC but that's not everything that rund within AWS.
Another area that can need an ability to detect network is IoT.

Because VPC resources and IoT resources can be drawn as independent entities (with small overlap), there should be a separate command to detect either of those two.

For VPC detection the following command can be issued:

$ ./aws-network-discovery.py vpc --vpc-id vpc-xxxxxxx --region-name xx-xxxx-xxx

For IoT detection the following command can be issued:

$ ./aws-network-discovery.py iot --vpc-id vpc-xxxxxxx --region-name xx-xxxx-xxx

There could be overlaps e.g. an IoT Rule can call a Lambda Function but that is acceptable to have.

There should be an option to generate a diagram of the GCP computing infrastructure.

Tags in AWS are used for various purposes to group resources [1].

It should be possible to filter out detected resources by tags, so that it would be possible to report only a specific group of resources instead of all resources within an account.

There can be an additional parameter for that purpose that would accept tag name and value(s) --filter="Name=tags.costCenter;Value=20000:'20001:1'". The mentioned example reports resources only with tag tags.costCenter with values 20000 and 20001:1.

[1] https://d1.awsstatic.com/whitepapers/aws-tagging-best-practices.pdf

VPCs can be referred in IAM policies. It would be good to detect them.

Target groups are crucial part of load balancers and ECS clusters. The tool should:

1. List TGs
2. List related load balancers
3. List related ECS services

It should be possible to report Azure VMs, network infrastructure and DBs.

Would be good to create a HTML report with resources found and relations.

To better understand relations between a policy and resources, an analysis of statements should be made, with focus on principals.

The diagram could provide two types of connections:

• One from a policy with Allow statement to a principal, maybe there should be a green line
• One from a policy with Deny statement to a principal, perhaps a red line should be drawn

It would be good to list:

• S3 Policies like in #2

If a resource stores a reference to a removed subnet, the tool stops working. Call to get subnet details should be wrapped with @exception annotation.

Such situation happens to SageMaker:

Collecting data from Sagemaker Notebook instances...

Error running check SAGEMAKERNOTEBOOK.get_resources. Error message An error occurred (InvalidSubnetID.NotFound) when calling the DescribeSubnets operation: The subnet ID 'subnet-0db79a1feafe30580' does not exist
Collecting data from Security Groups...

Add i18n support.

It should be possible to detect possible resources from boto3 SDK and list them.

Reference implementation: https://github.com/jckuester/awsls

A typical AWS deployment consists often of multiple AWS accounts/organization units.

To be able to properly analyze such deployment, it should be possible to ensure generated report and diagram names won't overlap across different accounts.

An explicit license file is something good to have if it's about to be used for commercial purposes and encourage contributors to add improvements.
Recommended one: Apache License 2.0

It's possible to allow access based on aws:SourceIpkeyword [1].

The task is to detect that and try to correlate VPC with VPC's subnet.
Sine there is no 100% guarantee the correlation will be correct, an information about similarity must be displayed, if potential subnets are detected within account.

Refs
[1] https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourceip

It would be good to have:

• Elasticsearch
• DocumentDB

All resources should have attributes and tags, so that it would be possible to filter on them.

Change diagram version to support new IoT and Sagemaker icons.

It would be good to alert the services that is not available and skip check..

Using pip package I get an exception..

It would be good to list all VPCs, that is ones defined within a region as well as ones peered directly as well as through a transit gateway attachment.

• Synthetics canaries
• Cloudformation

It would be good to list:

• ECS Fargate clusters
• EKS clusters
• EMR clusters
• MSK clusters
• SQS Policies (like in #2)
• ASGs (through subnets)

For performance reasons, commands execution should be parallelized.

Currently https://github.com/mingrammer/diagrams tool is being used for diagrams, which is a free, easy to use and extensive library. Although, during development of the cloudiscovery, we face the following issues with it:

1. Security Groups, VPCs and other resources cannot be represented as areas, making diagrams more complex because of a need of edges to represent relations between resources (instead of just a placement of a resource in a designated area). Related issue: mingrammer/diagrams#162
2. Usage of graphviz engine - although it is good for scientific documents, its UX is disputable. The project tweaks this engine (see mingrammer/diagrams#193) but the visualizations are worse in quality than commercial tools.\

The task is to think about ways to have good diagrams.

It should be possible to read TF files, so that discovery of undeployed infrastructure can be made.