trek10inc / awsets Goto Github PK
View Code? Open in Web Editor NEWA utility for crawling an AWS account and exporting all its resources for further analysis.
License: MIT License
A utility for crawling an AWS account and exporting all its resources for further analysis.
License: MIT License
Currently, Log Groups are being queried, but they are not tied to anything. Where possible, add a relationship to the appropriate Lambda, EC2, ECS, EKS, etc resource.
Update code after this bug is fixed in the AWS Go SDK so that S3 buckets include region
Currently, ARN parsing is very rudimentary. It needs updated to handle the more complicated formats, including ones with versions.
awsets list --regions eu-central-1 -o all.json --include ec2
yields an empty file:
❯ cat all.json
[]%
``
Verbose output:
❯ awsets list --regions eu-central-1 -o all.json --include ec2 -v
regions: [eu-central-1]
resource types: [ec2/eip ec2/flowlog ec2/image ec2/instance ec2/internetgateway ec2/keypair ec2/launchtemplate ec2/natgateway ec2/networkacl ec2/networkinterface ec2/routetable ec2/securitygroup ec2/snapshot ec2/subnet ec2/transitgateway ec2/volume ec2/vpc ec2/vpcpeering ec2/vpngateway]
querying 0 combinations
9: finished worker
1: finished worker
2: finished worker
0: finished worker
7: finished worker
3: finished worker
4: finished worker
6: finished worker
5: finished worker
8: finished worker
I double checked and `aws ec2 describe-instances` works as expected. What could be the issue?
I'm unable to install via Homebrew. Further, visiting the Homebrew website, I cannot find any reference to the Trek10 tap.
❯ brew tap trek10inc/tap
Error: Invalid formula: /opt/homebrew/Library/Taps/trek10inc/homebrew-tap/Formula/awsets.rb
formulae require at least a URL
Error: Cannot tap trek10inc/tap: invalid syntax in tap!
First, thanks for adding DDB backups to 0.6.1.
Commit 0925415... / issue #14 added DDB backups as sub-resources to ddb/table
. DynamoDB backups can exist of now-deleted tables that still need to be cleaned up.
To gather those, I think there needs to be a separate resource type ddb/backup
added instead of backups being a sub-resource listed on existing tables. Another option would be to make a fake ddb/table
resource with a status of DELETED
and put those backups as sub-resources of.
Steps:
Running ./awsets list -o all.json
with a custom AWS_PROFILE
set.
Stack:
1: failed job us-west-1 - AWSGreengrassGroup
with error: failed to list greengrass groups: exceeded maximum number of attempts, 3, TooManyRequestsException: Too Many Requests
status code: 429, request id: 04eafedd-fdaf-4442-a1c1-113415c22ffc
3: failed job us-west-1 - AWSGreengrassCoreDefinition
with error: failed to list greengrass core definitions: exceeded maximum number of attempts, 3, TooManyRequestsException: Too Many Requests
status code: 429, request id: c0bf6a76-77e8-4fbe-a890-61c2801710c1
2: failed job us-west-1 - AWSGreengrassFunctionDefinition
with error: failed to list greengrass function definitions: exceeded maximum number of attempts, 3, TooManyRequestsException: Too Many Requests
status code: 429, request id: 688f3272-5b39-41a8-aa6c-c238081bf0e4
6: failed job ap-northeast-3 - AWSEc2TransitGateway
with error: InvalidAction: The action DescribeTransitGateways is not valid for this web service.
status code: 400, request id: 01331f76-be70-4a48-a5cc-d4f892b48bc4
2: failed job ap-northeast-3 - AWSCloudFormationStackSet
with error: ValidationError: AWS CloudFormation StackSets is not supported in this region
status code: 400, request id: b4041a21-b14f-4a9a-b775-be75c2df9962
7: failed job us-west-1 - AWSGreengrassResourceDefinition
with error: failed to list greengrass resource definitions: exceeded maximum number of attempts, 3, TooManyRequestsException: Too Many Requests
status code: 429, request id: 275297d2-96f5-4efa-8aa5-ba5748a35e61
0: failed job ap-northeast-3 - AWSApiGatewayApiKey
with error: AccessDeniedException:
status code: 403, request id: d89513e7-db9c-4da9-8796-e2534586e0e5
0: failed job us-west-1 - AWSGreengrassDeviceDefinition
with error: failed to list greengrass device definitions: exceeded maximum number of attempts, 3, TooManyRequestsException: Too Many Requests
status code: 429, request id: 614d40e2-a902-4691-9d10-277f674df0a9
5: failed job us-west-1 - AWSGreengrassSubscriptionDefinition
with error: failed to list greengrass subscription definitions: exceeded maximum number of attempts, 3, TooManyRequestsException: Too Many Requests
status code: 429, request id: 636d7bd6-51af-4369-8c75-bdbfff9d5960
9: failed job us-west-1 - AWSGreengrassConnectorDefinition
with error: failed to list greengrass connector definitions: exceeded maximum number of attempts, 3, TooManyRequestsException: Too Many Requests
status code: 429, request id: d34de1f4-69ba-4ec6-b020-039b2972e769
9: failed job us-west-1 - AWSGreengrassLoggerDefinition
with error: failed to list greengrass logger definitions: exceeded maximum number of attempts, 3, TooManyRequestsException: Too Many Requests
status code: 429, request id: f4cdb082-0f2c-4294-9a82-c0aa185e1018
6: failed job ap-northeast-3 - AWSApiGatewayVpcLink
with error: AccessDeniedException:
status code: 403, request id: e6bc79c1-2879-4787-b836-f786bb65716e
6: failed job ap-northeast-3 - AWSApiGatewayV2DomainName
with error: failed to list apigatewayv2 domain names: AccessDeniedException:
status code: 403, request id: 5c487e09-35bc-4b6d-aed7-01da76ee3128
7: failed job ap-northeast-3 - AWSApiGatewayDomainName
with error: AccessDeniedException:
status code: 403, request id: b54f89b7-1409-41ac-b9fb-773b083d6784
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x3278f33]
goroutine 41 [running]:
github.com/trek10inc/awsets/arn.ParseP(...)
/home/runner/work/awsets/awsets/arn/main.go:32
github.com/trek10inc/awsets/lister.AWSEcsCluster.List(0xc00041baa0, 0x9, 0x42d8840, 0xc00023e9c0, 0x42d2ce0, 0xc0004395e0, 0x42d2ae0, 0xc0000f2cc0, 0xc0013be450, 0x2, ...)
/home/runner/work/awsets/awsets/lister/ecs_cluster.go:80 +0x973
github.com/trek10inc/awsets.List.func1(0xc00003cb40, 0xc000456180, 0xc00043a1c0, 0xc00045e4a0, 0x7, 0xc00053e060)
/home/runner/work/awsets/awsets/main.go:206 +0x9ba
created by github.com/trek10inc/awsets.List
/home/runner/work/awsets/awsets/main.go:171 +0x1e7
Hello,
Would be great to be able to do brew install awssets
. Once this reaches 1.0.0 or when you have time, would you mind updating the distribution mechanism to brew
for Mac binaries?
ref: https://medium.com/@ben.lafferty/deploying-go-cli-applications-316e9cca16a4
Hello,
Discovered this by accident, trying to list only EC2
resources.
./awsets list -o all.json include ec2
scans all the resources, instead of throwing an error../awsets list -o all.json --include ec2
works correctlyThe following is a list of resource type & count from a recent scan. Support for these resources should be prioritized, or they need to be explicitly excluded in the mappings.
AWS::CodeStar::SeedRepository,008
AWS::Route53Resolver::ResolverEndpoint,004
AWS::KinesisAnalytics::ApplicationOutput,006
AWS::DirectoryService::SimpleAD,001
AWS::ApiGateway::Account,023
AWS::EC2::TransitGatewayRoute,005
AWS::EC2::VPNGatewayRoutePropagation,020
AWS::Inspector::AssessmentTarget,004
AWS::S3::AccessPoint,011
AWS::IAM::UserToGroupAddition,014
AWS::CDK::Metadata,006
AWS::CodeStar::SyncResources,029
AWS::Route53Resolver::ResolverQueryLoggingConfig,001
AWS::RDS::DBSecurityGroup,003
AWS::ServiceCatalog::LaunchRoleConstraint,146
AWS::DLM::LifecyclePolicy,024
AWS::Inspector::ResourceGroup,001
AWS::Chatbot::SlackChannelConfiguration,005
AWS::EC2::ClientVpnEndpoint,003
AWS::Route53Resolver::ResolverRule,010
AWS::SecretsManager::SecretTargetAttachment,006
AWS::KinesisAnalytics::Application,006
AWS::RDS::EventSubscription,004
AWS::CodeStarNotifications::NotificationRule,006
AWS::Route53Resolver::ResolverRuleAssociation,010
AWS::OpsWorksCM::Server,001
AWS::RDS::OptionGroup,013
AWS::ServiceDiscovery::Service,006
AWS::DirectoryService::MicrosoftAD,002
AWS::ServiceCatalog::LaunchNotificationConstraint,140
AWS::ServiceCatalog::PortfolioProductAssociation,146
AWS::Route53::RecordSetGroup,148
AWS::ElasticLoadBalancingV2::ListenerCertificate,009
AWS::IAM::ServiceLinkedRole,035
AWS::SecurityHub::Hub,001
AWS::Inspector::AssessmentTemplate,004
AWS::ServiceDiscovery::PrivateDnsNamespace,003
AWS::Route53Resolver::ResolverQueryLoggingConfigAssociation,001
AWS::StepFunctions::Activity,003
Hi,
I have several profiles in my .aws/credentials and don't have the [default] one.
So, I can't choose which profile to use (for example like in aws-cli) and have an error:
2020/09/17 14:37:23 unable to load regions: failed to query regions: EC2RoleRequestError: no EC2 instance role found
caused by: exceeded maximum number of attempts, 3, request send failed, Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp 169.254.169.254:80: connect: host is down
It's because after checking default profile in .aws/credentials and envs - it tries to use InstanceRole.
I think it will be very convenient (for example):
awsets list --include iam/user --profile dev
Do you think the --profile
arg will be useful?
Hi,
Awesome work on this project!
I've read the documentation and I didn't find a way to run awsets list -o all.json
without having aws cli configured (most probably I'm missing something).
How we can run awsets list -o all.json
without aws cli configured?
I would gladly help with work on this project, but I'm still a beginner in Go..
Update: Or role-arn?
The following Listers returned data from rds and are "Type"ing as their own.
docdb/cluster
returns data."Type": "docdb/cluster"
.rds/*
resource information and are "Type"ing as their own.awsets list --profile "my-profile" --show-progress --regions "us-east-1" --include "docdb" --output "docdb-output.json"
awsets list --profile "my-profile" --show-progress --regions "us-east-1" --include "neptune" --output "neptune-output.json"
awsets - version: 1.0.4 commit: ba77e77d0c6b00fbdd59712a0b4d08c49f2d9b61 date: 2021-12-10T17:21:29Z
Hi
the code currently supports AWS_PROFILE env var, do you think you could like that back to the name of the cache file, or smuggle account number into there somewhere?
I would love to have a cache of a few of the clients i work on, at the moment I have to do a little bit of files/folder manipulation to add this 'external' metadata.
Open to your thoughts.
Terry
After installing GO and then awsets, it does not work as a command.
GO111MODULE="on" go get github.com/trek10inc/awsets/cmd/awsets
Is there another step before we can run it as a command?
❯ GO111MODULE="on" go get github.com/trek10inc/awsets/cmd/awsets
go: github.com/trek10inc/awsets/cmd/awsets upgrade => v0.0.0-20200903202143-ec27b76811e6
❯ awsets --help
zsh: command not found: awsets
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.