vbmacher / aws-scripts Goto Github PK

in account / emr there should be a "special" configuration allowing defining custom instance fleet configs

currently only instance fleets are available

it is not good that we have several root scripts, like emr-start, codeartifact, etc. There should be one consolidated name pointing to aws scripts and all other services should be sub-commands, exactly as AWS CLI has.

Options name which can be considered:

• awsome [service] [opions]
• a [service] [options]
• as [service] [options]
• awsscripts [service] [options] - favourite
• aws2 [service] [options]
• awsex [service] [options]
• awsc [service] [options] - favourite

AWSome scripts should utilize configuration files instead of CLI parameters as AWS CLI does. One configuration file should contain default values for various AWS services which are usually repeated over and over, like subnets, security groups, etc.

It can be said that one configuration file describes values for one "account" or "profile". This kind of accounts/profiles are different to AWS profiles. Maybe there is better name for this concept to not interfere with AWS naming, but I'm out of ideas right now. The name of a configuration file defines one account/profile.

Configuration files should be stored in a directory, best under user home, e.g.: ~/.aws-scripts/accounts/.
As was mentioned, there can be multiple files in the config directory; each file represents an account. It's probably better to store accounts in a subdirectory /accounts since there might be a global config file for aws-scripts in the future.

Also, there should be possibility to define default profile. One idea is - if there exists an account with file name default.yaml(format TB determined) this one can be considered as default account. The default file can be created by symbolic link to real account file name.

Last thing - the management of configuration files (accounts) should be done with a script. The manipulation of accounts - CRUD and making default account should be fully automatic. Users should be just editing the files and filling the values in their accounts. We shall name the script aws-accounts

it should have a DSL able to:

• create cluster
• EMR step submiting
• terminate cluster
• copy data across S3, HDFS
• delete data based on dates (date=YYYY-MM-DD or year=xxxx/month=xx/day=xx)
• iteration of EMR step submitting based on date range

For example, if account doesnt exist or if the AWS service is not configured, print an error message instead of throwing an exception

it is buggy and git-based dynamic versioning doesnt work. Use standard setup instead