Start, Reboot, Stop and Back Up AWS Resources with Tags

Benefits

Save money by stopping EC2 instances and RDS databases during off-hours
Take backups more frequently
Use tags to schedule operations
Secure tags and backups using Identity and Access Management (IAM) policies
Install and update easily, with CloudFormation (and optionally, StackSets)

Jump to: Installation • Operation Tags • Schedule Tags • Logging • Security • Multi-region/multi-account

Comparison with Lifecycle Manager

In July, 2018, Amazon introduced Data Lifecycle Manager. It's a start, but...

Tags determine which volumes will be backed up, not when. There is a new, single-purpose API for scheduling.
Snapshots are taken only every 12 or 24 hours.
"Last x" snapshot retention is not flexible enough for true archival policies (e.g., keep daily snapshots for a month, plus monthly snapshots for a year).
You can create snapshots of single volumes, but not images covering all of an EC2 instance's volumes. Also missing is an option to reboot first.
The same IAM role confers authority to create and delete snapshots, and anyone who can update a lifecycle policy can reduce the retention period to delete snapshots. These are significant risks, especially in light of the data loss prevention provisions in the European Union General Data Protection Regulation.

By all means, set up Data Lifecycle Manager if you have no automation in place, but consider this project for more flexibility -- and sign up as a contributor if you would like to help develop the backup retention part!

Quick Start

Log in to the AWS Web Console as a privileged user.

Security Tip: To see what you'll be installing, look in the CloudFormation template.
grep 'Type: "AWS::' aws_tag_sched_ops.yaml | sort | uniq works well.
Go to Instances in the EC2 Console. Right-click the Name or ID of an instance, select Instance Settings, and then select Add/Edit Tags. Add:

Key Value Note

managed-image Leave value blank

managed-image-periodic d=*,H:M=11:30 Replace 11:30 with current UTC time + 20 minutes
Go to the S3 Console. Click the name of the bucket where you keep AWS Lambda function source code. (This may be the same bucket where you keep CloudFormation templates.) If you are creating the bucket now, be sure to create it in the region where you intend to install TagSchedOps; appending the region to the bucket name (for example, my-bucket-us-east-1) is recommended. Upload the compressed source code of the AWS Lambda function, aws_tag_sched_ops_perform.py.zip

Security Tip: Remove public read and write access from the S3 bucket. Carefully limit write access.

Security Tip: Download the file from S3 and verify it. (In some cases, you can simply compare the ETag reported by S3.)
md5sum aws_tag_sched_ops_perform.py.zip should match aws_tag_sched_ops_perform.py.zip.md5.txt
Go to the CloudFormation Console. Click Create Stack. Click Choose File, immediately below "Upload a template to Amazon S3", and navigate to your locally downloaded copy of cloudformation/aws_tag_sched_ops.yaml. On the next page, set:

Item Value

Stack name TagSchedOps

LambdaCodeS3Bucket Name of your S3 bucket

MainRegion Current region, if other than us-east-1

For all other paramters, keep the default values.
After 20 minutes, check Images in the EC2 Console.
Before deregistering (deleting) the sample image, note its ID, so that you can delete the associated Snapshots. Also untag the instance.
Go to Users in the IAM Console. Click your regular (uprivileged) username. Click Add Permissions, then click "Attach existing policies directly". In the Search box, type TagSchedOpsAdminister. Add the two matching policies.

Security Tip: Review everyone's EC2 and RDS tagging privileges!
Log out of the AWS Console. You can now manage relevant tags, view logs, and decode errors, without logging in as a privileged user.

Key	Value	Note
`managed-image`		Leave value blank
`managed-image-periodic`	`d=*,H:M=11:30`	Replace `11:30` with current UTC time + 20 minutes

Item	Value
Stack name	`TagSchedOps`
LambdaCodeS3Bucket	Name of your S3 bucket
MainRegion	Current region, if other than `us-east-1`

Warnings

Check that scheduled AWS operations have completed successfully. Verification of completion is beyond the scope of this code.
Test your backups! Can they be restored successfully?
Weigh the benefits of rebooting against the risks. Rebooting is usually necessary to make software updates take effect, but a system may stop working afterward.
Be aware of AWS charges, including but not limited to: the costs of running the AWS Lambda function, storing CloudWatch logs, and storing images and snapshots; the whole-hour cost when you stop an RDS, EC2 Windows, or EC2 commercial Linux instance (but most EC2 instances have a 1-minute minimum charge); the continuing cost of storage for stopped instances; and costs that resume when AWS automatically starts an RDS instance that has been stopped for too many days.
Secure your own AWS environment. Test the function and the IAM policies from end-to-end, to make sure that they work correctly and meet your expectations. To help improve this project, please submit bug reports and feature requests, as well as proposed changes.

Enabling Operations

To enable an operation, add a tag from the table. Leave the value blank.

	Start	Create Image	Reboot then Create Image	Reboot then Fail Over	Reboot	Create Snapshot	Create Snapshot then Stop	Stop
EC2 compute instance	`managed-start`	`managed-image`	`managed-reboot-image`		`managed-reboot`			`managed-stop`
EC2 EBS disk volume						`managed-snapshot`
RDS database instance	`managed-start`			`managed-reboot-failover`	`managed-reboot`	`managed-snapshot`	`managed-snapshot-stop`	`managed-stop`

Also add tags for repetitive (-periodic) and/or one-time (-once) schedules. Prefix with the operation.
If there are no corresponding schedule tags, an enabling tag will be ignored, and the operation will never occur.
To temporarily suspend an operation, delete its enabling tag. You may leave its schedule tag(s) in place.

Examples (for an EC2 or RDS instance):

Set of Tags	Works?	Comment
`managed-start` `managed-start-periodic`=`u=1,H=09,M=05`	Yes	Enabled and scheduled
`managed-start`=`No` `managed-start-periodic`=`u=1,H=09,M=05`	Yes	Value of enabling tag is always ignored
`managed-start` `managed-start-once`=`2017-12-31T09:05`	Yes
`managed-start` `managed-start-periodic`=`u=1,H=09,M=05` `managed-start-once`=`2017-12-31T09:05`	Yes	Both repetitive and one-time schedule tags are allowed
`managed-start`	No	No schedule tag
`managed-start-once`=`2017-12-31T09:05`	No	No enabling tag (operation is suspended)
`managed-start` `managed-start-once`	No	Blank schedule
`managed-start` `managed-start-periodic`=`Monday`	No	Invalid schedule
`managed-start` `managed-stop-periodic`=`u=1,H=09,M=05`	No	Enabling tag and schedule tag cover different operations

Scheduling Operations

All times are UTC, on a 24-hour clock.
The function runs once every 10 minutes. The last digit of the minute is always ignored. For example, M=47 means one time, between 40 and 50 minutes after the hour.
Month and minute values must have two digits. Use a leading zero if necessary. (Weekday numbers have only one digit.)
Use a comma (,) or a space ( ) to separate components. (RDS does not allow commas in tag values.) The order of components within a tag value does not matter.
T separates date from time; it is invariable.
Repetitive (-periodic) and one-time (-once) schedule tags are supported. Prefix with the operation.
If the corresponding enabling tag is missing, schedule tags will be ignored, and the operation will never occur.

Repetitive Schedules

Tag suffix: -periodic

Values: one or more of the following components:

Name	Minimum	Maximum	Wildcard	Combines With
Day of month (`d`)	`d=01`	`d=31`	`d=*`	`H` and `M`, or `H:M`
Weekday (`u`)	`u=1` (Monday)	`u=7` (Sunday)		`H` and `M`, or `H:M`
Hour (`H`)	`H=00`	`H=23`	`H=*`	`d` or `u`, and `M`
Minute (`M`)	`M=00`	`M=59`		`d` or `u`, and `H`
Hour and minute (`H:M`)	`H:M=00:00`	`H:M=23:59`		`d` or `u`
Day of month, hour and minute (`dTH:M`)	`dTH:M=01T00:00`	`dTH:M=31T23:59`
Weekday, hour and minute (`uTH:M`)	`uTH:M=1T00:00`	`uTH:M=7T23:59`

To be valid, a component or combination of components must specify a day, hour and minute.
Repeat a whole component to specify multiple values. For example, d=01,d=11,d=21 means the 1st, 11th and 21st days of the month.
The * wildcard is allowed for day (every day of the month) and hour (every hour of the day).
For consistent one-day-a-month scheduling, avoid d=29 through d=31.
Label letters match strftime and weekday numbers are ISO 8601-standard (different from cron).

Examples:

Value of `-periodic` Schedule Tag	Demonstrates	Operation Begins
`d=28,H=14,M=25` or `dTH:M=28T14:25`	Monthly event	Between 14:20 and 14:30 on the 28th day of every month.
`d=1,d=8,d=15,d=22,H=03,H=19,M=01`	`cron`-style schedule	Between 03:00 and 03:10 and again between 19:00 and 19:10, on the 1st, 8th, 15th, and 22nd days of every month.
`d=,H=,M=15,M=45,H:M=08:50`	Extra event in the day	Between 10 and 20 minutes after the hour and 40 to 50 minutes after the hour, every hour of every day, and also every day between 08:50 and 09:00.
`d=*,H=11,M=00,uTH:M=2T03:30,uTH:M=5T07:20`	Extra weekly events	Between 11:00 and 11:10 every day, and also every Tuesday between 03:30 and 03:40 and every Friday between 07:20 and 7:30.
`u=3,H=22,M=15,dTH:M=01T05:20`	Extra monthly event	Between 22:10 and 22:20 every Wednesday, and also on the first day of every month between 05:20 and 05:30.

Remember to use spaces ( ) instead of commas (,) in RDS! (Either separator character is fine in EC2.)

One-Time Schedules

Tag suffix: -once
Values: one or more ISO 8601 combined date and time strings, of the form 2017-03-21T22:40 (March 21, 2017, in this example)
- Remember, the code runs once every 10 minutes and the last digit of the minute is ignored
- Omit seconds and fractions of seconds
- Omit time zone; UTC is always used

Output

After logging in to the AWS Web Console, go to the Log Group for the AWS Lambda function, in the CloudWatch Logs Console. If you gave the CloudFormation stack a name other than TagSchedOps, check the list of Log Groups for all AWS Lambda functions instead.

Sample output:

`initiated`	`rsrc_id`	`op`	`child_rsrc_type`	`child`	`child_op`	`note`
`9`						`2017-09-12T20:40`
`1`	`i-08abefc70375d36e8`	`reboot-image`	`Image`	`zm-my-server-20170912T2040-83xx7`
`1`	`i-08abefc70375d36e8`	`reboot-image`	`Image`	`ami-bc9fcbc6`	`tag`
`1`	`i-04d2c0140da5bb13e`	`start`
`0`	`i-09cdea279388d35a2`	`start,stop`				`OPS_UNSUPPORTED`
`0`	`my-database`	`reboot-failover`				...`ForceFailover cannot be specified`...

This run began September 12, 2017 between 20:40 and 20:50 UTC. An EC2 instance (ID prefix i-) is being rebooted and backed up, but the instance may not yet be ready again, and the image may not yet be complete; the image is named zm-my-server-20170912T2040-83xx7. The image has received ID ami-bc9fcbc6, and has been tagged. A different EC2 instance is starting up, but may not yet be ready. A third EC2 instance is tagged for simultaneous start and stop, a combination that is not supported. An RDS database instance (no i- or vol- ID prefix) could not be rebooted with fail-over. (The full error message goes on to explain that it is not multi-zone.)

There is a header line, an information line, and one line for each operation requested. (Tagging is usually a separate operation.)
Values are tab-separated (but the CloudWatch Logs Console seems to collapse multiple tabs).

Columns and standard values:

`initiated`	`rsrc_id`	`op`	`child_rsrc_type`	`child`	`child_op`	`note`
Operation initiated?	Resource ID	Operation	Child type	Pointer to child	Child operation	Message
`0` No `1` Yes `9` Info. only	`i-` EC2 instance ID `vol-` EBS volume ID RDS instance name	See table	`Image` `Snapshot` `DBSnapshot`	Name, ID, or ARN, as available	`tag`

Although the TagSchedOpsAdminister and TagSchedOpsTagSchedule policies authorize read-only access to the logs via the AWS API, and seem to be sufficient for using the links provided above, users who are not AWS administrators may also want additional privileges for the CloudWatch Console.

Debugging Mode

If the DEBUG environment variable is set, the function outputs internal reference data, including the regular expressions used to match schedule tags.

To use the debugging mode,

Log in to the AWS Web Console as a privileged user. AWS Lambda treats changes to environment variables like changes to code.
Click on the TagSchedOpsPerformLambdaFn AWS Lambda function.
Open the Code tab and scroll to the bottom. In the "Environment variables" section, type DEBUG in the first empty Key box. Leave Value blank.
Scroll back to the top and click the white Save button. Do not click the orange "Save and test" button; that would cause the function to run more than once in the same 10-minute interval.
After 10 minutes, find the debugging information in CloudWatch Logs.
Turn off debugging mode right away, because the extra information is lengthy. Back on the Code tab, scroll down and click Remove, to the far right of DEBUG. Repeat Step 4 to save.

On/Off Switch

The TagSchedOpsAdminister policies authorize turning the function on or off completely.
After logging in to the AWS Web Console, go to Rules in the CloudWatch Events Console. Click the radio button to the left of TagSchedOpsPerform10MinEventRule, then select Enable or Disable from the Actions pop-up menu, next to the blue "Create rule" button.
This toggle is per-region and per-AWS-account.
Missed operations will not occur when the function is turned back on; there is no queue or backlog.

Child Resources

Some operations create a child resource (image or snapshot) from a parent resource (instance or volume).

Naming

The name of the child consists of these parts, separated by hyphens (-):

#	Part	Example	Purpose
1	Prefix	`zm`	Identifies and groups children created by this project, for interfaces that do not expose tags. `z` will sort after most manually-created images and snapshots. `m` stands for "managed".
2	Parent name or identifier	`webserver`	Conveniently indicates the parent. Derived from the `Name` tag (if not blank), the logical name (if supported), or the physical identifier (as a last resort). Multiple children of the same parent will sort together, by creation date (see next row).
3	Date/time	`20171231T1400Z`	Indicates when the child was created. Always includes the `Z` suffix to indicate UTC. The last digit of the minute is normalized to 0. The `-` and `:` separators are removed for brevity, and because AWS does not allow `:` in resource names. (The `managed-date-time` tag preserves the separators.)
4	Random string	`g3a8a`	Guarantees unique names. Five characters are chosen from a small set of unambiguous letters and numbers.

If parsing is ever necessary, keep in mind that the second part may contain internal hyphens.
This project replaces characters forbidden by AWS with X.
For some resource types, the description is also set to the name, in case interfaces expose only one or the other.

Special Tags

Special tags are added to the child:

Tag(s)	Purpose
`Name`	Supplements EC2 resource identifier. The key is renamed `managed-parent-name` when the value is passed from parent to child, because the child has a `Name` tag of its own. The code handles `Name` specially for both EC2 and RDS, in case AWS someday extends EC2-style tag semantics to RDS.
`managed-parent-name`	The `Name` tag value from the parent. May be blank.
`managed-parent-id`	The identifier of the parent instance or volume. AWS stores this in metadata for some but not all resource types, and the retrieval key differs for each resource type.
`managed-origin`	The operation (for example, `snapshot`) that created the child. Identifies resources created by this project. Also distinguishes special cases, such as whether an EC2 instance was or was not rebooted before an image was created.
`managed-date-time`	Groups resources created during the same 10-minute interval. The last digit of the minute is normalized to 0, and `Z` is always appended, to indicate UTC. AWS stores the exact time (too specific for grouping) in metadata, and the retrieval key and the format differ for each resource type!

Tags other than operation-enabling tags, schedule tags, and the Name tag, are copied from parent to child. (The deletion tag, managed-delete, would not make sense on instances and volumes, but if it is present, it is not copied to images and snapshots.)

Operation Combinations

Multiple non-simultaneous operations on the same resource are allowed.

Supported Combinations

If two or more operations on the same resource are scheduled for the same 10-minute interval, the function combines them, where possible:

Resource	Simultaneous Operations	Effect
EC2 instance	Create Image + Reboot	Reboot then Create Image
EC2 or RDS instance	Stop + Reboot	Stop
RDS instance	Stop + Create Snapshot	Create Snapshot then Stop

The Create Image + Reboot combination for EC2 instances is useful. For example, you could take hourly backups but reboot only in conjunction with the midnight backup. The midnight backup would be guaranteed to be coherent for all files, but you could safely retrieve static files as of any given hour, from the other backups. To set up this example:

Tag	Value
`managed-image`
`managed-image-periodic`	`d=,H=,M=59`
`managed-reboot`
`managed-reboot-periodic`	`d=*,H=23,M=59`

23:59, which for the purposes of this project represents the last 10-minute interval of the day, is the unambiguous way to express almost the end of some designated day, on any system. 00:00 and 24:00 could refer to the start or the end of the designated day, and not all systems accept 24:00, in any case. Remember that all times are UTC; adjust for night-time in your time zone!

Unsupported Combinations

Resources tagged for unsupported combinations of operations are logged (with message OPS_UNSUPPORTED) and skipped.

Bad Combination	Reason	Example
Mutually exclusive operations	The operations conflict.	Start + Stop
Choice of operation depends on current state of instance	The state of the instance could change between the status query and the operation request.	Start + Reboot
Sequential or dependent operations	The logical order cannot always be inferred. Also, operations proceed asynchronously; one might not complete in time for another to begin. Note that Reboot then Create Image (EC2 instance) and Create Snapshot then Stop (RDS instance) are single AWS operations.	Start + Create Image

Security Model

Prevent unauthorized changes to the AWS Lambda function by attaching the TagSchedOpsPerformLambdaFnProtect IAM policy to most entities with write privileges for:
- AWS Lambda
- CloudFormation Events
- CloudFormation Logs
- IAM (roles and/or policies)
Allow only a few trusted users to tag EC2 and RDS resources, because tags determine which resources are started, backed up, rebooted, and stopped.
Tag backups for deletion, but let a special IAM user or role actually delete them. To mark images and snapshots for (manual) deletion, add the managed-delete tag.
Do not allow the same entities that create backups to delete backups (or even to tag them for deletion).

Choose from a library of IAM policies:

Policy Name	Manage Enabling Tags	Manage One-Time Schedule Tags	Manage Repetitive Schedule Tags	Back Up	Manage Deletion Tag	Delete
Scope →	Instances, Volumes	Instances, Volumes	Instances, Volumes	Instances, Volumes	Images, Snapshots	Images, Snapshots
TagSchedOpsAdminister	Allow	Allow	Allow	No effect	Deny	Deny
TagSchedOpsTagScheduleOnce	Deny ⁱ	Allow ⁱⁱ	Deny	No effect	Deny	Deny
TagSchedOpsTagSchedulePeriodic	Deny ⁱ	No effect	Allow ⁱⁱ	No Effect	Deny	Deny
TagSchedOpsTagForDeletion	Deny	Deny	Deny	Deny	Allow	Deny
TagSchedOpsBackupDelete	Deny	Deny	Deny	Deny	Deny	Allow
TagSchedOpsNoTag	Deny	Deny	Deny	No effect	Deny	Deny

Footnotes:

For RDS, No Effect.
Enabling tag required. For example, a user could only add managed-image-once to an EC2 instance already tagged with managed-image.

These policies cover all regions. If you use regions to differentiate production and non-production resources, copy the policies and adapt them.

Because Deny always takes precendence in IAM, some policy combinations conflict.

A known shortcoming is that, in some cases, you cannot add, change or delete more than one tag in the same operation.

Although the TagSchedOpsAdminister and TagSchedOpsTag policies authorize tagging via the AWS API, users who are not AWS administrators may also want:
- AmazonEC2ReadOnlyAccess, to use the EC2 Console
- AmazonRDSReadOnlyAccess, to use the RDS Console
You may have to decode authorization errors. The TagSchedOpsAdminister and TagSchedOpsTag policies grant the necessary privilege.
Note these AWS technical limitations/oversights:
- An IAM user or role that can create an image of an EC2 instance can force a reboot by omitting the NoReboot option. (Explicitly denying the reboot privilege does not help.) The unavoidable pairing of a harmless privilege, taking a backup, with a risky one, rebooting, is unfortunate.
- Tags are ignored when deleting EC2 images and snapshots. Limit EC2 image and snapshot deletion privileges -- even Ec2TagSchedOpsDelete -- to highly-trusted entities.
- In RDS, an IAM user or role that can add specific tags can add any other tags in the same request. Limit RDS tagging privileges -- even the provided policies -- to highly-trusted users.

Advanced Installation

Before starting a multi-region and/or multi-account installation, check all regions, in all AWS accounts, for TagSchedOps CloudFormation stacks created using the Quick Start instructions. Delete them now.

Multi-Region Configuration

If you intend to install TagSchedOps in multiple regions,

Create S3 buckets in all regions where you intend to install TagSchedOps. The bucket names must share the same prefix, followed by a hyphen (-) and a suffix for the region. The region in which each bucket is created must match the suffix in the bucket's name.
Upload aws_tag_sched_ops_perform.py.zip to each bucket. The need for copies in multiple regions is an AWS Lambda limitation.

Keep the following rules in mind when setting parameters, later:

Parameter	Value
LambdaCodeS3Bucket	Use the shared prefix; for example, if you created `my-bucket-us-east-1` and `my-bucket-us-west-2` , use `my-bucket`
MainRegion	Always use the same value, to prevent the creation of duplicate sets of user policies
StackSetsOrMultiRegion	Yes
TagSchedOpsPerformCodeS3VersionID	Leave blank, because the value would differ in every region; only the latest version of the AWS Lambda function source code file in each region's S3 bucket can be used

Multi-Account Configuration

If you intend to install TagSchedOps in multiple AWS accounts,

In every target AWS account, create cloudformation/tag-sched-ops-install.yaml. Set:

Item	Value
Stack name	`TagSchedOpsInstall`
AWSCloudFormationStackSetExecutionRoleStatus	Choose carefully!
AdministratorAccountId	AWS account number of main (or only) account; leave blank if AWSCloudFormationStackSetExecutionRole existed before this stack was created
LambdaCodeS3Bucket	Name of AWS Lambda function source code bucket (shared prefix, in a multi-region scenario)

For the AWS Lambda function source code S3 bucket in each region, create a bucket policy allowing access by every target AWS account's AWSCloudFormationStackSetExecutionRole (StackSet installation) or TagSchedOpsCloudFormation role (manual installation with ordinary CloudFormation). The full name of the TagSchedOpsCloudFormation role will vary; for every target AWS account, look up the random suffix in the list of IAM roles or by selecting the TagSchedOpsInstall stack in the list of CloudFormation stacks and drilling down to Resources. S3 bucket policy template:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::TARGET_AWS_ACCOUNT_NUMBER_1:role/AWSCloudFormationStackSetExecutionRole",
          "arn:aws:iam::TARGET_AWS_ACCOUNT_NUMBER_1:role/TagSchedOpsInstall-TagSchedOpsCloudFormation-RANDOM_SUFFIX_1",

          "arn:aws:iam::TARGET_AWS_ACCOUNT_NUMBER_2:role/AWSCloudFormationStackSetExecutionRole",
          "arn:aws:iam::TARGET_AWS_ACCOUNT_NUMBER_2:role/TagSchedOpsInstall-TagSchedOpsCloudFormation-RANDOM_SUFFIX_2"

        ]
      },
      "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion"
      ],
      "Resource": "arn:aws:s3:::BUCKET_NAME/*"
    }
  ]
}

CloudFormation StackSet Installation

Follow the multi-region steps, even for a multi-account, single-region scenario.
Follow the multi-account steps. In a single-account, multi-region scenario, no S3 bucket policy is needed.
If StackSets has never been used, create AWSCloudFormationStackSetAdministrationRole. Do this one time, in your main (multi-account scenario) or only (single-account scenario) AWS account. There is no need to create AWSCloudFormationStackSetExecutionRole using Amazon's template; the TagSchedOpsInstall stack provides it, when necessary.
In the AWS account with the AWSCloudFormationStackSetAdministrationRole, go to the StackSets Console.

Click Create StackSet, then select "Upload a template to Amazon S3", then click Browse and select your locally downloaded copy of cloudformation/aws_tag_sched_ops.yaml. On the next page, set:

Item	Value
StackSet name	`TagSchedOps`
LambdaCodeS3Bucket	Use the shared prefix; for example, if you created `my-bucket-us-east-1` , use use `my-bucket`
MainRegion	Must be a StackSet target region
StackSetsOrMultiRegion	Yes
TagSchedOpsPerformCodeS3VersionID	In a multi-region scenario, leave blank

On the next page, specify the target AWS accounts, typically by entering account numbers below "Deploy stacks in accounts". Then, move the target region(s) from "Available regions" to "Deployment order". It is a good idea to put the main region first.

Manual Installation

Manual installation is adequate if the number of installations is small, but keeping more than one installation up-to-date could be difficult.

Follow the multi-region steps, if applicable.
Follow the multi-account steps, if applicable.

Follow the Quick Start installation steps in each target region and/or target AWS account. Set parameters based on the multi-region and/or multi-account rules.

In a multi-account scenario, select the TagSchedOpsCloudFormation IAM Role during this step; CloudFormation may not be able to create the stack without assuming this role.

If the user invoking CloudFormation is not an administrator, attach the following policies to the user beforhand:

Policy	Source
AmazonS3FullAccess	AWS
AmazonSNSReadOnlyAccess	AWS
IAMReadOnlyAccess	AWS
TagSchedOpsCloudFormationRolePass	TagSchedOpsInstall stack
CloudFormationFullAccess	TagSchedOpsInstall stack

Detach these policies -- particularly AmazonS3FullAccess and CloudFormationFullAccess -- after the stack has been created. AWS does not publish the minimum privileges needed to create a stack. Full S3 access is obviously risky, and full CloudFormation access allows a non-administrator to modify or delete any stack with an IAM Role.

Software Updates

New versions of the AWS Lambda function source code and the CloudFormation template will be released from time to time.

CloudFormation Stack Update

Log in to the AWS Web Console as a privileged user.
Go to the S3 Console. Click the name of the bucket where you keep CloudFormation templates and their dependencies. Open the Properties tab. If Versioning is disabled, click anywhere inside the box, select "Enable versioning", and click Save.
Open the Overview tab. Upload the latest version of aws_tag_sched_ops_perform.py.zip to S3.
Click the checkbox to the left of the newly-uploaded file. In the window that pops up, look below the Download button and reselect "Latest version". In the Overview section of the pop-up window, find the Link and copy the text after versionId=. (The Version ID will not appear unless you expressly select "Latest version".)

Security Tip: Download the file from S3 and verify it. (In some cases, you can simply compare the ETag reported by S3.)
md5sum aws_tag_sched_ops_perform.py.zip should match aws_tag_sched_ops_perform.py.zip.md5.txt
Go to Stacks in the CloudFormation Console. Click the checkbox to the left of TagSchedOps (you might have given the stack a different name). From the Actions pop-up menu next to the blue Create Stack button, select Create Change Set For Current Stack.
Click Choose File, immediately below "Upload a template to Amazon S3", and navigate to your locally downloaded copy of the latest version of cloudformation/aws_tag_sched_ops.yaml. On the next page, set:

Item Value

Change set name Type a name of your choice

TagSchedOpsPerformCodeS3VersionID Paste the Version ID, from S3

A different S3 Version ID makes CloudFormation recognize new AWS Lambda function source code. Once you are familiar with the full update procedure, you may skip Steps 2-5 and leave TagSchedOpsPerformCodeS3VersionID as it was, when you are certain that only the CloudFormation template, not the function source code, is changing.
Click through the remaining steps. Finally, click "Create change set".
In the Changes section, check the Replacement column.

If True is shown on any row, check the Logical ID.
1. If the resource is for internal use, ignore it.
2. If, however, it a user policy, such as TagSchedOpsAdminister, open another Web browser window, go to Policies in the IAM Console, click the name of the policy, open the "Attached entities" tab, and detach the policy from all entities. Keep notes!
Click Execute, below the top-right corner of the CloudFormation Console window.
Refresh until the stack's status shows UPDATE_COMPLETE, in green.
If you had to detach any IAM policies, return to the IAM Console and attach the replacement policies to the original entities.
If TagSchedOps is installed in multiple regions, repeat the update steps in each region. The S3 Version IDs will differ.

Item	Value
Change set name	Type a name of your choice
TagSchedOpsPerformCodeS3VersionID	Paste the Version ID, from S3

CloudFormation StackSet Update

Differences when updating a StackSet instead of an ordinary stack:

Click the radio button to the left of TagSchedOps, in the list of StackSets. From the Actions pop-up menu next to the blue Create StackSet button, select "Manage stacks in StackSet". Then, select "Edit stacks". On the next page, select "Upload a template to Amazon S3" and upload the latest version of cloudformation/aws_tag_sched_ops.yaml.
A single update covers all target regions and/or AWS target accounts.
The TagSchedOpsPerformCodeS3VersionID parameter must remain blank. So that CloudFormation will recognize new source code for the AWS Lambda function, rename aws_tag_sched_ops_perform.py.zip to aws_tag_sched_ops_perform_20170924.py.zip (substitute current date) before uploading the file to the regional S3 bucket(s). Change the TagSchedOpsPerformCodeName parameter accordingly.
Change Sets are not supported. There is no preliminary feedback about the scope of changes.

Future Work

Automated testing, consisting of a CloudFormation template to create sample AWS resources, and a program (perhaps another AWS Lambda function!) to check whether the intended operations were performed. An AWS Lambda function would also be ideal for testing security policies, while cycling through different IAM roles.
Archival policy system (ISO 8601 standard duration/interval syntax will be used), and automatic application of managed-delete to expired backups. A correct archival policy is not strictly age-based. For example, you might preserve the last 30 daily backups, and beyond 30 days, the first backup of every month. Consider the flaw in the snapshot retention property of RDS database instances: the daily automatic snapshots created when that property is set can never be kept longer than 35 days.
Additional AWS Lambda function, to automatically delete backups tagged managed-delete
Makefile
Tags and reference dictionary updates to support scheduled restoration of images and snapshots (for backup testing?)

Dedication

This work is dedicated to Ernie Salazar, Régis and Marianne Marcelin, and my wonderful colleagues of the past few years.

Licensing

Scope	License	Copy Included
Source code files	GNU General Public License (GPL) 3.0	zlicense-code.txt
Source code within documentation files	GNU General Public License (GPL) 3.0	zlicense-code.txt
Documentation files (including this readme file)	GNU Free Documentation License (FDL) 1.3	zlicense-doc.txt

Contact: marcelin at cmu.edu (replace at with @)

thiagoalexandria / aws-tag-sched-ops Goto Github PK

aws-tag-sched-ops's Introduction