webdigi / aws-ami-automated-creation-deletion

Automated AMI creation & deletion using serverless AWS Lambda

AWS AMI Automated Creation & Deletion System

A simple AWS lambda project to help automate creation and deletion of AMIs. The scripts are written in JavaScript and will run on the the serverless AWS Lambda platform.

Motivation

Creating an AMI automatically snapshots all the associated EBS volumes for that instance. This makes instance recovery much more reliable and faster. A daily or weekly backup schedule is recommended for instances and to make sure you have a backup if ever needed. The system also removes the automatically created AMIs and any associated snapshots as per the settings.

Update 31 March 2017 - Added feature to prevent reboot while creating AMI. Add a tag BackupNoReboot with value true if want to avoid rebooting that instance.
Update December 2018 - Replaced screenshots to reflect AWS Console changes.

Setup / Installation of the Lambda script

1. Go to the AWS Lambda Console and click Create function.
   

2. Name the function and select Create a custom role.
   

3. Give the custom role a name and paste the contents of roles.json into the edit box.

4. Now the function has been created, and you'll be presented with the lambda configuration screen.

5. Scroll down to the code editor, and remove the stub code you see in the editor.

6. Paste the contents of createAMI.js into the edit box.
   

7. Scroll down further and modify the Basic Settings.
   Provide a description and set the timeout to 5 minutes.
   

8. Scroll back to the top and click Save.
   Add a trigger by selecting CloudWatch Event.
   

9. To set up the trigger select Create a new rule.
   

10. Configure the Rule:

• Provide a rule name and description.
• Set the event pattern or schedule.
  • The cron expression showed below will run at 2AM every day.
• Finish by clicking Add at the bottom.
  

11. Success!
    

12. Create a second function named deleteAMI.
    Follow the same basic steps as above, but this time:

• Use deleteAMI.js
• Use the existing lambda Role.
• Use the existing CloudWatch Rule.

Setting the tags for EC2 instances

Set the tags on the instances you want backed up.

• Backup: yes
• BackupRetentionDays: a positive integer
• BackupNoReboot: true (this tag is optional - reboot will happen unless the tag is present and value is true)

For multiple instances it is easiest to use the Tag Editor.

Notes

• This is another open source project bought to you by Webdigi - Web development, London.
• Please submit your pull requests or suggestions to improve this script.
• You can read about updates to this project on our blog - AWS Instance Automated AMI Creation & Deletion System

aws-ami-automated-creation-deletion's Issues

Option to exclude volumes from AMI

Not a bug/issue but rather a feature request. I'm not sure how to add it myself.
Would like the option to be able to exclude attached (non root) volumes from the AMI. Ideally this would be a list of one or many volumes that can be defined as tags.

No AMI created (snapshots are created)

I have automated AMI creation that runs everyday at 12:00 am. It created associated snapshots, but I am not able to find AMI that should have been created.

Read a particular tag from instance and add the tag to the ami & snapshot created

please me to setup it. I want to read a particular tag from the instance and add the tag to the ami & snapshot created with this script

Lambda Dasboard Update

The lambda dashboard has been updated...

Would you please be able to update your instructions so that I can follow them,

I thought I managed to get it right but, they don't seem to be running yet :/

Thanks

Henry

Syntax Error

I followed all of the instructions, but the AMI wasn't made. In the lambda logs I see this error (for the create script)

Syntax error in module 'index': SyntaxError
at exports.runInThisContext (vm.js:53:16)
at Module._compile (module.js:373:25)
at Object.Module._extensions..js (module.js:416:10)
at Module.load (module.js:343:32)
at Function.Module._load (module.js:300:12)
at Module.require (module.js:353:17)
at require (internal/module.js:12:17)

The only thing I changed in the script was the region.

Issues setting up the deleteAMI function

I've got the createAMI to work but I was wondering what kind of tags you're using to set up the deleteAMI function.

Malformed AMI name

Currently the AMI name is based on the EC2 instance tag name. But the name of the instance allows more characters than the name of the AMI. For the instance I was trying to back up, the script throws this error:

2017-07-06T03:10:32.880Z	aa144caa-61f8-11e7-b3c3-515643d3a81c	{ [InvalidAMIName.Malformed: AMI names must be between 3 and 128 characters long, and may contain letters, numbers, '(', ')', '.', '-', '/' and '_']
message: 'AMI names must be between 3 and 128 characters long, and may contain letters, numbers, \'(\', \')\', \'.\', \'-\', \'/\' and \'_\'',
code: 'InvalidAMIName.Malformed',
time: Thu Jul 06 2017 03:10:32 GMT+0000 (UTC),
requestId: 'd0dc4e92-fe6c-4d3e-84ea-1e9bda700bc7',
statusCode: 400,
retryable: false,
retryDelay: 99.96723912190646 } 'InvalidAMIName.Malformed: AMI names must be between 3 and 128 characters long, and may contain letters, numbers, \'(\', \')\', \'.\', \'-\', \'/\' and \'_\'\n at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/services/ec2.js:50:35)\n at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:105:20)\n at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:77:10)\n at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:678:14)\n at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)\n at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)\n at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10\n at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)\n at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:680:12)\n at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:115:18)'

I'll try submit a pull request to fix this in a bit.

Error

Hi, iam very new to the Lambda, but when i execute your scripts in lambda, here iam getting the Error , please find below
2017-05-07T23:00:14.702Z e6ba23fe-3378-11e7-b141-6384f3daab93 TypeError: Cannot read property 'SnapshotId' of undefined
at null._onTimeout (/var/task/index.js:76:73)
at Timer.listOnTimeout (timers.js:92:15)

Snapshot deletion failing

I get the same error as the other open issue, creating a new one not to confuse discussion.
Here is the log from last night's run.

START RequestId: 298ce261-4ff5-11e7-b20c-116eea350642 Version: $LATEST
2017-06-13T05:00:16.184Z 298ce261-4ff5-11e7-b20c-116eea350642 Time to delete Managed Server - DFW_DATETODEL-1497322822087
2017-06-13T05:00:16.185Z 298ce261-4ff5-11e7-b20c-116eea350642 image that is going to be deregistered: Managed Server - DFW_DATETODEL-1497322822087
2017-06-13T05:00:16.185Z 298ce261-4ff5-11e7-b20c-116eea350642 image id: ami-00cd9416
2017-06-13T05:00:16.185Z 298ce261-4ff5-11e7-b20c-116eea350642 { ImageId: 'ami-00cd9416' }
2017-06-13T05:00:16.367Z 298ce261-4ff5-11e7-b20c-116eea350642 Not yet time to delete Managed Server - DFW_DATETODEL-1497409222824
2017-06-13T05:00:16.367Z 298ce261-4ff5-11e7-b20c-116eea350642 Time to delete Managed Server - DFW_DATETODEL-1497299795687
2017-06-13T05:00:16.367Z 298ce261-4ff5-11e7-b20c-116eea350642 image that is going to be deregistered: Managed Server - DFW_DATETODEL-1497299795687
2017-06-13T05:00:16.367Z 298ce261-4ff5-11e7-b20c-116eea350642 image id: ami-a5eeb0b3
2017-06-13T05:00:16.368Z 298ce261-4ff5-11e7-b20c-116eea350642 { ImageId: 'ami-a5eeb0b3' }
2017-06-13T05:00:16.425Z 298ce261-4ff5-11e7-b20c-116eea350642 Not yet time to delete Managed Server - DFW_DATETODEL-1497582025126
2017-06-13T05:00:16.425Z 298ce261-4ff5-11e7-b20c-116eea350642 Not yet time to delete Managed Server - DFW_DATETODEL-1497495618537
2017-06-13T05:00:16.886Z 298ce261-4ff5-11e7-b20c-116eea350642 Image Deregistered
2017-06-13T05:00:16.945Z 298ce261-4ff5-11e7-b20c-116eea350642 Image Deregistered
2017-06-13T05:00:26.431Z 298ce261-4ff5-11e7-b20c-116eea350642 Time to delete Managed Server - DFW_DATETODEL-1497322822087
2017-06-13T05:00:26.431Z 298ce261-4ff5-11e7-b20c-116eea350642 snap-0ebc3f736cc4176b1
2017-06-13T05:00:26.587Z 298ce261-4ff5-11e7-b20c-116eea350642 snap-0195dc9146f620493
2017-06-13T05:00:26.645Z 298ce261-4ff5-11e7-b20c-116eea350642 snap-034f66a28d8b59186
2017-06-13T05:00:26.646Z 298ce261-4ff5-11e7-b20c-116eea350642 snap-0559aba9dd9dfb868
2017-06-13T05:00:26.707Z 298ce261-4ff5-11e7-b20c-116eea350642 TypeError: Cannot read property 'SnapshotId' of undefined
at null._onTimeout (/var/task/index.js:76:73)
at Timer.listOnTimeout (timers.js:92:15)
END RequestId: 298ce261-4ff5-11e7-b20c-116eea350642
REPORT RequestId: 298ce261-4ff5-11e7-b20c-116eea350642 Duration: 12167.37 ms Billed Duration: 12200 ms Memory Size: 128 MB Max Memory Used: 44 MB
RequestId: 298ce261-4ff5-11e7-b20c-116eea350642 Process exited before completing request

START RequestId: 298ce261-4ff5-11e7-b20c-116eea350642 Version: $LATEST
2017-06-13T05:01:26.805Z 298ce261-4ff5-11e7-b20c-116eea350642 Not yet time to delete Managed Server - DFW_DATETODEL-1497409222824
2017-06-13T05:01:26.805Z 298ce261-4ff5-11e7-b20c-116eea350642 Not yet time to delete Managed Server - DFW_DATETODEL-1497582025126
2017-06-13T05:01:26.805Z 298ce261-4ff5-11e7-b20c-116eea350642 Not yet time to delete Managed Server - DFW_DATETODEL-1497495618537
2017-06-13T05:01:36.817Z 298ce261-4ff5-11e7-b20c-116eea350642 Not yet time to delete Managed Server - DFW_DATETODEL-1497409222824
2017-06-13T05:01:36.817Z 298ce261-4ff5-11e7-b20c-116eea350642 Not yet time to delete Managed Server - DFW_DATETODEL-1497582025126
2017-06-13T05:01:36.817Z 298ce261-4ff5-11e7-b20c-116eea350642 Not yet time to delete Managed Server - DFW_DATETODEL-1497495618537
END RequestId: 298ce261-4ff5-11e7-b20c-116eea350642
REPORT RequestId: 298ce261-4ff5-11e7-b20c-116eea350642 Duration: 11771.28 ms Billed Duration: 11800 ms Memory Size: 128 MB Max Memory Used: 42 MB

I have the following permissions for the IAM role:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:::"
},
{
"Effect": "Allow",
"Action": [
"ec2:CreateImage",
"ec2:CreateTags",
"ec2:DeleteSnapshot",
"ec2:DeregisterImage",
"ec2:DescribeImages",
"ec2:DescribeInstances",
"ec2:CreateNetworkInterface",
"ec2:DescribeNetworkInterfaces",
"ec2:DeleteNetworkInterface"
],
"Resource": ""
}
]
}

It should have found and deleted 8 snapshots, 4 for each AMI:
snap-0195dc9146f620493,
snap-034f66a28d8b59186,
snap-0559aba9dd9dfb868,
snap-0ebc3f736cc4176b1,
snap-0824f5772cbd01b29,
snap-01c6172b946fea805,
snap-0bd52e7b4cad93d71,
snap-00d74ed75f2f9aaa5

EBS Snapshot Retention

This script automates the creation of AMI and EBS snapshots very neatly, and the AMI snapshot retention is working as intended. However the EBS snapshots are being retained indefinitely, is there any reason why it's not included in the deletion?

I've noticed the EBS Snapshot Lifecycle Manager in AWS Console, however it doesn't seem capable of being configured to delete snapshots after a certain time.

I'm currently using this script to create EC2 backups. How are you handling your EBS snapshots?