Schedule AMI cleanup in AWS accounts using CDK

Amazon Machine Images (AMIs) and underlying Amazon Elastic Block Store (Amazon EBS) Snapshots are often created automatically based on schedule or other automated process. Old AMIs that are no longer in use and left unattended for a long time can add unnecessary costs. Identifying and clearing up these unused AMIs and related snapshots manually is time consuming. This Serverless event driven pattern helps in identifying those unused images and snapshots and de-registers them on a scheduled basis. This helps in reducing snapshot costs.

This pattern creates the required AWS Lambda services using AWS Cloud Development Kit (AWS CDK) that runs on a schedule to identify those unused AMI based on set rules and cleans up automatically.
This pattern, also uses tags and or, dictionary, Max age configuration to identify pattern, fresh/master AMIs to be excluded from the cleanup process.
This pattern describes how to use this code to automatically get the snapshot ids and deregister the older AMI associated with it after deleting the snapshots and send notifications.

Table Of Contents

Limitations
Requirements
Target Architecture
Automation and scale
Costs
Code
Implementation
Troubleshooting
Related Resources
Additional information

Limitations

Targeted AMIs will be scoped to the account where the pattern is being deployed.
This pattern does not support AMI’s created from other region/account

Requirements

An Understanding of AWS CDK and usage.
An active AWS account
AWS CDK CLI 2.0 +
Python CLI 3.8+
A web browser that is supported for use with the AWS Management Console. (See the list of supported browsers)

Target Architecture

The diagram shows the following workflow:

AWS CDK stack deploys the Amazon Amazon EventBridge, AWS Lambda, Amazon SNS.
Amazon EventBridge triggers the Lambda based on the schedule defined.
AWS Lambda identifies the AMIs based on the rule set and then it removes those AMIs and sends email notification.
Logs are written to Amazon CloudWatch.

Automation and scale

This pattern creates a event driven serverless architecture that scales automaticly.
Code is managed and deployed by AWS CDK, backed by AWS CloudFormation stack.
More information on Scaling and Concurrency in Lambda can be found here.
How to increase concurrency Limit?

Costs

Costs for this solution (in us-east-1) include:

$0/month for Lambda to execute once a day to clean up 10 AMIs takes approximately 4845 ms, which falls within the free tier (400000 free tier GB-s).
More Pricing details can be found here.

Code

The code for this pattern is available on GitHub, in the ami_cleanup_script_cdk repository. You can also configure the code to setup exclusions and schedules.

The code repository contains the following files and folders:
app.py – The Python script file to deploy CDK Application
ami_clean_up folder – Contains all related CDK Modules used in the project
config folder – contains config.json file.
config/config.json – Contains all parameters and parameter values. You update this file to change parameter values, as described in the Epics section.
lambda_cleaner folder – contains python files for AWS Lambda function

Implementation

Setup your environment

Git can be downloaded here
To pull down the repo via ssh, run
- git clone https://github.com/kalyansundars/AMI-cleanup-in-AWS-accounts-using-CDK.git
- This creates a folder named eks-cicd
- Run cd awscdk_ami_cleanup_script
- Set up the Python virtual environment by running python3 -m venv .venv
- if you are using Unix, active the virtualenv with: source .venv/bin/activate
- If you are using Windows, activate the virtualenv with: .\venv\Scripts\activate.bat
- Once the virtualenv is activated, you can install the required dependencies with:
- pip3 install -r requirements.txt
Update Config
- To create the stack for a specific region, open config/config.json file.
- Update config.json file with AWS Region, Cron Schedule, rule and email address to be notified
- Schema for config.json can be found here
- If this is your first time running CDK, ensure you bootstrap the environment with cdk bootstrap. AWS Credentials for CDK can be provided through environment variables or an AWS CLI profile.

Installation

login to linux instance where the code is cloned and python environment is setup
Execute the following command to create the CDK stack that creates the necessary resources for clean up

$cdk synth
$cdk deploy

Verification

Verification of Stack creation
- Login to AWS console
- Navigate to cloudformation Service
- Look for name “AmiCleanUpStack”
- Verify stack creation under “resources” tab
Verify AMi & Snapshot Deletion
- Login to AWS console
- Open EC2 service and Amazon CloudWatch
- Once the Amazon EventBridge triggers the AWS Lambda based on the schedule, verify the deletion of AMIs and associated snapshots on EC2 service.
- Logs are written to Amazon CloudWatch and email is triggered based on configuration with list of deleted AMIs and snapshots.

######## Below List of AMIs will be removed ############
ami-xxxxxxxxxxxxxx
########################################################
Deregistering: <ami-name> ami-xxxxxxxxxxxxxx 
########################################################
Searching Associated Snapshots for removal...
Deleting Snapshot: snap-xxxxxxxxxxxxxx ami-xxxxxxxxxxxxxx
--

Clean up
- To tear down the solution, run cdk destroy. This should remove all nested stacks. In the AWS CloudFormation console verify AmiCleanUpStack is no longer present in the region in which it was deployed.

Troubleshooting

Issue	Solution
CAWS Lambda execution error	User can check Amazon Cloudwatch logs for AWS Lambda execution for any error.

Related Resources

Additional information

Addition configuration such as age of AMI’s, tags, regex pattern for AMI, and cron schedule can be setup under lambda_clean_up.py