Purpose:
- Generate and plot statistics for CloudWatch EC2 instance.
- All Metrics and statistics are supported
Included:
-
Statistics: Sum,Maximum,Minimum,SampleCount,Average
-
Metrics: CPUUtilization,NetworkIn,NetworkOut,NetworkPacketsIn, NetworkPacketsOut,DiskWriteBytes,DiskReadBytes,DiskWriteOps, DiskReadOps,CPUCreditBalance,CPUCreditUsage,StatusCheckFailed, StatusCheckFailed_Instance,StatusCheckFailed_System
Wrote using Python/boto3. Compiled using PyInstaller
##Other scripts
- Oracle -> Redshift data loader
- PostgreSQL -> Redshift data loader
- MySQL -> Redshift data loader
- Oracle -> S3 data loader
- CSV -> Redshift data loader
- [EC2 Metcics Plotter] (https://github.com/alexbuz/EC2_Metrics_Plotter/blob/master/README.md)
- [Oracle->Oracle] (https://github.com/alexbuz/TabZilla/blob/master/README.md) data loader.
- [Oracle->MySQL] (https://github.com/alexbuz/Oracle-to-MySQL-DataMigrator/blob/master/README.txt) data loader.
- [CSV->S3] (https://github.com/alexbuz/S3_File_Uploader/blob/master/README.md) data uploader.
##Purpose
- Generate plots for AWS-ECS metrics and statistics.
- Helps you generate plots on demand and review them using generated html report.
- ec2metrics.exe connects to EC2 and reads datapoints for given CloudWatch EC2 instance/metric/statistic combo.
- Using matplotlib plot is created and saved on the filesystem.
- Html report is generated allowing preview saved metric plots.
- It will not work for group CloudWatch EC2 instances metrics.
##Audience
Database/ETL developers, Data Integrators, Data Engineers, Business Analysts, AWS Developers, DevOps
##Designated Environment Pre-Prod (UAT/QA/DEV)
##Usage
C:\Python35-32>dist\ec2metrics\ec2metrics.exe
## Plots EC2 CPUUtilization metric for given instance id.
##
## Generates matplotlib plots for given instance/statistic/metric.
##
Usage:
set AWS_ACCESS_KEY_ID=<you access key>
set AWS_SECRET_ACCESS_KEY=<you secret key>
set AWS_DEFAULT_REGION=<your region > (for example:us-west-2 )
ec2metrics.exe [<instance>] [<period_min>] [<from_min>] [<to_min>]
[<statistic>] [<metric_name>] [<namespace>]
[<show_plot> or <show_report>]
[<plot_dir>] [<plot_dir>]
[-b] --instance -- EC2 instance name (i-********).
[-p] --period_min -- Aggregation interval (5 min).
[-f] --from_min -- Start from, min (60).
[-t] --to_min -- End at, min (0 - present).
[-s] --statistic -- Statistic type (Average).
Could be one of: Sum,Maximum,Minimum,SampleCount,Average
[-m] --metric_name -- Metric name (CPUUtilization)
Could be one of:
CPUUtilization,NetworkIn,NetworkOut,NetworkPacketsIn,
NetworkPacketsOut,DiskWriteBytes,DiskReadBytes,DiskWriteOps,
DiskReadOps,CPUCreditBalance,CPUCreditUsage,StatusCheckFailed,
StatusCheckFailed_Instance,StatusCheckFailed_System
[-g] --namespace -- CloudWatch namespace,
container for metric (AWS/EC2).
[-r] --show_plot -- Open plotter window (False).
[-n] --show_report -- Open browser with html report (True).
[-d] --plot_dir -- Target plot dir (plots).
[-e] --plot_dir -- Timestamp for to append to plot_dir
(current date).
Index.html is generated in <plot_dir>\<timestamp>
##Environment variables
set AWS_ACCESS_KEY_ID=<you access key>
set AWS_SECRET_ACCESS_KEY=<you secret key>
set AWS_DEFAULT_REGION=<your region > (for example:us-west-2 )
#Examples
ec2metrics.exe --instance i-fe9cea26 -f 1000 -p 10 -s Average,Minimum -m NetworkIn -r
200(0100/0100): i-fe9cea26: NetworkIn: Sum
200(0100/0100): i-fe9cea26: NetworkIn: Maximum
200(0100/0100): i-fe9cea26: NetworkIn: Minimum
200(0100/0100): i-fe9cea26: NetworkIn: SampleCount
200(0100/0100): i-fe9cea26: NetworkIn: Average
Report is at: C:\Python35-32\plots\20160327_220118\index.html
####Result:
![NetworkIn/Average/10min] (https://raw.githubusercontent.com/alexbuz/EC2_Metrics_Plotter/master/plots/EC2_NetworkIn/by_metric/NetworkIn/Average/10/NetworkIn.Average.10.i-fe9cea26.png)
Report is generated with preview for all plots created with this job.
ec2metrics.exe --instance i-fe9cea26 -f 500 -p 1 -s Sum,Maximum,Minimum,SampleCount,Average -m CPUUtilization -r
200(0084/0099): i-fe9cea26: CPUUtilization: Sum
200(0084/0099): i-fe9cea26: CPUUtilization: Maximum
200(0000/0099): i-fe9cea26: CPUUtilization: Minimum
200(0099/0099): i-fe9cea26: CPUUtilization: SampleCount
200(0084/0099): i-fe9cea26: CPUUtilization: Average
Report is at: c:\Python35-32\plots\20160328_113906\index.html
####Result:
![CPUCreditUsage/Average/30min] (https://raw.githubusercontent.com/alexbuz/EC2_Metrics_Plotter/master/plots/CPUUtilization/by_instance/i-fe9cea26/1/CPUUtilization.Average.1.i-fe9cea26.png)
ec2metrics.exe --instance i-fe9cea26 -f 6000 -p 30 -s Sum,Maximum,Minimum,SampleCount,Average -m CPUCreditUsage -r -t 3000 -e CPUCreditUsage
200(0006/0027): i-fe9cea26: CPUCreditUsage: Sum
200(0006/0027): i-fe9cea26: CPUCreditUsage: Maximum
200(0001/0027): i-fe9cea26: CPUCreditUsage: Minimum
200(0027/0027): i-fe9cea26: CPUCreditUsage: SampleCount
200(0006/0027): i-fe9cea26: CPUCreditUsage: Average
Report is at: c:\Python35-32\plots\CPUCreditUsage\index.html
####Result:
![CPUCreditUsage/Average/30min] (https://raw.githubusercontent.com/alexbuz/EC2_Metrics_Plotter/master/plots/CPUCreditUsage/by_metric/CPUCreditUsage/Average/30/CPUCreditUsage.Average.30.i-fe9cea26.png)
c:\Python35-32>dist\ec2metrics\ec2metrics.exe --from_min 3000 --instance 'i-fe9cea26,i-fe9cea26' --metric_name CPUUtilization,NetworkIn,NetworkOut,NetworkPacketsIn,NetworkPacketsOut,DiskWriteBytes,DiskReadBytes,DiskWriteOps,DiskReadOps,CPUCreditBalance,CPUCreditUsage,StatusCheckFailed,StatusCheckFailed_Instance,StatusCheckFailed_System --namespace AWS/EC2 --period_min 1 --plot_dir C:\Python35-32\plots --statistic Average,Minimum,Maximum,Sum --to_min 2000 -r -e All_Metrics
200(0174/0200): i-fe9cea26: CPUUtilization: Average
200(0001/0200): i-fe9cea26: CPUUtilization: Minimum
200(0174/0200): i-fe9cea26: CPUUtilization: Maximum
200(0174/0200): i-fe9cea26: CPUUtilization: Sum
200(0200/0200): i-fe9cea26: NetworkIn: Average
...
200(0000/1000): i-fe9cea26: StatusCheckFailed_Instance: Sum
200(0000/1000): i-fe9cea26: StatusCheckFailed_System: Average
200(0000/1000): i-fe9cea26: StatusCheckFailed_System: Minimum
200(0000/1000): i-fe9cea26: StatusCheckFailed_System: Maximum
200(0000/1000): i-fe9cea26: StatusCheckFailed_System: Sum
Report is at: C:\Python35-32\plots\All_Metrics\index.html
####Result: One of the plots: ![NetworkIn/Average/10min] (https://raw.githubusercontent.com/alexbuz/EC2_Metrics_Plotter/master/plots/CPUCreditBalance/by_instance/i-fe9cea26/30/CPUCreditBalance.Sum.30.i-fe9cea26.png)
Report is generated with preview for all plots created with this job.
##Download
- Master Release --
ec2metrics 0.1.0
#FAQ
Yes, it is the main purpose of this tool.
Yes. Assuming they are doing it on OS Windows.
I use matplotlib.pyplot
Python module to do it.
Script is accessing CloudWatch using boto, retrieving metric data and generating plot images.
Reports include all plot for a given script execution. You can see all plots on one page.
No, you need detailed monitoring to support aggregate statistics.
I used Python
, matplotlib.pyplot
, and boto3
to write it and pyInstaller
to compile it in 64-bit windows executable.
No, I use boto3 to make cals to CloudWatch API.
This version of Metrics Plotter is tested for EC2 metrics only. Metrics included: CPUUtilization,NetworkIn,NetworkOut,NetworkPacketsIn, NetworkPacketsOut,DiskWriteBytes,DiskReadBytes,DiskWriteOps, DiskReadOps,CPUCreditBalance,CPUCreditUsage,StatusCheckFailed, StatusCheckFailed_Instance,StatusCheckFailed_System
No
No, it uses AWS API to read metrics directly from CloudWatch bypassing reports generated on S3.
Please, contact me for sources.
Yes, please, ask me for new features.
- [Oracle_To_S3_Data_Uploader] (https://github.com/alexbuz/Oracle_To_S3_Data_Uploader) - Stream Oracle data to Amazon- S3.
- [CSV_Loader_For_Redshift] (https://github.com/alexbuz/CSV_Loader_For_Redshift/blob/master/README.md) - Append CSV data to Amazon-Redshift from Windows.
- [S3_Sanity_Check] (https://github.com/alexbuz/S3_Sanity_Check/blob/master/README.md) - let's you
ping
Amazon-S3 bucket to see if it's publicly readable. - Oracle-To-Redshift-Data-Loader - plots any CloudWatch EC2 instance metric stats.
- S3_File_Uploader - uploads file from Windows to S3.
Yes, AWS Certified Developer (Associate)
Yes, you can PM me here or email at [email protected]
.
I'll get back to you within hours.
###Links