aws-samples / amazon-ecr-continuous-scan

Example container image re-scan with Amazon ECR

ECR Container Image Re-Scan

This repo shows how to use the ECR image scanning feature for a scheduled re-scan, that is, scanning images on a regular basis. We will walk you through the setup and usage of this demo.

Installation

In order to build and deploy the service, clone this repo and make sure you've got the following available, locally:

• The aws CLI
• The SAM CLI
• Go 1.12 or above

Additionally, having jq installed it recommended.

Preparing the S3 buckets (make sure that you pick different names for the ECR_SCAN_* buckets):

export ECR_SCAN_SVC_BUCKET=ecr-continuous-scan-svc
export ECR_SCAN_CONFIG_BUCKET=ecr-continuous-scan-config

aws s3api create-bucket \
            --bucket $ECR_SCAN_SVC_BUCKET \
            --create-bucket-configuration LocationConstraint=$(aws configure get region) \
            --region $(aws configure get region)

aws s3api create-bucket \
            --bucket $ECR_SCAN_CONFIG_BUCKET \
            --create-bucket-configuration LocationConstraint=$(aws configure get region) \
            --region $(aws configure get region)

Make sure that you have the newest Go SDK installed, supporting the image scanning feature. In addition, you need to go get github.com/gorilla/feeds as the one other dependency outside of the standard library. Then execute:

make deploy

which will build the binaries and deploy the Lambda functions.

You're now ready to use the demo.

Architecture

The overall architecture of the demo is as follows:

There are four Lambda functions and an S3 buckets to hold the scan configurations involved.

The HTTP API is made up of the following three Lambda functions:

• ConfigsFunc handles the management of scan configs, allowing you to store, list, and delete them.
• SummaryFunc provides a summary of the scan findings across all scan configs.
• FindingsFunc provides a detailed Atom feed of the scan findings per scan config.

In addition, there is a StartScanFunc that is triggered by a CloudWatch event, kicking off the image scan.

Scan configurations

To specify which repositories should be re-scanned on a regular basis, one has to provide a scan configuration.

This scan configuration has three required fields, region, registry (your AWS account ID), and the repository itself:

{
    "region": "us-west-2",
    "registry": "123456789012",
    "repository": "amazonlinux",
    "tags": [
        "2018.03"
    ]
}

Note that tags is optional and if not provided, all tags of the repository will be scanned.

API

The following HTTP API is exposed:

Scan configurations:

• GET configs/ … lists all registered scan configurations, returns JSON
• POST configs/ … adds a scan configuration, returns scan ID
• DELETE configs/{scanid} … removes a registered scan configuration by scan ID or 404 if it doesn't exist

Scan findings:

• GET summary/ … provides high-level summary of findings across all registered scan configurations
• GET findings/{scanid} … provides detailed findings on a scan configuration bases, returns an Atom feed

Usage walkthrough

The following walkthrough assumes that the ECR repositories have been set up (using aws ecr create-repository) and the container images have been pushed to the repositories, accordingly.

First, in order to interact with the HTTP API, capture the base URL in an environment variable ECRSCANAPI_URL like so:

export ECRSCANAPI_URL=$(aws cloudformation describe-stacks --stack-name ecr-continuous-scan | jq '.Stacks[].Outputs[] | select(.OutputKey=="ECRScanAPIEndpoint").OutputValue' -r)

Now, add some scan configurations (part of this repo):

curl -s --header "Content-Type: application/json" --request POST --data @scan-config-amazonlinux.json $ECRSCANAPI_URL/configs/
curl -s --header "Content-Type: application/json" --request POST --data @scan-config-centos.json $ECRSCANAPI_URL/configs/
curl -s --header "Content-Type: application/json" --request POST --data @scan-config-ubuntu.json $ECRSCANAPI_URL/configs/

List all registered scan configurations:

$ curl $ECRSCANAPI_URL/configs/
[
  {
    "id": "4471c156-29f5-40fe-883b-3cd26738d5a6",
    "created": "1569927812",
    "region": "us-west-2",
    "registry": "123456789012",
    "repository": "amazonlinux",
    "tags": [
      "2018.03"
    ]
  },
  {
    "id": "612fccea-9545-45d0-8feb-cdc20c4c3061",
    "created": "1569927820",
    "region": "us-west-2",
    "registry": "123456789012",
    "repository": "test/centos",
    "tags": null
  },
  {
    "id": "fc41dda8-f15e-4826-8908-11603b01dac4",
    "created": "1569927828",
    "region": "us-west-2",
    "registry": "123456789012",
    "repository": "test/ubuntu",
    "tags": [
      "16.04",
      "latest"
    ]
  }
]

Get an overview of the scan result findings across the registered scan configurations:

$ curl $ECRSCANAPI_URL/summary
Results for amazonlinux:2018.03 in us-west-2:


Results for test/centos:7 in us-west-2:
 HIGH: 7
 LOW: 7
 MEDIUM: 20


Results for test/ubuntu:16.04 in us-west-2:
 INFORMATIONAL: 19
 LOW: 24
 MEDIUM: 8


Results for test/ubuntu:latest in us-west-2:
 MEDIUM: 7
 INFORMATIONAL: 9
 LOW: 13

Get a detailed feed of findings for test/ubuntu (with scan ID fc41dda8-f15e-4826-8908-11603b01dac4):

$ curl $ECRSCANAPI_URL/findings/fc41dda8-f15e-4826-8908-11603b01dac4
<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <title>ECR repository test/ubuntu in us-west-2</title>
  <id>https://us-west-2.console.aws.amazon.com/ecr/repositories/test/ubuntu/</id>
  <updated></updated>
  <subtitle>Details of the image scan findings across the tags: [16.04] [latest] </subtitle>
  <link href="https://us-west-2.console.aws.amazon.com/ecr/repositories/test/ubuntu/"></link>
  <author>
    <name>ECR</name>
  </author>
  <entry>
    <title>[MEDIUM] in image test/ubuntu:16.04 found CVE-2016-1585</title>
    <updated>2019-10-01T11:27:17Z</updated>
    <id>16.04</id>
    <link href="http://people.ubuntu.com/~ubuntu-security/cve/CVE-2016-1585" rel="alternate"></link>
    <summary type="html">In all versions of AppArmor mount rules are accidentally widened when compiled.</summary>
  </entry>
  ...
</feed>  

The Atom feeds can be consumed in a feed reader, for example:

You can remove scan configs like so:

curl --request DELETE $ECRSCANAPI_URL/configs/4471c156-29f5-40fe-883b-3cd26738d5a6

amazon-ecr-continuous-scan's Issues

Problems with getting dependencies

I'm having trouble running make deploy as instructed in the Installation section.

I've started with a relatively clean (has docker installed already but that's about it) CentOS 7 VM, and then installed AWS CLI, SAM CLI, go 1.17, and jq. I was able to create the buckets using the commands in the instructions. I ran go get github.com/gorilla/feeds.

The instructions for getting the Go SDK were not clear so I ran a few commands that were in the Download instructions from the Go SDK repo:

go get github.com/aws/aws-sdk-go-v2/aws
go get github.com/aws/aws-sdk-go-v2/config

I cloned this repo to ~/go/src so it was within the go project directory

I then tried to run make deploy from the ~/go/src/amazon-ecr-continuous-scan folder and got

GOOS=linux GOARCH=amd64 go build -v -ldflags '-d -s -w' -a -tags netgo -installsuffix netgo -o bin/configs ./configs
go: cannot find main module, but found .git/config in /home/jadair/go/src/amazon-ecr-continuous-scan
	to create a module there, run:
	go mod init
make: *** [bconfigs] Error 1

So I ran go mod init then ran make deploy again. This time I got

$ make deploy
GOOS=linux GOARCH=amd64 go build -v -ldflags '-d -s -w' -a -tags netgo -installsuffix netgo -o bin/configs ./configs
configs/main.go:12:2: no required module provides package github.com/aws/aws-lambda-go/events; to add it:
	go get github.com/aws/aws-lambda-go/events
configs/main.go:13:2: no required module provides package github.com/aws/aws-lambda-go/lambda; to add it:
	go get github.com/aws/aws-lambda-go/lambda
configs/main.go:14:2: no required module provides package github.com/aws/aws-sdk-go-v2/aws/external; to add it:
	go get github.com/aws/aws-sdk-go-v2/aws/external
configs/main.go:15:2: no required module provides package github.com/aws/aws-sdk-go-v2/service/s3; to add it:
	go get github.com/aws/aws-sdk-go-v2/service/s3
configs/main.go:16:2: no required module provides package github.com/aws/aws-sdk-go-v2/service/s3/s3manager; to add it:
	go get github.com/aws/aws-sdk-go-v2/service/s3/s3manager
configs/main.go:17:2: no required module provides package github.com/aws/aws-sdk-go/aws; to add it:
	go get github.com/aws/aws-sdk-go/aws
configs/main.go:19:2: no required module provides package github.com/satori/go.uuid; to add it:
	go get github.com/satori/go.uuid`

So then I try to pull down those packages. Most of them were successful, except for these two

$ go get github.com/aws/aws-sdk-go-v2/aws/external
go get: module github.com/aws/aws-sdk-go-v2@upgrade found (v1.8.0), but does not contain package github.com/aws/aws-sdk-go-v2/aws/external
$ go get github.com/aws/aws-sdk-go-v2/service/s3/s3manager
go get: module github.com/aws/aws-sdk-go-v2/service/s3@upgrade found (v1.12.0), but does not contain package github.com/aws/aws-sdk-go-v2/service/s3/s3manager

It looks like aws/external got renamed to config at some point.

I can't seem to find anything on service/s3/s3mananger. Maybe it's feature/s3/manager now?

I'm new to go, so any help you can provide is appreciated.

Missing go sdk moduules

Hi,

Am trying to build the code. I run a make deploy but the code cannot load two modules, s3manager and external.

I do not see these directories in github. How can I proceed?

[johnson@haysimp amazon-ecr-continuous-scan]$ make deploy
GOOS=linux GOARCH=amd64 go build -v -ldflags '-d -s -w' -a -tags netgo -installsuffix netgo -o bin/configs ./configs
configs/main.go:14:2: no required module provides package github.com/aws/aws-sdk-go-v2/aws/external; to add it:
go get github.com/aws/aws-sdk-go-v2/aws/external
configs/main.go:16:2: no required module provides package github.com/aws/aws-sdk-go-v2/service/s3/s3manager; to add it:
go get github.com/aws/aws-sdk-go-v2/service/s3/s3manager
make: *** [bconfigs] Error 1
[johnson@haysimp amazon-ecr-continuous-scan]$ go get github.com/aws/aws-sdk-go-v2/aws/external
go get: module github.com/aws/aws-sdk-go-v2@upgrade found (v1.3.0), but does not contain package github.com/aws/aws-sdk-go-v2/aws/external
[johnson@haysimp amazon-ecr-continuous-scan]$ go get github.com/aws/aws-sdk-go-v2/service/s3/s3manager
go get: module github.com/aws/aws-sdk-go-v2/service/s3@upgrade found (v1.3.0), but does not contain package github.com/aws/aws-sdk-go-v2/service/s3/s3manager
[johnson@haysimp amazon-ecr-continuous-scan]$

Thank you for your help.

Johnson Hays