Confirm application connectivity requirements (current and to-be) so that we can configure the AWS networking.

Connectivity Information: from_port, to_port, protocol, TLS/SSL, Certificate Info,

AWS Config: Network Interfaces, Route Tables, Internet Gateways, DNS, Elastic IP Addresses, VPC Endpoints, NAT, VPC Peering

Acceptance Criteria:

• Connectivity confirmed

Required before creating Terraform scripts.

Adopt from a combination of ACP and IPT?

An example template to build an AMI that is published and shared with multiple amazon accounts
we can demonstrate this building our mock application and having it published to both amazon accounts

Confirm application connectivity requirements (current and to-be) so that we can configure the AWS networking.

Connectivity Information: from_port, to_port, protocol, TLS/SSL, Certificate Info,

AWS Config: Network Interfaces, Route Tables, Internet Gateways, DNS, Elastic IP Addresses, VPC Endpoints, NAT, VPC Peering

Acceptance Criteria:

• Connectivity confirmed

ACP can offer ELK:

• 2 week retention
• currently only has a kubernetes schema

Confirm application connectivity requirements (current and to-be) so that we can configure the AWS networking.

Connectivity Information: from_port, to_port, protocol, TLS/SSL, Certificate Info,

AWS Config: Network Interfaces, Route Tables, Internet Gateways, DNS, Elastic IP Addresses, VPC Endpoints, NAT, VPC Peering

Acceptance Criteria:

• Connectivity confirmed

ACP can offer authd vpn profiles:

• VPN that can route to either of our prod/not-prod peering VPCs

Goal: define terraform testing

Repo: https://github.com/UKHomeOffice/dq-aws-transition-testing

Acceptance Criteria

• Code in open github repo
• Explore testing frameworks for Terraform
• Explore Terraform parsing options
• Trial kitchen-terraform
• Create Terraform Runner

Trial terraform_validate
Write additional terraform cases
Explore CloudWatch

Confirm application connectivity requirements (current and to-be) so that we can configure the AWS networking.

Connectivity Information: from_port, to_port, protocol, TLS/SSL, Certificate Info,

AWS Config: Network Interfaces, Route Tables, Internet Gateways, DNS, Elastic IP Addresses, VPC Endpoints, NAT, VPC Peering

Acceptance Criteria:

• Connectivity confirmed

Confirm application connectivity requirements (current and to-be) so that we can configure the AWS networking.

Connectivity Information: from_port, to_port, protocol, TLS/SSL, Certificate Info,

AWS Config: Network Interfaces, Route Tables, Internet Gateways, DNS, Elastic IP Addresses, VPC Endpoints, NAT, VPC Peering

Acceptance Criteria:

• Connectivity confirmed

ACP can offer peering to our Prod/not-prod peering VPCs with two choices/caveats:

1. we can have a pool of nodes that can VPC peer to our peering VPCs
2. we can trust the authentication mechanisms and expose our application services other ACP tenants in that cluster
3. we can use a kong gateway or similar to authenticate traffic going to our peering VPCs has come from our ACP tenancy

We require a mock application that we can use for connectivity testing

Run like:

$ CHECK_ONE=10.0.0.2:80 \ 
CHECK_ANOTHER=google.com:80 \
LISTEN_FOO=0.0.0.0:80 \ 
LISTEN_BAR=10.0.0.1:8080 \
./app-name

N.B. specifying not 0.0.0.0 for the listening IP means you'll need to know the IP of the instance. You can not specify a name to listen on.

You can then curl the application (and/or chain many instances of the application to demonstrate a full stack) on any of the listening ports, if any of the CHECKS fail (at runtime) it will return a 500 error, otherwise 200.
it will also return a text HTTP body of:

CHECK_ONE=10.0.0.2:80 [OK]
CHECK_ANOTHER=google.com:80 [FAIL]

The AMI image should set the required environment variables based on the ec2 instance tags assigned to it at runtime

Acceptance Criteria

• Application that runs with the minimum dependencies possible
• Code in open github repo
• Docker image published on quay
• Windows, Mac, Linux binaries published to github releases
• Example kubernetes repository published with deployment, service + ingress rules with several interconnected instances moved to #49
• Packer + Ansible linux build AMI
• Packer + Ansible windows build AMI moved to #49
• AMI published in DQ Amazon account
• Example terraform

Populate list of VPC for peering.
Variables:

VPCID
VPCOwnerID
PeerVPCID
PeeringRouteID
RouteTableID
VPCPeeringConnectionID

Acceptance criteria

• Peering is Active/Active within same AWS account

• Peering is Active/Active across AWS accounts

• Code cleaned up and ready for review

We should get access to the IPT Terraform scripts and review these to see how much opportunity there is for re-use.

Confirm application connectivity requirements (current and to-be) so that we can configure the AWS networking.

Connectivity Information: from_port, to_port, protocol, TLS/SSL, Certificate Info,

AWS Config: Network Interfaces, Route Tables, Internet Gateways, DNS, Elastic IP Addresses, VPC Endpoints, NAT, VPC Peering

Acceptance Criteria:

• Connectivity confirmed

Confirm application connectivity requirements (current and to-be) so that we can configure the AWS networking.

Connectivity Information: from_port, to_port, protocol, TLS/SSL, Certificate Info,

AWS Config: Network Interfaces, Route Tables, Internet Gateways, DNS, Elastic IP Addresses, VPC Endpoints, NAT, VPC Peering

Acceptance Criteria:

• Connectivity confirmed

Confirm application connectivity requirements (current and to-be) so that we can configure the AWS networking.

Connectivity Information: from_port, to_port, protocol, TLS/SSL, Certificate Info,

AWS Config: Network Interfaces, Route Tables, Internet Gateways, DNS, Elastic IP Addresses, VPC Endpoints, NAT, VPC Peering

Acceptance Criteria:

• Connectivity confirmed

ACP can provide ingress + TLS termination and we could route all traffic via them

• easy WAF configuration
• easy TLS configuration inc certificate renwals
• easy whitelisting setups for known POISE traffic (and the different IPs that might change to)
• No external facing surface area to any of our estate

ACP can also if we do need external presence provide delegation to our Route53 DNS

Acceptance criteria

• kubernetes configuration deployed to ACP that routes traffic to the peering VPC

• document drafted
• peer reviewed
• final document
• client sign off
• invitation to tender issued

Complete the Migration plan deliverable

This will detail the order by which we will transition the existing digital services to AWS, along with the supporting dependencies.

Confirm application connectivity requirements (current and to-be) so that we can configure the AWS networking.

Connectivity Information: from_port, to_port, protocol, TLS/SSL, Certificate Info,

AWS Config: Network Interfaces, Route Tables, Internet Gateways, DNS, Elastic IP Addresses, VPC Endpoints, NAT, VPC Peering

Acceptance Criteria:

• Connectivity confirmed

Confirm application connectivity requirements (current and to-be) so that we can configure the AWS networking.

Connectivity Information: from_port, to_port, protocol, TLS/SSL, Certificate Info,

AWS Config: Network Interfaces, Route Tables, Internet Gateways, DNS, Elastic IP Addresses, VPC Endpoints, NAT, VPC Peering

Acceptance Criteria:

• Connectivity confirmed

Continuation from #38

• Example kubernetes repository published with deployment, service + ingress rules with several interconnected instances
• Packer + Ansible windows build AMI

Complete the first iteration of the AWS design document

Contents:

• Design principles
• High level design
• Security design/assurance approach
• CI/CD process

Design principles

Explain our design principles

High level design

Covering:

• Environments (Dev, Pre-prod, Prod)
• VPCs, VPC peering, subnets, firewall tables, NAT gateways and proxies
• Interfaces to third parties
• Data pipeline
• Monitoring and logging
• Placement of apps and components (including mock apps to support early testing)

Security design/assurance approach

An overview of how we will achieve the required assurance to gain Authority To Operate.
This will follow an iterative approach.

CI/CD process

Overview of the CI/CD process and what requirements/expectations this places on the development teams to get their code through the process and into Production.

Setup routing between multiple VPCs within the same account and test connectivity using EC2 instances.

Acceptance criteria

Routing works between all dedicated VPCs defined above

• Routing up/up between 3 VPC

• Code cleanup for review

Confirm application connectivity requirements (current and to-be) so that we can configure the AWS networking.

Connectivity Information: from_port, to_port, protocol, TLS/SSL, Certificate Info,

AWS Config: Network Interfaces, Route Tables, Internet Gateways, DNS, Elastic IP Addresses, VPC Endpoints, NAT, VPC Peering

Acceptance Criteria:

• Connectivity confirmed

Confirm application connectivity requirements (current and to-be) so that we can configure the AWS networking.

Connectivity Information: from_port, to_port, protocol, TLS/SSL, Certificate Info,

AWS Config: Network Interfaces, Route Tables, Internet Gateways, DNS, Elastic IP Addresses, VPC Endpoints, NAT, VPC Peering

Acceptance Criteria:

• Connectivity confirmed

ACP can offer Sysdig:

• container orientated
• no windows support (though can monitor cloudwatch that monitors windows)

Confirm application connectivity requirements (current and to-be) so that we can configure the AWS networking.

Connectivity Information: from_port, to_port, protocol, TLS/SSL, Certificate Info,

AWS Config: Network Interfaces, Route Tables, Internet Gateways, DNS, Elastic IP Addresses, VPC Endpoints, NAT, VPC Peering

Acceptance Criteria:

• Connectivity confirmed

• dq-tf-infra to use terragrunt to deploy with s3 state

Investigate proxy options:

HAProxy
NGINX

and

Routing Table
NAT Gateway
NAT Instance
ELB
VPN instances
AWS Marketplace VPN device

Acceptance criteria

• Traffic traverse between dedicated VPCs successfully

• Analyse Proxy options

• Analyse NAT GW and NAT instance

• Analyse ELB

• Analyse VPN instance solution

• Analyse AWS Marketplace Virtual Router solution

• Functional example of VPC peering with Proxy server and mock app working.

• s3 configurable haproxy config

• packer+ansible haproxy image build

Composable ec2 architecture of:

• 1 repository per application image/AMI (packer+ansible) [open source]
  • e.g. dq-packer-greenplum
• 1 repository per application module (terraform) [open source]
  • entirely self contained module of the application which should allow you to stand up the application in total isolation with parameters for ips, subnets, naming schemes, etc - and any other external dependencies.
  • should have a paramterized switch for IN_TEST that sets machine types to nano and all images to nano
  • e.g. dq-tf-greenplum
• 1 repository with common base terraform modules [open source]
  • these should be simply abstracting away from the basic terraform to 'DQify' them
  • e.g. dq-tf-common
• 1 repository for the overarching infrastructure (terraform) [closed source]
  • e.g. dq-tf-infra
  • describes what an instance of a 'DQ' is, and the permutations of that for different environments
  • references the environment n times - deployed to each account
  • defines all the parameters that are applied for downstream dependents (that are not maintained in drone secrets)

Composable acp/kubernetes architecture of:

• 1 repository per application (docker) image [open/closed source]
  • built and pushed to quay/artifiactory by CI
  • e.g. dq-docker-crt
• 1 repository per application module (kubernetes) [open source]
  • e.g. dq-kube-cty

(other repos for tech spikes created ad-hoc as required)

Acceptance Criteria:

• Design (as above)
• Create immediately required repos
• Document strategy
  • include some sort of evolving glossary of terms (e.g. application modules, images, environments, accounts)
• Setup CI skeleton pipeline jobs
• Setup repository permissions
  • protected master branch
  • requires admin peer review to merge to master
  • manually add external collaborators
  • add DQ transition team

Draft the Security Management Plan

Customise the standard Aker Systems Security Management Plan for the project.

Confirm changes to existing S4, Maytech and ACL ingest scripts/processes

Acceptance Criteria:

• Documented changes