googlecloudplatform / pci-gke-blueprint

PCI on GKE Blueprint: PCI Deployable Architecture on Google Cloud and GKE

PCI on GKE Blueprint

This is the companion repository to the PCI on GKE Security Blueprint for the Google Cloud Platform. It contains a set of Terraform configurations and scripts to help demonstrate how to bootstrap a PCI environment in GCP. When appropriate, we also showcase GCP services, tools, or projects we think might be useful to start your own GCP PCI environment or as samples for any other purposes.

Here are the projects/services we make use of in this Blueprint:

• Terraform
• Helm
• Google Kubernetes Engine (GKE)
• Istio
• Cloud Armor
• Google-managed SSL Certificates
• GoogleCloudPlatform/microservices-demo

Documentation

• Quickstart
• Prerequisites
  • Installation Dependencies
  • GCP IAM Requirements
• Workstation Configuration
• Building the Infrastructure
• Deploying the Application
• Diagrams
• Kubernetes RBAC via Google Groups membership demonstration
• Development
• Continuous Integration with Cloud Build
• Known Issues and Limitations
• Helpful Links

Quickstart

We recommend you read through the documentation in Building the Infrastructure and Deploying the Application but if you just want to get started:

1. Follow the steps in Prerequisites
2. Set-up the workstation.env file Workstation Configuration
3. Run ./_helpers/build-infra.sh
4. Run ./_helpers/deploy-app.sh

Prerequisites

Before starting, we need to make sure that our local environment is configured correctly. We need to make sure we have the correct tools and a GCP account with the correct permissions.

Installation Dependencies

• Terraform
• gcloud
• kubectl
• kubectx
• helm, version 3

GCP IAM Requirements

In order to execute this module you will need access to a Google Cloud Organization, with Organization Admin and Folder Admin permissions.

Authenticate to gcloud

• Once the gcloud SDK is installed, run gcloud auth login to authenticate with your Google Account.

Workstation Configuration

This project comes with a workstation.env.example file that is intended to be copied and customized for your environment.

cp workstation.env.example workstation.env

You can find the values for YOUR_ORG_ID and YOUR_BILLING_ACCOUNT_ID using the following commands:

gcloud organizations list
gcloud beta billing accounts list

To create a folder follow these instructions.

Most variables can be left as-is, this is a list of the ones that are required to be set, see the comments in-line for details:

• TF_VAR_org_id
• TF_VAR_gsuite_id
• TF_VAR_billing_account
• TF_VAR_folder_id
• TF_ADMIN_BUCKET
• TF_VAR_frontend_zone_dns_name
• GOOGLE_GROUPS_DOMAIN
• SRC_PATH
• REPOSITORY_NAME

You'll need to source your workstaion.env file before executing any of the steps in this Blueprint:

source workstation.env

• At this point, your workstation is ready. Continue from here by either running ./_helpers/build-infra.sh, or following the stepwise instructions for that script in Building the Infrastructure.

Known Issues and Limitations

• If your GCP Organization is shared between other users or teams, consult your Organization Admins before building the Blueprint.
• This Blueprint does not implement a multi-environment setup. There is no "pre-prod", "staging", or "production" differentiation. However, there is no reason that this Blueprint couldn't be expanded to accommodate such a setup if you so choose.
• This Blueprint is meant to showcase various GCP features and act as a starting point to build a security-focused environment focused on PCI compliance. This Blueprint has been reviewed by Coalfire but deploying an application into this environment does not qualify as being PCI-DSS compliant.
• As currently designed, http:// requests are redirected to https:// via HTTP header inspection by the frontend microservice. More details in HTTP to HTTPS redirection

Helpful Links

• Significant portions of this project are based on "Building a multi-cluster service mesh on GKE using replicated control-plane architecture"

pci-gke-blueprint's Issues

Refactor helm charts

• refactor dir structure #39
• Investigate removing helm charts' default template helpers
• Rename/remove references to "old" in configmaps and deployments in the fluentd helm charts

Add steps for creating TLS certificates for Istio

Currently, deploy-app.sh includes a section that configures Istio to use Istio supplied certificates:

pci-gke-blueprint/_helpers/deploy-app.sh

Lines 44 to 48 in a9ad9cf

	kubectl create secret generic cacerts -n istio-system \
	--from-file=${SRC_PATH}/istio-${ISTIO_VERSION}/samples/certs/ca-cert.pem \
	--from-file=${SRC_PATH}/istio-${ISTIO_VERSION}/samples/certs/ca-key.pem \
	--from-file=${SRC_PATH}/istio-${ISTIO_VERSION}/samples/certs/root-cert.pem \
	--from-file=${SRC_PATH}/istio-${ISTIO_VERSION}/samples/certs/cert-chain.pem

It would be helpful to add steps that facilitate creating and using self-generated certificates.

Enable Packet Mirroring for IDS / IPS requirement

As described in https://cloud.google.com/vpc/docs/packet-mirroring

Move the frontend and checkout services to the in-scope cluster

Refactor Terraform variables

• Where applicable, like where there are long service account names, use locals.

Upgrade to Stackdriver Kubernetes Engine Monitoring aka v2(beta) Stackdriver for GKE

The project is currently using Legacy Stackdriver. (See Which Stackdriver support does my cluster use? )

More precisely, GKE data is visible in Stackdriver's Legacy Kubernetes dashboard as a result of Stackdriver Monitoring being set to Enabled ) ( and not v2(beta) ). Stackdriver Logging is disabled due to the fluentd customizations in place (centralized logging, and DLP API filtering).

In order to upgrade to "Stackdriver Kubernetes Engine Monitoring," the recommended method is to enable both Stackdriver Monitoring and Stackdriver Logging, in order to choose "Enabled v2(beta)". This is undesirable because that will then cause the stock fluentd to be installed, which would then cause logs to go direct to Stackdriver Logs and bypass the DLP API filter.

The most viable option is to package the v2(beta) configurations, and customize them as needed, while leaving the stock GKE Monitoring and Logging settings disabled.

Investigate VPC Service Controls

Is it possible to connect to DLP API over a private endpoint?

Update documentation

Ensure commented descriptions on all IAM binding resources

From code review tasks #50

Add a reference to the cluster resource in the node pool resource

Add a reference to the cluster resource in the node pool resource so the order of node pool and cluster is determined correctly.

For example, here, in the in-scope cluster, as well as the out-of-scope cluster.

Create helm charts to allow Kubernetes config templating

kustomize appears to be built for the use case of customizing Kubernetes configurations, in a style that will likely allow for the simplification of the current shell generate-config.sh which uses sed on "template-like" files to then create Kubernetes configs.

• See: https://youtu.be/ahMIBxufNR0?t=300
• https://github.com/kubernetes-sigs/kustomize/blob/master/docs/eschewedFeatures.md#build-time-side-effects-from-cli-args-or-env-variables Although listed as an eschewed feature, the style to be adopted is to have bash set with the necessary variables, run kustoimize edit ..., and apply.

Investigate adding Stackdriver IRM to the demo

Terraform can't handle GKE issue_client_certificate w/ K8S version 1.12

• We're affected by this terraform / GCP bug tracked here: hashicorp/terraform-provider-google#3369

forseti service account permissions

When running : ./_helpers/forseti_admin_permissions.sh I get:

ERROR: Policy modification failed. For a binding with condition, run "gcloud alpha iam policies lint-condition" to identify issues in condition. ERROR: (gcloud.organizations.add-iam-policy-binding) INVALID_ARGUMENT: Role roles/serviceusage.serviceAccountAdmin is not supported for this resource.

The link in the shell script is no longer valid. <https://forsetisecurity.org/docs/v2.0/concepts/service-accounts.html#permissions

Which set of permissions from here https://forsetisecurity.org/docs/latest/concepts/service-accounts.html should we be setting?

Code Review with someone from the CFT team

Maybe @morgante or someone from his team can do a code review.

Add the image cleanup lines from the upstream fluentd-gcp Dockerfile

The addition of the Dockerfile that builds a custom fluentd-gcp container is based on the Dockerfile in google-fluentd. The cleanup section (beginning at line 20) of that file should be tested and added back in.

Add docs describing connecting Forseti to Security Command Center

README: Direct user during project setup to create a shell script for env vars

Terraform 0.12 support

When running ./_helpers/build-infra.sh I get:

Downloading modules/core_project_factory for project_network.project-factory...
Downloading terraform-google-modules/network/google 0.6.0 for vpc_pci...

• vpc_pci in .terraform/modules/vpc_pci/terraform-google-modules-terraform-google-network-1d3242b

Error: Module not found

The module address "modules/gsuite_group" could not be resolved.

If you intended this as a path relative to the current module, use
"./modules/gsuite_group" instead. The "./" prefix indicates that the address
is a relative filesystem path.

I am using terraform 0.12

I looks like it is related to terraform-google-modules/terraform-google-project-factory#214

Improve, iterate on diagrams in documentation

Additional documentation

• Call out that environment (prod v non-prod) are not considered in this demo
• Consider creating a “Known Issues” / “Gotcha” section in documentation. ie. That Forseti needs to be created before some variables can be set and re-applied

GKE version is shared.tf is unsupported

When trying to deploy the components infrastructure you currently run into the error "Master version "1.14.8-gke.12" is unsupported. This value needs to be upgraded to at least 1.14.10-gke.27.

Demonstrate a SCC/Forseti use case

List which PCI requirements we are meeting with this project

Add a section that defines how we are meeting different PCI requirements with this project.

Reduce the oauth scopes provisioned to the in scope cluster

Migrate to the CFT GKE module

Enable the Debugger API

Stackdriver logs were getting populated with errors relating to the fact that the Debug API was not enabled, but data was being sent.

add deployment constraints

it would be nice to highlight config validator with the PCI terraform

Final PCI team review

I would like myself, Ian and Zeal to review everything right before we release and give a LGTM comment here.

Configure fluentd to send all GKE logs (both clusters) to the mgmt project's Stackdriver

• update terraform to include additional required permissions in the destination project

Update configurations to enable demonstrating the Debug API

The Cloud Debug API is enable onthe projects, and some of the services are sending data, but additional configuration is required. See GoogleCloudPlatform/microservices-demo#87 for more details.

Decide on the requirements for the certificates used between services

Currently, grpc_ssl_verify is set to off on checkoutservice's Nginx's grpc proxy. It should be possible to set this to on.

If we choose not to do this, let's make a note of it in the readme.

Migrate project to Terraform v0.12.0

Upgrade guide here: https://www.terraform.io/upgrade-guides/0-12.html

Add feature list/tour to docs

Review for consistency with Google’s coding style guides

• Notably, the bash style guide here: https://google.github.io/styleguide/shell.xml

HTTPS Only

Code Review follow up tasks

• Call out that environment (prod v non-prod) are not considered in this demo #61
• Consider creating a “Known Issues” / “Gotcha” section in documentation. ie. That forsetti needs to be created before some variables can be set and re-applied #61
• Review for consistency with Google’s internal terraform standards guide #62
• in shared.tf, separate out variables and locals into separate files
• where applicable, like where there are long service account names, use locals. Example #63
• Ensure commented descriptions on all IAM bindings #55
• Migrate to use CFT log-export module #54
• Remove unused cluster outputs #53

Create a single master build script

Let's create one or two scripts that will build all the infrastructure and then deploy the applications.

First terraform apply for project/network always fails creating NAT

There's an error stating that the NAT isn't created on first run because subnetwork isn't in "ready" state.

Automation: Set up continuous deployment with GCP tools

Upgrade to latest Istio version with latest installation steps.

This project is currently on v1.3.3 of Istio, and 1.5 is current.

pci-gke-blueprint/_helpers/deploy-app.sh

Line 30 in a9ad9cf

export ISTIO_VERSION=1.3.3

Add a Cleanup section to documentation

Instructions on how to tear down the demo infrastructure

Review firewall rules

Since the changes in #24 included provisioning the Kubernetes scheduler with the custom role "Firewall Admin", and the re-arrangement of the frontend to the in-scope cluster, some of the firewall rules that are in place are likely superfluous or incorrect.
Additionally, we need to add a rule to block port 80 to the frontend IP address entirely. See also kubernetes/ingress-gce#290:

apiVersion: extensions/v1beta1
metadata:
  annotations:
    kubernetes.io/ingress.allow-http: "false"

Setting the above doesn't close port 80 on the http/s LB, it disables that Ingress rule from serving content on port 80.

Configure Tiller to use TLS Auth

The instructions at https://github.com/GoogleCloudPlatform/terraform-pci-starter/blob/master/README.md#helm-installation-and-setup will most likely want to be modified to follow the guidance here: https://helm.sh/docs/using_helm/#using-ssl-between-helm-and-tiller to better protect the Tiller gRPC endpoint from malicious pods being able to access Tiller directly and install a privileged chart.

/cc @binamov

Migrate to CFT Log-export module

From code review follow up tasks #50

Logging: Allow both in- and out-of-scope logs to be simultaneously viewable

Currently, in-scope logs (application, audit, etc.) are sent to the in-scope project's Stackdriver logs, and the out-of-scope's are going to the out-of-scope project's. This means that logs are not viewable/ searchable across clusters. We'd probably want to change things so that they're both going to a single Stackdriver logging target.

Mask CC and CVV on frontend payment form

This will require a PR/Issue with the Microservices Demo project: https://github.com/GoogleCloudPlatform/microservices-demo

Add code header boilerplate/licensing

Remove unused cluster outputs

From code review tasks #50

Refactor terraform files to prepare/ease transition to a multiple repository layout

The terraform code should be refactored based on IAM, Network, IP, Projects, and Components so that users can be able to separate those categories into separate repositories if desired. Doing so would help to prevent a wide blast radius of an error/misconfiguration in a single terraform run. This is true especially as a preparation in moving to a CI/CD pipeline.