GithubHelp home page GithubHelp logo

canada-ca / cloud-guardrails-gcp Goto Github PK

View Code? Open in Web Editor NEW
22.0 22.0 14.0 392 KB

Recommended configuration guidance for Google Cloud Platform / Conseils de configuration recommandés pour Platforme infonuagique de Google

License: Other

Dockerfile 4.01% Shell 19.81% Open Policy Agent 76.17%

cloud-guardrails-gcp's People

Contributors

cartyc avatar fmichaelobrien avatar jacyang2010 avatar ptd-tbs avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

cartyc ssc-spc-ccoe-cei claudianavarrofragoso fmichaelobrien obriensystems vinovee aravind-dussani99 danishkamili yw-liftandshift stanimprover sunilatntt faisalnizam cloudlandingzone

cloud-guardrails-gcp's Issues

validation run broken on missing assets/*.json files from the asset inventory export and hardcoded "my-unique-bucket-name"

following
https://github.com/canada-ca/cloud-guardrails-gcp/tree/main/guardrails-validation

Workaround - turn off regional restriction on the project - or delete it on the parent and the rerun a terraform apply to get it back after

admin_@cloudshell:~/cloudshell_open (gr-bootstrap-ggz)$ gcloud services enable cloudasset.googleapis.com
Operation "operations/acat.p2-502392433631-09e81fe7-570c-44a3-8345-9852d82fd884" finished successfully.
admin_@cloudshell:~/cloudshell_open (gr-bootstrap-ggz)$ export MY_BUCKET_NAME=validation-ggz
admin_@cloudshell:~/cloudshell_open (gr-bootstrap-ggz)$ gsutil mb gs://$MY_BUCKET_NAME
Creating gs://validation-ggz/...
admin_@cloudshell:~/cloudshell_open (gr-bootstrap-ggz)$ gcloud asset export --output-path=gs://$MY_BUCKET_NAME/resource_inventory.json --content-type=resource --project=gr-bootstrap-ggz
Export in progress for root asset [projects/gr-bootstrap-ggz].
Use [gcloud asset operations describe projects/502392433631/operations/ExportAssets/RESOURCE/c6cfd41c3c7720348b468221cf6c688e] to check the status of the operation.
admin_@cloudshell:~/cloudshell_open (gr-bootstrap-ggz)

admin_@cloudshell:~/cloudshell_open (gr-bootstrap-ggz)$ git clone https://github.com/canada-ca/cloud-guardrails-gcp.git


admin_@cloudshell:~/cloudshell_open (gr-bootstrap-ggz)$ ls
accelerators_accelerateurs-gcp  cloud-guardrails-gcp
admin_@cloudshell:~/cloudshell_open (gr-bootstrap-ggz)$ gsutil cp gs://$MY_BUCKET_NAME/resource_inventory.json ./assets
Copying gs://validation-ggz/resource_inventory.json...
/ [1 files][ 16.8 KiB/ 16.8 KiB]
Operation completed over 1 objects/16.8 KiB.
admin_@cloudshell:~/cloudshell_open (gr-bootstrap-ggz)


dmin_@cloudshell:~/cloudshell_open (gr-bootstrap-ggz)$ cd
accelerators_accelerateurs-gcp/ cloud-guardrails-gcp/
admin_@cloudshell:~/cloudshell_open (gr-bootstrap-ggz)$ cd cloud-guardrails-gcp/
.git/                  guardrails/            guardrails-validation/
admin_@cloudshell:~/cloudshell_open (gr-bootstrap-ggz)$ cd cloud-guardrails-gcp/guardrails-validation/
admin_@cloudshell:~/cloudshell_open/cloud-guardrails-gcp/guardrails-validation (gr-bootstrap-ggz)$ ls
assets  cloudbuild.yaml  Dockerfile  install.sh  policies  README.md  run-all.sh  run.sh  tests.sh
admin_@cloudshell:~/cloudshell_open/cloud-guardrails-gcp/guardrails-validation (gr-bootstrap-ggz)$


min_@cloudshell:~/cloudshell_open/cloud-guardrails-gcp/guardrails-validation (gr-bootstrap-ggz)$ ./install.sh
--2022-11-20 15:39:04--  https://github.com/open-policy-agent/conftest/releases/download/v0.32.1/conftest_0.32.1_Linux_x86_64.tar.gz
Resolving github.com (github.com)... 140.82.113.3
Connecting to github.com (github.com)|140.82.113.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/178249461/a9c964a8-a471-41f8-aed7-86bca64ad3f8?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221120%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221120T153904Z&X-Amz-Expires=300&X-Amz-Signature=7b360ba6a1ab670e8c8957132cfcfda8d28cb797571ec78759636b3b6e402da8&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=178249461&response-content-disposition=attachment%3B%20filename%3Dconftest_0.32.1_Linux_x86_64.tar.gz&response-content-type=application%2Foctet-stream [following]
--2022-11-20 15:39:04--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/178249461/a9c964a8-a471-41f8-aed7-86bca64ad3f8?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221120%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221120T153904Z&X-Amz-Expires=300&X-Amz-Signature=7b360ba6a1ab670e8c8957132cfcfda8d28cb797571ec78759636b3b6e402da8&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=178249461&response-content-disposition=attachment%3B%20filename%3Dconftest_0.32.1_Linux_x86_64.tar.gz&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.108.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.110.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 12264521 (12M) [application/octet-stream]
Saving to: ‘conftest_0.32.1_Linux_x86_64.tar.gz’

conftest_0.32.1_Linux_x86_64.tar.gz      100%[=================================================================================>]  11.70M  44.0MB/s    in 0.3s

2022-11-20 15:39:04 (44.0 MB/s) - ‘conftest_0.32.1_Linux_x86_64.tar.gz’ saved [12264521/12264521]

LICENSE
README.md
conftest
admin_@cloudshell:~/cloudshell_open/cloud-guardrails-gcp/guardrails-validation (gr-bootstrap-ggz)$


admin_@cloudshell:~/cloudshell_open/cloud-guardrails-gcp/guardrails-validation (gr-bootstrap-ggz)$ conftest --version
Conftest: 0.32.1
OPA: 0.40.0

run n/a due to missing json in the assets dir
dmin_@cloudshell:~/cloudshell_open/cloud-guardrails-gcp/guardrails-validation (gr-bootstrap-ggz)$ ./run.sh
Checking ./assets/*.json
cat: './assets/*.json': No such file or directory
admin_@cloudshell:~/cloudshell_open/cloud-guardrails-gcp/guardrails-validation (gr-bootstrap-ggz)$ ls ../
guardrails  guardrails-validation  LICENSE  README.md
admin_@cloudshell:~/cloudshell_open/cloud-guardrails-gcp/guardrails-validation (gr-bootstrap-ggz)$ ls
assets  cloudbuild.yaml  Dockerfile  install.sh  policies  README.md  report.txt  run-all.sh  run.sh  tests.sh
admin_@cloudshell:~/cloudshell_open/cloud-guardrails-gcp/guardrails-validation (gr-bootstrap-ggz)$ ls assets/
admin_@cloudshell:~/cloudshell_open/cloud-guardrails-gcp/guardrails-validation (gr-bootstrap-ggz)$


run-all also requires a rename of"my-unique-bucket-name" as well as addition storage admin role

admin_@cloudshell:~/cloudshell_open/cloud-guardrails-gcp/guardrails-validation (gr-bootstrap-ggz)$ ./run-all.sh
Your active configuration is: [cloudshell-22055]
Creating gs://my-unique-bucket-name/...
ServiceException: 409 A Cloud Storage bucket named 'my-unique-bucket-name' already exists. Try another name. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization.
ERROR: (gcloud.asset.export) code: 403
message: The billing account for the owning project is disabled in state closed
status: PERMISSION_DENIED
AccessDeniedException: 403 [email protected] does not have storage.objects.list access to the Google Cloud Storage bucket. Permission 'storage.objects.list' denied on resource (or it may not exist).
cat: './assets/*.json': No such file or directory

Validation verification

Doing a run on a recent guardrails install
following
https://github.com/canada-ca/cloud-guardrails-gcp/blob/main/guardrails-validation/README.md
from canada-ca/accelerators_accelerateurs-gcp#40
https://github.com/canada-ca/accelerators_accelerateurs-gcp/tree/main/deployment-templates/Terraform/guardrails

previous on [email protected]
cd accelerators_accelerateurs-gcp/
   25  gcloud services enable cloudasset.googleapis.com
   26  export MY_BUCKET_NAME=sccninfo-guardrails-validator
   27  gsutil mb gs://$MY_BUCKET_NAME
   28  gcloud asset export --output-path=gs://$MY_BUCKET_NAME/resource_inventory.json --content-type=resource --project=guardrails-eaba
   29  gcloud config set project guardrails-eaba
   30  gcloud services enable cloudasset.googleapis.com
   31  export MY_BUCKET_NAME=sccninfo-guardrails-validator-gr
   32  gsutil mb gs://$MY_BUCKET_NAME
   33  gcloud config set project sscncinfo-seed-project
   34  gcloud services enable cloudasset.googleapis.com
   35  gcloud config set project sscncinfo-seed-project-seed
   36  gcloud config set project sscncinfo-seed-project
   37  gcloud services enable cloudasset.googleapis.com
   38  export MY_BUCKET_NAME=sccninfo-guardrails-validator-seed
   39  gsutil mb gs://$MY_BUCKET_NAME
   40  history
   41  gcloud config set project guardrails-eaba
   42  gcloud config set project accelerator-nc-info
   
   Continue
admin_root@cloudshell:~/cloudshell_open/accelerators_accelerateurs-gcp (accelerator-nc-info)$ gcloud config set project guardrails-eaba
Updated property [core/project].
   admin_root@cloudshell:~/cloudshell_open/accelerators_accelerateurs-gcp (guardrails-eaba)$ gcloud services enable cloudasset.googleapis.com
admin_root@cloudshell:~/cloudshell_open/accelerators_accelerateurs-gcp (guardrails-eaba)$ export MY_BUCKET_NAME=sccninfo-guardrails-validator-seed
admin_root@cloudshell:~/cloudshell_open/accelerators_accelerateurs-gcp (guardrails-eaba)$ gsutil mb gs://$MY_BUCKET_NAME
Creating gs://sccninfo-guardrails-validator-seed/...
PreconditionException: 412 'us' violates constraint 'constraints/gcp.resourceLocations'
   
   admin_root@cloudshell:~/cloudshell_open/accelerators_accelerateurs-gcp (guardrails-eaba)$ gcloud compute project-info describe --project guardrails-eaba | grep google-compute-default-region
admin_root@cloudshell:~/cloudshell_open/accelerators_accelerateurs-gcp (guardrails-eaba)$ gcloud compute project-info add-metadata --metadata google-compute-default-region=northamerica-northeast1,google-compute-default-zone=northamerica-northeast1a
Updated [https://www.googleapis.com/compute/v1/projects/guardrails-eaba].
admin_root@cloudshell:~/cloudshell_open/accelerators_accelerateurs-gcp (guardrails-eaba)$ gcloud compute project-info describe --project guardrails-eaba | grep google-compute-default-region
  - key: google-compute-default-region
admin_root@cloudshell:~/cloudshell_open/accelerators_accelerateurs-gcp (guardrails-eaba)$ gsutil mb -l northamerica-northeast1 gs://$MY_BUCKET_NAME
Creating gs://sccninfo-guardrails-validator-seed/...
admin_root@cloudshell:~/cloudshell_open/accelerators_accelerateurs-gcp (guardrails-eaba)$ gsutil ls
gs://bkt-guardrails-eaba-org-logs-5ufo/
gs://sccninfo-guardrails-validator-seed/
gs://sscncinfo-guardrails-assets/

Add docker build step for running tests in container

Current steps list a docker run without build, easy enough to figure out but for clarity it might make sense to add a build step as well.

# Container
docker build -t gc-guardrails:<tagname> .
docker run -v $(pwd):/app gc-guardrails:<tagname>

Validation readme changes for bucket location restriction

The validation readme requires a location restriction to northamerica-northeast1 or 2

Existing readme

admin_root@cloudshell:~/cloudshell_open/accelerators_accelerateurs-gcp (guardrails-eaba)$ gsutil mb gs://$MY_BUCKET_NAME
Creating gs://sccninfo-guardrails-validator-seed/...
PreconditionException: 412 'us' violates constraint 'constraints/gcp.resourceLocations'

Adjustment (optional set default region)

admin_root@cloudshell:~/cloudshell_open/accelerators_accelerateurs-gcp (guardrails-eaba)$ gsutil mb -l northamerica-northeast1 gs://$MY_BUCKET_NAME
Creating gs://sccninfo-guardrails-validator-seed/...

admin_root@cloudshell:~/cloudshell_open/accelerators_accelerateurs-gcp (guardrails-eaba)$ gsutil ls
gs://bkt-guardrails-eaba-org-logs-5ufo/
gs://sccninfo-guardrails-validator-seed/
gs://sscncinfo-guardrails-assets/

Guardrails failure Issues and hardcoded resource names in rego policy

A rego policy parse error is spotted from the cloud build issued by the guardrails validation function as shown below and there is not any validation report generated because of this error.

starting build "14d58fa0-fda5-4cb7-9a34-ce2c132154fd"

FETCHSOURCE
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint: 
hint: 	git config --global init.defaultBranch <name>
hint: 
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint: 
hint: 	git branch -m <name>
Initialized empty Git repository in /workspace/.git/
From https://source.developers.google.com/p/lzpe-js08-guardrailsjs08/r/LzPeCLD-guardrails-policies-csr
 * branch            8b1241263fe9ae3cfd766e244a8dd131b82a1ff9 -> FETCH_HEAD
HEAD is now at 8b12412 Merge pull request #9 from cartyc/main
BUILD
Starting Step #0
Step #0: Already have image (with digest): gcr.io/cloud-builders/gcloud
Step #0: Copying gs://lzpe565977066779assetsguardrailsjs08/organizations/565977066779.json...
Step #0: / [0 files][    0.0 B/  3.2 MiB]                                                
/ [1 files][  3.2 MiB/  3.2 MiB]                                                
Step #0: Operation completed over 1 objects/3.2 MiB.                                      
Finished Step #0
Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/docker
Finished Step #1
Starting Step #2
Step #2: Already have image (with digest): gcr.io/cloud-builders/docker
Step #2: Unable to find image 'northamerica-northeast1-docker.pkg.dev/lzpe-js08-guardrailsjs08/lzpecld-guardrails-af-registry-afr/lzpeccr-guardrails-policies-cntr:latest' locally
Step #2: latest: Pulling from lzpe-js08-guardrailsjs08/lzpecld-guardrails-af-registry-afr/lzpeccr-guardrails-policies-cntr
Step #2: 26c5c85e47da: Already exists
Step #2: 89c09bbbc10a: Pulling fs layer
Step #2: b4dab82f7782: Pulling fs layer
Step #2: 1b2c23d7ae23: Pulling fs layer
Step #2: 89c09bbbc10a: Verifying Checksum
Step #2: 89c09bbbc10a: Download complete
Step #2: b4dab82f7782: Verifying Checksum
Step #2: b4dab82f7782: Download complete
Step #2: 89c09bbbc10a: Pull complete
Step #2: b4dab82f7782: Pull complete
Step #2: 1b2c23d7ae23: Verifying Checksum
Step #2: 1b2c23d7ae23: Download complete
Step #2: 1b2c23d7ae23: Pull complete
Step #2: Digest: sha256:99e07a711bacfe921a049a43ec2b266570f6287d573bbc3a7553ec14ad9e9c64
Step #2: Status: Downloaded newer image for northamerica-northeast1-docker.pkg.dev/lzpe-js08-guardrailsjs08/lzpecld-guardrails-af-registry-afr/lzpeccr-guardrails-policies-cntr:latest
Step #2: Checking ./assets/asset_inventory.json
Step #2: Error: running test: build compiler: parse module: 1 error occurred: policies/11-logging-and-monitoring/11-Logging-and-Monitoring.rego:18: rego_parse_error: unexpected import path, must begin with one of: {data, input}, got: future
Step #2: 	import future.keywords.in
Step #2: 	       ^
Finished Step #2
Starting Step #3
Step #3: Already have image (with digest): gcr.io/cloud-builders/docker
Step #3: ./assets/asset_inventory.json
Step #3: 
Finished Step #3
Starting Step #4
Step #4: Already have image (with digest): gcr.io/cloud-builders/docker
Finished Step #4
Starting Step #5
Step #5: Already have image (with digest): gcr.io/cloud-builders/gcloud
Step #5: Copying file:///assets/565977066779.json [Content-Type=application/json]...
Step #5: / [0 files][    0.0 B/   31.0 B]                                                
/ [1 files][   31.0 B/   31.0 B]                                                
Step #5: Operation completed over 1 objects/31.0 B.                                       
Finished Step #5
PUSH
DONE

After upgrading the conftest version to latest, some hardcoded very specific resource names are found from the validation report generated as shown below.

./assets/asset_inventory.json
+---------+------+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| RESULT  | FILE | NAMESPACE |                                                                                                        MESSAGE                                                                                                        |
+---------+------+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| failure | -
| |
|         |
| failure | -
| |
| failure | -
| |
|         |
| failure | -
| |
| main
|
|
| main
|
| main
|
|
| main
|
| Guardrail # 11: No                                                                                                                                                                                                    |
| storage bucket matching                                                                                                                                                                                               |
| 'logginglogsink-goc' found.                                                                                                                                                                                           |
| Guardrail # 11: The log sink                                                                                                                                                                                          |
| 'org_log_sink' does not exist.                                                                                                                                                                                        |
| Guardrail # 5: Resource containerregistry.googleapis.com/Image

The proposed tested changes is attached below.
cloud-guardrails-gcp.patch

The changes from the patch.

  • Upgraded the conftest version to solve the rego keyword failures.
  • Refactored 11-Logging-and-Monitoring.rego to solve the sink and bucket name matching issues.

sed commands fail on mac

sed commands fail on mac with the following

./run.sh                                  
Checking ./assets/resource_inventory.json
sed: -: No such file or directory
sed: 1: "$ a ]}": command a expects \ followed by text

Workaround:
run as docker container

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.