canada-ca / cloud-guardrails-gcp Goto Github PK
View Code? Open in Web Editor NEWRecommended configuration guidance for Google Cloud Platform / Conseils de configuration recommandés pour Platforme infonuagique de Google
License: Other
Recommended configuration guidance for Google Cloud Platform / Conseils de configuration recommandés pour Platforme infonuagique de Google
License: Other
following
https://github.com/canada-ca/cloud-guardrails-gcp/tree/main/guardrails-validation
Workaround - turn off regional restriction on the project - or delete it on the parent and the rerun a terraform apply to get it back after
admin_@cloudshell:~/cloudshell_open (gr-bootstrap-ggz)$ gcloud services enable cloudasset.googleapis.com
Operation "operations/acat.p2-502392433631-09e81fe7-570c-44a3-8345-9852d82fd884" finished successfully.
admin_@cloudshell:~/cloudshell_open (gr-bootstrap-ggz)$ export MY_BUCKET_NAME=validation-ggz
admin_@cloudshell:~/cloudshell_open (gr-bootstrap-ggz)$ gsutil mb gs://$MY_BUCKET_NAME
Creating gs://validation-ggz/...
admin_@cloudshell:~/cloudshell_open (gr-bootstrap-ggz)$ gcloud asset export --output-path=gs://$MY_BUCKET_NAME/resource_inventory.json --content-type=resource --project=gr-bootstrap-ggz
Export in progress for root asset [projects/gr-bootstrap-ggz].
Use [gcloud asset operations describe projects/502392433631/operations/ExportAssets/RESOURCE/c6cfd41c3c7720348b468221cf6c688e] to check the status of the operation.
admin_@cloudshell:~/cloudshell_open (gr-bootstrap-ggz)
admin_@cloudshell:~/cloudshell_open (gr-bootstrap-ggz)$ git clone https://github.com/canada-ca/cloud-guardrails-gcp.git
admin_@cloudshell:~/cloudshell_open (gr-bootstrap-ggz)$ ls
accelerators_accelerateurs-gcp cloud-guardrails-gcp
admin_@cloudshell:~/cloudshell_open (gr-bootstrap-ggz)$ gsutil cp gs://$MY_BUCKET_NAME/resource_inventory.json ./assets
Copying gs://validation-ggz/resource_inventory.json...
/ [1 files][ 16.8 KiB/ 16.8 KiB]
Operation completed over 1 objects/16.8 KiB.
admin_@cloudshell:~/cloudshell_open (gr-bootstrap-ggz)
dmin_@cloudshell:~/cloudshell_open (gr-bootstrap-ggz)$ cd
accelerators_accelerateurs-gcp/ cloud-guardrails-gcp/
admin_@cloudshell:~/cloudshell_open (gr-bootstrap-ggz)$ cd cloud-guardrails-gcp/
.git/ guardrails/ guardrails-validation/
admin_@cloudshell:~/cloudshell_open (gr-bootstrap-ggz)$ cd cloud-guardrails-gcp/guardrails-validation/
admin_@cloudshell:~/cloudshell_open/cloud-guardrails-gcp/guardrails-validation (gr-bootstrap-ggz)$ ls
assets cloudbuild.yaml Dockerfile install.sh policies README.md run-all.sh run.sh tests.sh
admin_@cloudshell:~/cloudshell_open/cloud-guardrails-gcp/guardrails-validation (gr-bootstrap-ggz)$
min_@cloudshell:~/cloudshell_open/cloud-guardrails-gcp/guardrails-validation (gr-bootstrap-ggz)$ ./install.sh
--2022-11-20 15:39:04-- https://github.com/open-policy-agent/conftest/releases/download/v0.32.1/conftest_0.32.1_Linux_x86_64.tar.gz
Resolving github.com (github.com)... 140.82.113.3
Connecting to github.com (github.com)|140.82.113.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/178249461/a9c964a8-a471-41f8-aed7-86bca64ad3f8?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221120%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221120T153904Z&X-Amz-Expires=300&X-Amz-Signature=7b360ba6a1ab670e8c8957132cfcfda8d28cb797571ec78759636b3b6e402da8&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=178249461&response-content-disposition=attachment%3B%20filename%3Dconftest_0.32.1_Linux_x86_64.tar.gz&response-content-type=application%2Foctet-stream [following]
--2022-11-20 15:39:04-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/178249461/a9c964a8-a471-41f8-aed7-86bca64ad3f8?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221120%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221120T153904Z&X-Amz-Expires=300&X-Amz-Signature=7b360ba6a1ab670e8c8957132cfcfda8d28cb797571ec78759636b3b6e402da8&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=178249461&response-content-disposition=attachment%3B%20filename%3Dconftest_0.32.1_Linux_x86_64.tar.gz&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.108.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.110.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 12264521 (12M) [application/octet-stream]
Saving to: ‘conftest_0.32.1_Linux_x86_64.tar.gz’
conftest_0.32.1_Linux_x86_64.tar.gz 100%[=================================================================================>] 11.70M 44.0MB/s in 0.3s
2022-11-20 15:39:04 (44.0 MB/s) - ‘conftest_0.32.1_Linux_x86_64.tar.gz’ saved [12264521/12264521]
LICENSE
README.md
conftest
admin_@cloudshell:~/cloudshell_open/cloud-guardrails-gcp/guardrails-validation (gr-bootstrap-ggz)$
admin_@cloudshell:~/cloudshell_open/cloud-guardrails-gcp/guardrails-validation (gr-bootstrap-ggz)$ conftest --version
Conftest: 0.32.1
OPA: 0.40.0
run n/a due to missing json in the assets dir
dmin_@cloudshell:~/cloudshell_open/cloud-guardrails-gcp/guardrails-validation (gr-bootstrap-ggz)$ ./run.sh
Checking ./assets/*.json
cat: './assets/*.json': No such file or directory
admin_@cloudshell:~/cloudshell_open/cloud-guardrails-gcp/guardrails-validation (gr-bootstrap-ggz)$ ls ../
guardrails guardrails-validation LICENSE README.md
admin_@cloudshell:~/cloudshell_open/cloud-guardrails-gcp/guardrails-validation (gr-bootstrap-ggz)$ ls
assets cloudbuild.yaml Dockerfile install.sh policies README.md report.txt run-all.sh run.sh tests.sh
admin_@cloudshell:~/cloudshell_open/cloud-guardrails-gcp/guardrails-validation (gr-bootstrap-ggz)$ ls assets/
admin_@cloudshell:~/cloudshell_open/cloud-guardrails-gcp/guardrails-validation (gr-bootstrap-ggz)$
run-all also requires a rename of"my-unique-bucket-name" as well as addition storage admin role
admin_@cloudshell:~/cloudshell_open/cloud-guardrails-gcp/guardrails-validation (gr-bootstrap-ggz)$ ./run-all.sh
Your active configuration is: [cloudshell-22055]
Creating gs://my-unique-bucket-name/...
ServiceException: 409 A Cloud Storage bucket named 'my-unique-bucket-name' already exists. Try another name. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization.
ERROR: (gcloud.asset.export) code: 403
message: The billing account for the owning project is disabled in state closed
status: PERMISSION_DENIED
AccessDeniedException: 403 [email protected] does not have storage.objects.list access to the Google Cloud Storage bucket. Permission 'storage.objects.list' denied on resource (or it may not exist).
cat: './assets/*.json': No such file or directory
Doing a run on a recent guardrails install
following
https://github.com/canada-ca/cloud-guardrails-gcp/blob/main/guardrails-validation/README.md
from canada-ca/accelerators_accelerateurs-gcp#40
https://github.com/canada-ca/accelerators_accelerateurs-gcp/tree/main/deployment-templates/Terraform/guardrails
previous on [email protected]
cd accelerators_accelerateurs-gcp/
25 gcloud services enable cloudasset.googleapis.com
26 export MY_BUCKET_NAME=sccninfo-guardrails-validator
27 gsutil mb gs://$MY_BUCKET_NAME
28 gcloud asset export --output-path=gs://$MY_BUCKET_NAME/resource_inventory.json --content-type=resource --project=guardrails-eaba
29 gcloud config set project guardrails-eaba
30 gcloud services enable cloudasset.googleapis.com
31 export MY_BUCKET_NAME=sccninfo-guardrails-validator-gr
32 gsutil mb gs://$MY_BUCKET_NAME
33 gcloud config set project sscncinfo-seed-project
34 gcloud services enable cloudasset.googleapis.com
35 gcloud config set project sscncinfo-seed-project-seed
36 gcloud config set project sscncinfo-seed-project
37 gcloud services enable cloudasset.googleapis.com
38 export MY_BUCKET_NAME=sccninfo-guardrails-validator-seed
39 gsutil mb gs://$MY_BUCKET_NAME
40 history
41 gcloud config set project guardrails-eaba
42 gcloud config set project accelerator-nc-info
Continue
admin_root@cloudshell:~/cloudshell_open/accelerators_accelerateurs-gcp (accelerator-nc-info)$ gcloud config set project guardrails-eaba
Updated property [core/project].
admin_root@cloudshell:~/cloudshell_open/accelerators_accelerateurs-gcp (guardrails-eaba)$ gcloud services enable cloudasset.googleapis.com
admin_root@cloudshell:~/cloudshell_open/accelerators_accelerateurs-gcp (guardrails-eaba)$ export MY_BUCKET_NAME=sccninfo-guardrails-validator-seed
admin_root@cloudshell:~/cloudshell_open/accelerators_accelerateurs-gcp (guardrails-eaba)$ gsutil mb gs://$MY_BUCKET_NAME
Creating gs://sccninfo-guardrails-validator-seed/...
PreconditionException: 412 'us' violates constraint 'constraints/gcp.resourceLocations'
admin_root@cloudshell:~/cloudshell_open/accelerators_accelerateurs-gcp (guardrails-eaba)$ gcloud compute project-info describe --project guardrails-eaba | grep google-compute-default-region
admin_root@cloudshell:~/cloudshell_open/accelerators_accelerateurs-gcp (guardrails-eaba)$ gcloud compute project-info add-metadata --metadata google-compute-default-region=northamerica-northeast1,google-compute-default-zone=northamerica-northeast1a
Updated [https://www.googleapis.com/compute/v1/projects/guardrails-eaba].
admin_root@cloudshell:~/cloudshell_open/accelerators_accelerateurs-gcp (guardrails-eaba)$ gcloud compute project-info describe --project guardrails-eaba | grep google-compute-default-region
- key: google-compute-default-region
admin_root@cloudshell:~/cloudshell_open/accelerators_accelerateurs-gcp (guardrails-eaba)$ gsutil mb -l northamerica-northeast1 gs://$MY_BUCKET_NAME
Creating gs://sccninfo-guardrails-validator-seed/...
admin_root@cloudshell:~/cloudshell_open/accelerators_accelerateurs-gcp (guardrails-eaba)$ gsutil ls
gs://bkt-guardrails-eaba-org-logs-5ufo/
gs://sccninfo-guardrails-validator-seed/
gs://sscncinfo-guardrails-assets/
Current steps list a docker run without build, easy enough to figure out but for clarity it might make sense to add a build step as well.
# Container
docker build -t gc-guardrails:<tagname> .
docker run -v $(pwd):/app gc-guardrails:<tagname>
The validation readme requires a location restriction to northamerica-northeast1 or 2
Existing readme
admin_root@cloudshell:~/cloudshell_open/accelerators_accelerateurs-gcp (guardrails-eaba)$ gsutil mb gs://$MY_BUCKET_NAME
Creating gs://sccninfo-guardrails-validator-seed/...
PreconditionException: 412 'us' violates constraint 'constraints/gcp.resourceLocations'
Adjustment (optional set default region)
admin_root@cloudshell:~/cloudshell_open/accelerators_accelerateurs-gcp (guardrails-eaba)$ gsutil mb -l northamerica-northeast1 gs://$MY_BUCKET_NAME
Creating gs://sccninfo-guardrails-validator-seed/...
admin_root@cloudshell:~/cloudshell_open/accelerators_accelerateurs-gcp (guardrails-eaba)$ gsutil ls
gs://bkt-guardrails-eaba-org-logs-5ufo/
gs://sccninfo-guardrails-validator-seed/
gs://sscncinfo-guardrails-assets/
The following step downloads a file name "cai-dir" without a .json file extension and as a result run.sh cannot find the file.
https://github.com/canada-ca/cloud-guardrails-gcp/blob/main/guardrails-validation/README.md
gsutil cp gs://$MY_BUCKET_NAME/resource_inventory.json ./cai-dir
Should be updated to
gsutil cp gs://$MY_BUCKET_NAME/resource_inventory.json ./assets
A rego policy parse error is spotted from the cloud build issued by the guardrails validation function as shown below and there is not any validation report generated because of this error.
starting build "14d58fa0-fda5-4cb7-9a34-ce2c132154fd"
FETCHSOURCE
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: git branch -m <name>
Initialized empty Git repository in /workspace/.git/
From https://source.developers.google.com/p/lzpe-js08-guardrailsjs08/r/LzPeCLD-guardrails-policies-csr
* branch 8b1241263fe9ae3cfd766e244a8dd131b82a1ff9 -> FETCH_HEAD
HEAD is now at 8b12412 Merge pull request #9 from cartyc/main
BUILD
Starting Step #0
Step #0: Already have image (with digest): gcr.io/cloud-builders/gcloud
Step #0: Copying gs://lzpe565977066779assetsguardrailsjs08/organizations/565977066779.json...
Step #0: / [0 files][ 0.0 B/ 3.2 MiB]
/ [1 files][ 3.2 MiB/ 3.2 MiB]
Step #0: Operation completed over 1 objects/3.2 MiB.
Finished Step #0
Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/docker
Finished Step #1
Starting Step #2
Step #2: Already have image (with digest): gcr.io/cloud-builders/docker
Step #2: Unable to find image 'northamerica-northeast1-docker.pkg.dev/lzpe-js08-guardrailsjs08/lzpecld-guardrails-af-registry-afr/lzpeccr-guardrails-policies-cntr:latest' locally
Step #2: latest: Pulling from lzpe-js08-guardrailsjs08/lzpecld-guardrails-af-registry-afr/lzpeccr-guardrails-policies-cntr
Step #2: 26c5c85e47da: Already exists
Step #2: 89c09bbbc10a: Pulling fs layer
Step #2: b4dab82f7782: Pulling fs layer
Step #2: 1b2c23d7ae23: Pulling fs layer
Step #2: 89c09bbbc10a: Verifying Checksum
Step #2: 89c09bbbc10a: Download complete
Step #2: b4dab82f7782: Verifying Checksum
Step #2: b4dab82f7782: Download complete
Step #2: 89c09bbbc10a: Pull complete
Step #2: b4dab82f7782: Pull complete
Step #2: 1b2c23d7ae23: Verifying Checksum
Step #2: 1b2c23d7ae23: Download complete
Step #2: 1b2c23d7ae23: Pull complete
Step #2: Digest: sha256:99e07a711bacfe921a049a43ec2b266570f6287d573bbc3a7553ec14ad9e9c64
Step #2: Status: Downloaded newer image for northamerica-northeast1-docker.pkg.dev/lzpe-js08-guardrailsjs08/lzpecld-guardrails-af-registry-afr/lzpeccr-guardrails-policies-cntr:latest
Step #2: Checking ./assets/asset_inventory.json
Step #2: Error: running test: build compiler: parse module: 1 error occurred: policies/11-logging-and-monitoring/11-Logging-and-Monitoring.rego:18: rego_parse_error: unexpected import path, must begin with one of: {data, input}, got: future
Step #2: import future.keywords.in
Step #2: ^
Finished Step #2
Starting Step #3
Step #3: Already have image (with digest): gcr.io/cloud-builders/docker
Step #3: ./assets/asset_inventory.json
Step #3:
Finished Step #3
Starting Step #4
Step #4: Already have image (with digest): gcr.io/cloud-builders/docker
Finished Step #4
Starting Step #5
Step #5: Already have image (with digest): gcr.io/cloud-builders/gcloud
Step #5: Copying file:///assets/565977066779.json [Content-Type=application/json]...
Step #5: / [0 files][ 0.0 B/ 31.0 B]
/ [1 files][ 31.0 B/ 31.0 B]
Step #5: Operation completed over 1 objects/31.0 B.
Finished Step #5
PUSH
DONE
After upgrading the conftest version to latest, some hardcoded very specific resource names are found from the validation report generated as shown below.
./assets/asset_inventory.json
+---------+------+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| RESULT | FILE | NAMESPACE | MESSAGE |
+---------+------+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| failure | -
| |
| |
| failure | -
| |
| failure | -
| |
| |
| failure | -
| |
| main
|
|
| main
|
| main
|
|
| main
|
| Guardrail # 11: No |
| storage bucket matching |
| 'logginglogsink-goc' found. |
| Guardrail # 11: The log sink |
| 'org_log_sink' does not exist. |
| Guardrail # 5: Resource containerregistry.googleapis.com/Image
The proposed tested changes is attached below.
cloud-guardrails-gcp.patch
The changes from the patch.
sed commands fail on mac with the following
./run.sh
Checking ./assets/resource_inventory.json
sed: -: No such file or directory
sed: 1: "$ a ]}": command a expects \ followed by text
Workaround:
run as docker container
It would be nice to see more information on the successes, so they can be added to the evidence package submission.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.