googlecloudplatform / blueprints

Sample blueprints for Google Cloud

Blueprints

This repository contains sample blueprints for Google Cloud.

blueprints's Issues

Evaluate depends-on for blueprints

Now that kpt supports depends-on, we should evaluate adding this in blueprints. A common usecase is depending on service enablement to prevent activation errors on service resources which results in suboptimal UX.

org-id quirk in hierarchy from fn eval order

The hierarchy blueprints have the org-id setter.yaml here:

blueprints/catalog/hierarchy/team/Kptfile

Lines 7 to 11 in 07c398d

	pipeline:
	mutators:
	- image: gcr.io/kpt-fn/apply-setters:v0.1
	configPath: setters.yaml
	- image: gcr.io/yakima-eap/generate-folders:latest

blueprints/catalog/hierarchy/team/setters.yaml

Line 7 in 07c398d

org-id: "123456789012"

Which seems to be designed for overridden by being nested in the landing zone:

blueprints/catalog/landing-zone/setters.yaml

Lines 14 to 15 in 07c398d

	# Organization ID and billing account
	org-id: "123456789012"

But the order of functions seems to be:

1. apply the hierarchy setter.yaml
2. expand the local-config kind: ResourceHierarchy into kind: Folder resources
3. apply the landing-zone setters.yaml

The effect is that the ResourceHierarchy.spec.parentRef.external always has the correct landing-zone org-id, but the derived top-level Folder.spec.organizationRef.external ends up with the placeholder value '123456789012'. It also shows up as an "permission denied" error once deployed so it took longer to figure out that I'd like to admit. :)

I believe the expected behavior would be for the gcr.io/yakima-eap/generate-folders:latest mutator to have written external: '123456789012' # kpt-set: ${org-id} to the top level Folder? That way it would be technically correct for the state of the files at step 2, but would also be updated with the final org-id at step 3.

hope to see ResourceManagerPolicy with condition of expression of resource.matchTag(...)

many healthcare clients would like to implement organization policies to conditionally restrict PHI data in the regions based on the project's tag: us or tag: eu at policies
The doc page does not provide a config connector method to create the conditional policy but the gcloud command method only. Is creating such sample policy in the landing zone folder in scope?

External HTTP(s) Loadbalancer blueprint

I would love to see a blueprint for an end to end HTTP(s) Loadbalancer example:

• External LB listens on port 80
• Sample instance is created, added to an unmanaged instance group
• Sample instance replies to a healthcheck
• Instance group used as backend, replies to HTTP query with a sample HTML response

Additional example can contain SSL certificate, DNS.

doc bug: syncBranch field in the ConfigManagement object isn't valid for config sync

I tried to kubectl edit ConfigManagement -n config-management per the following in the doc:

Note: To use an alternate branch for Config Sync, you can specify the branch name using the syncBranch field in the ConfigManagement object followed by reapplying the blueprint.

Then the following error in the log happened:

manager
"msg"="error transforming manifest" "error"="setting spec.git requires setting spec.enableLegacyFields to true"

Then I tried to change branch: main to branch: master at kubectl edit RootSync -n config-management-system but the changes were overwritten after a minute. I think the doc page needs to have better instructions. I am guessing changing branch: main line is the correct instruction. Please confirm with config management product engineering team and update the doc.

${prefix} setter not present in networking/network

I've been following the landing zone tutorial, but with the kpt-v1 branch and 1.0.0-beta.1 releases download.

On the step to create the network I saw an error related to ${prefix} once the network package was put in place.

The same error reproduces rendering the source package directly.

When the apply-setter is running at the network level it seems it isn't picking up the ${prefix} value from network/subnet/setters.yaml?

Steps to reproduce:

git clone --branch=kpt-v1 https://github.com/GoogleCloudPlatform/blueprints
kpt fn render blueprints/catalog/networking/network

Package "network/subnet": 
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.1"
  Results:
    [INFO] set field value to "network-name-router-nat" in file "subnet/nat.yaml" in field "metadata.name"
    [INFO] set field value to "networking" in file "subnet/nat.yaml" in field "metadata.namespace"
    [INFO] set field value to "project-id" in file "subnet/nat.yaml" in field "metadata.annotations.cnrm.cloud.google.com/project-id"
    [INFO] set field value to "us-central1" in file "subnet/nat.yaml" in field "spec.region"
    ...(13 line(s) truncated, use '--truncate-output=false' to disable)

Package "network/vpc": 
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.1"
  Results:
    [INFO] set field value to "project-id-compute" in file "vpc/services.yaml" in field "metadata.name"
    [INFO] set field value to "project-id" in file "vpc/services.yaml" in field "metadata.annotations.cnrm.cloud.google.com/project-id"
    [INFO] set field value to "network-name" in file "vpc/vpc.yaml" in field "metadata.name"
    [INFO] set field value to "networking" in file "vpc/vpc.yaml" in field "metadata.namespace"
    ...(1 line(s) truncated, use '--truncate-output=false' to disable)

Package "network": 
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1"
[FAIL] "gcr.io/kpt-fn/apply-setters:v0.1"
  Results:
    [ERROR] failed to apply setters: values for setters [${prefix}] must be provided
  Stderr:
    "values for setters [${prefix}] must be providedvalues for setters [${prefix}] must be provided"
    ""
  Exit code: 1

Comments should not rely on field order

@morgante

https://github.com/GoogleCloudPlatform/blueprints/blob/kpt-v1/catalog/gitops/setters.yaml#L6

In this file, the comment relies on the order of fields. You can do that with beta.2 of kpt but beta.1 of kpt may not retain the initial order of fields. Please audit other files and ensure comments are tied to particular field and not rely on the field order.

Related: kptdev/kpt#2332

cc @mikebz @droot

Question about kpt-set: ${projects-namespace} not specified in setters.yaml

I noticed that line at

blueprints/catalog/project/project.yaml

Line 18 in 2b8afca

namespace: projects # kpt-set: ${projects-namespace}

has namespace: projects # kpt-set: ${projects-namespace} but it's not defined in setters.yaml. By following the steps, the upper level folder would have landing-zone/setters.yaml which also does not have projects-namespace defined. I checked the deployment repo and the final value is namespace: projects. Should @morgante define projects-namespace in project/setters.yaml?

Missing required "platform-project-id" in setters.yaml

This issue appears in two different blueprints:

https://github.com/GoogleCloudPlatform/blueprints/blob/main/catalog/acm/setters.yaml

https://github.com/GoogleCloudPlatform/blueprints/blob/main/catalog/anthos-cluster/setters.yaml

In both cases, we have "platform-project-id" missing in setters.yaml. As a result if customers try to apply these two blueprints they may see failure because this value is not set.

SA sync-krmapihost-config-controller-1 is more than 30 characters

The following command returns error
kubectl get gcp -n config-control -o yaml

status:
    conditions:
    - lastTransitionTime: "2022-08-02T00:20:03Z"
      message: 'Update call failed: error applying desired state: summary: Error creating
        service account: googleapi: Error 400: The account ID "sync-krmapihost-config-controller-1"
        does not have a length between 6 and 30., badRequest'
      reason: UpdateFailed
      status: "False"
      type: Ready
    observedGeneration: 1

  status:
    conditions:
    - lastTransitionTime: "2022-08-02T00:20:03Z"
      message: 'Update call failed: error setting policy: error applying changes:
        summary: Error setting IAM policy for project "$PROJECT_ID": googleapi:
        Error 400: Service account sync-krmapihost-config-controller-1@landing-zone-$PROJECT_ID.iam.gserviceaccount.com
        does not exist., badRequest'
      reason: UpdateFailed
      status: "False"
      type: Ready
    observedGeneration: 1

Observed:

blueprints/catalog/gitops/configsync/configsync-iam.yaml

Line 19 in 1ac8e49

name: sync-cluster-name # kpt-set: sync-${cluster-name}

after kfp fn render produces a SA of length==35. GCP does not allow creating service account with name > 30.
Fix: change from sync-${cluster-name} to sync-${sa-cluster-name}
3 lines in configsync-iam.yaml. One of them is

  name: sync-krmapihost-config-controller-1 # kpt-set: sync-${sa-cluster-name}

1 line in rootsync.yaml:

    gcpServiceAccountEmail: sync-krmapihost-config-controller-1@$PROJECT_ID.iam.gserviceaccount.com # kpt-set: sync-${sa-cluster-name}@${project-id}.iam.gserviceaccount.com

add value in setters.yaml: sa-cluster-name: krmapihost-cc-1

svpc service project uses same setter for networking and project ns

svpc service project uses same setter for networking and project ns. This can cause the blueprint to be deployed in the projects ns where the SA will not have necessary permissions.

config controller throws error because of hierarchy.yaml from the /catalog/hierarchy/simple blueprint

after migrating to kpt v1 and using the actual hierarchy blueprint from master (/catalog/hierarchy/simple) we are getting the following error in our config controller cluster

[1] KNV1021: No CustomResourceDefinition is defined for the type "ResourceHierarchy.blueprints.cloud.google.com" in the cluster. Resource types that are not native Kubernetes objects must have a CustomResourceDefinition. source: config/viesure/hierarchy/hierarchy.yaml namespace: hierarchy metadata.name: root-hierarchy group: blueprints.cloud.google.com version: v1alpha3 kind: ResourceHierarchy For more information, see https://g.co/cloud/acm-errors#knv1021

I am not sure whether this is an issue of the git importer from the config controller (as the hierarchy.yaml is annotated with config.kubernetes.io/local-config: 'true' and should be basically ignored by the git importer I guess???) or whether the kpt function for generating the folder structure should be responsible for removing the hierarchy.yaml from the output.

the old hierarchy blueprint for kpt < 1.0 which we used before removed the hierarchy.yaml from the output.

Validator functions should be moved to validators section in Kptfile

@morgante https://github.com/GoogleCloudPlatform/blueprints/blob/kpt-v1/catalog/landing-zone/Kptfile#L11

This function looks like a validator from its name. Please move it to validators section in Kptfile so that kpt can apply guard rails to not modify resources. Currently, kpt fn render leads to unnecessary diffs.

cc @mikebz @droot

error: object file .git/objects/3c/2dda6e6ba2c7929b2cb5e4c71c9584eb62654c is empty

Package "landing-zone":
Fetching https://github.com/GoogleCloudPlatform/blueprints@main
error: object file .git/objects/3c/2dda6e6ba2c7929b2cb5e4c71c9584eb62654c is empty
fatal: loose object 3c2dda6e6ba2c7929b2cb5e4c71c9584eb62654c (stored in .git/objects/3c/2dda6e6ba2c7929b2cb5e4c71c9584eb62654c) is corrupt
Error: Failed to execute git command "git origin origin --depth=1 main" against repo "https://github.com/GoogleCloudPlatform/blueprints" for reference "main" 

Details:
error: object file .git/objects/3c/2dda6e6ba2c7929b2cb5e4c71c9584eb62654c is empty
fatal: loose object 3c2dda6e6ba2c7929b2cb5e4c71c9584eb62654c (stored in .git/objects/3c/2dda6e6ba2c7929b2cb5e4c71c9584eb62654c) is corrupt 
fatal: pathspec 'landing-zone/' did not match any files
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

How to set the vCPU and memory resources for a GKE cluster?

We want to use the blueprint to create a cluster with minimum 128 GB memory. I didn't see any vCPU or memory resources configured for the cluster in https://github.com/GoogleCloudPlatform/blueprints/tree/main/catalog/anthos-cluster. Is there a way to specify the vCPU and memory resources in the blueprint?

Add local-config anno to all functionConfigs and kptfiles

• Add local-config anno to all functionConfigs and kptfiles
• Validate remove-local-config-resources works as expected for kf resources in pipeline

@droot suggested kpt fn eval . -i set-annotations:v0.1.4 --match-kind Kptfile -- config.kubernetes.io/local-config="true". We could also use this to match based on path for setters.yaml.

x-ref: kptdev/kpt#2894

Move resource reference links section into table in readmes

context: #128 (comment)

Add makefile target for regenerating docs and enable doc gen lint check in CI

Missing GCPEnforceNamingV2

kubectl get constrainttemplates -A|grep GCPEnforceNamingV2 returns nothing. @morgante , can you push a commit to fix it? I discussed with Poonam and she investigated with me. The following code file's folder should have the constraint template.

blueprints/catalog/hierarchy/env-bu/policies/naming-constraint.yaml

Line 15 in 2b8afca

kind: GCPEnforceNamingV2

The same applies to other 3 folders at https://github.com/GoogleCloudPlatform/blueprints/tree/2b8afca2ef0662cf5ea39c797832ac9c5ea67c7e/catalog/hierarchy

Onboard repo to use OSS prow

Attaching owner information to a resource created through blueprint

As for resources directly created through GCP console, the owner info is available in audit logs through the "principalEmail" field

But for the resources created through Config controller, will have the primary service account email of the namespace as the principalEmail.

How can we attach an owner information (email) to a resource created through config controller. I believe labels cannot be used for that (as per https://cloud.google.com/resource-manager/docs/creating-managing-labels#common-uses) and also labels will not allow special characters as values. Is there any other ways or custom annotation we can add for owner email?

cloud source repo defaults to master, not main

I followed the instructions but the source-repo and deployment-repo still produced the master branch which cause the error in the log assuming landing-zone-dev-0 is the project ID:

hydration-controller
Error in the git-sync container: {"Msg":"unexpected error syncing repo, will retry","Err":"Run(git clone -v --no-checkout -b main --depth 1 https://source.developers.google.com/p/landing-zone-dev-0/r/deployment-repo /repo/source): exit status 128: { stdout: "", stderr: "Cloning into '/repo/source'...\nPOST git-upload-pack (299 bytes)\nwarning: Could not find remote branch main to clone.\nfatal: Remote branch main not found in upstream origin\n" }","Args":{}}
git-sync
"msg"="unexpected error syncing repo, will retry" "error"="Run(git clone -v --no-checkout -b main --depth 1 https://source.developers.google.com/p/landing-zone-dev-0/r/deployment-repo /repo/source): exit status 128: { stdout: "", stderr: "Cloning into '/repo/source'...\nPOST git-upload-pack (299 bytes)\nwarning: Could not find remote branch main to clone.\nfatal: Remote branch main not found in upstream origin\n" }"

Then I tried to kubectl edit ConfigManagement -n config-management per the following in the doc:

Note: To use an alternate branch for Config Sync, you can specify the branch name using the syncBranch field in the ConfigManagement object followed by reapplying the blueprint.

Then the following error in the log happened:

manager
"msg"="error transforming manifest" "error"="setting spec.git requires setting spec.enableLegacyFields to true"

Then I tried to change branch: main to branch: master at kubectl edit RootSync -n config-management-system but the changes were overwritten after a minute.

Finally, I git checkout -b main and git push --set-upstream origin main to create the master branch to resolve the initial error.

re enable lint check status

lint check status disabled via #31 due to kptdev/kpt#2332

kpt live apply command stuck while the yaml has the wrong namespace.

I followed the instructions at https://github.com/GoogleCloudPlatform/blueprints/tree/a814f19df7a68e9e099203468177b4921bbb102b/catalog/project to create a project specific namespace. When the following IAM yaml file is applied with the commented line uncommented, the kpt live apply returned within 5 seconds. It took me almost 1 hour to figure out I was missing the namespace. None of the logs had any clues about why the kpt live apply was stuck as namespace: delete-me-356017 was missing. No IAM policy was created as expected. It was annoying to have a bug like this without explicit error.
iam-exampe.yaml

apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMPolicyMember
metadata:
  name: iam-editor-delete-me-356017
#  namespace: delete-me-356017
spec:
  member: user:[email protected] # kpt-set: ${google-account-user}
  role: roles/editor
  resourceRef:
    kind: Project
    namespace: projects # kpt-set: ${projects-namespace}
    name: name-of-kind-project # kpt-set: ${project-name}

Kptfile

apiVersion: kpt.dev/v1
kind: Kptfile
metadata:
  name: delete-me-356017
  annotations:
    blueprints.cloud.google.com/title: blueprint of project delete-me-356017
info:
  description: blueprint for project delete-me-356017
pipeline:
  mutators:
    - image: gcr.io/kpt-fn/apply-setters:v0.1
      configPath: setters.yaml

setters.yaml

apiVersion: v1
kind: ConfigMap
metadata: # kpt-merge: /setters
  name: setters
data:
  projects-namespace: projects
  project-name: delete-me-356017
  google-account-user: user:[email protected]

Observe the stuck command. I had to Ctrl+C.

$ kpt fn render && kpt live init --namespace ${NAMESPACE}
Package "delete-me-356017": 
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.1" in 400ms
  Results:
    [info] spec.bindings[0].members[0].member: set field value to "user:[email protected]"
    [info] spec.resourceRef.namespace: set field value to "projects"
    [info] spec.resourceRef.name: set field value to "delete-me-356017"

Successfully executed 1 function(s) in 1 package(s).
initializing Kptfile inventory info (namespace: config-control)...success
$ kpt live apply 
W0711 11:28:12.717846 2832869 gcp.go:120] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.25+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
installing inventory ResourceGroup CRD.
configmap/setters created
iampartialpolicy.iam.cnrm.cloud.google.com/iam-storage-delete-me-356017 created
2 resource(s) applied. 2 created, 0 unchanged, 0 configured, 0 failed
configmap/setters reconcile pending
E0711 11:28:14.092293 2832869 task.go:270] Empty object UID from ResourceCache (status: NotFound): default_iam-storage-delete-me-356017_iam.cnrm.cloud.google.com_IAMPartialPolicy
iampartialpolicy.iam.cnrm.cloud.google.com/iam-storage-delete-me-356017 reconcile pending
configmap/setters reconciled
^C

Possibly missing projects:cnrm-viewer for cnrm-controller-manager-logging

After deploying this package, embedded in landing-zone/logging/bigquery-export...

blueprints/catalog/log-export/org/bigquery-export/Kptfile

Lines 2 to 6 in 07c398d

	kind: Kptfile
	metadata:
	name: bigquery-export
	info:
	description: Creates a log export on a organization that sinks to BigQuery

One of the resources remained in an error state with this message:

[
  "logging",
  "bqlogexportdataset",
  "Update call failed: error fetching live state: error getting ID for resource: error getting value from reference: error resolving reference field: error getting referenced resource from API server: error getting referenced resource projects/logging-129082878300 with GroupVersionKind resourcemanager.cnrm.cloud.google.com/v1beta1, Kind=Project from API server: projects.resourcemanager.cnrm.cloud.google.com \"logging-129082878300\" is forbidden: User \"system:serviceaccount:cnrm-system:cnrm-controller-manager-logging\" cannot get resource \"projects\" in API group \"resourcemanager.cnrm.cloud.google.com\" in the namespace \"projects\""
]

I added this to landing-zone/namespaces/logging.yaml which seems to fix that

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: # kpt-merge: projects/allow-resource-reference-from-logging
  name: allow-resource-reference-from-logging
  namespace: projects
roleRef:
  name: cnrm-viewer
  kind: ClusterRole
  apiGroup: rbac.authorization.k8s.io
subjects:
  - name: cnrm-controller-manager-logging
    namespace: cnrm-system
    kind: ServiceAccount

based on seeing the similar role binding being applied to the networking sa

blueprints/catalog/landing-zone/namespaces/networking.yaml

Lines 108 to 120 in 07c398d

	apiVersion: rbac.authorization.k8s.io/v1
	kind: RoleBinding
	metadata:
	name: allow-resource-reference-from-networking
	namespace: projects
	roleRef:
	name: cnrm-viewer
	kind: ClusterRole
	apiGroup: rbac.authorization.k8s.io
	subjects:
	- name: cnrm-controller-manager-networking
	namespace: cnrm-system
	kind: ServiceAccount

Kpt render taking more time when there is large number of blueprints

We could see a behaviour that the kpt fn render in the blueprint root directory is taking more time when there are large number of blueprints.

As kpt is git aware, is there any way that the render will happen only for the changed or newly created blueprints in the repository?
Or any other steps to improve the rendering stage performance as we are planning to provision large number of projects and resources through our gitops implementation.

Please advise.

Unsupported ACM version 1.9.0 in GKEHubFeatureMembership

Failed to deploy the in GKEHubMembership/hub-membership in the blueprint anthos-cluster/acm/config-mgmt-csr.yaml. The error shows that acm v1.9.0 is not supported.

$ kubectl describe GKEHubFeatureMembership acm-membership-cluster-5 -n config-control
...
Status:
  Conditions:
    Last Transition Time:  2022-08-01T23:20:39Z
    Message:               Update call failed: error applying desired state: googleapi: Error 400: InvalidValueError for field version: unsupported ACM version 1.9.0 for Membership projects/817917611258/locations/global/memberships/hub-membership-cluster-5
    Reason:                UpdateFailed
    Status:                False
    Type:                  Ready
  Observed Generation:     1

Suggested fix:
configmanagement.version in feedbackGKEHubFeatureMembership is an optional field to set the ACM version. Defaults to the latest version. Hence, we should leave out the version and let it install the latest supported one.

follow-up questions:

1. Which component is no longer supported in v1.9.0?
2. Will the new ACM releases be compatible with the blueprint? Is it safe to use the latest version?
3. Is there notifications for the updates of the blueprints? We'd like to keep our code synced with the major releases.

Default branch is master in cloud source repo

Hi, @morgante , I realize that the steps would fail to create infra blueprints as cloud source repo has default branch of master. Here's my fix. I talked to @markebalch about it. The full story is in the doc

kpt v1 setter comments

v1 branch setter comments maybe reordered due to kptdev/kpt#2332

x-ref: https://github.com/GoogleCloudPlatform/blueprints/pull/20/files#r659989916

KRM updater is not working

Our latest release PR https://github.com/GoogleCloudPlatform/blueprints/pull/130/files does not seem to be running updaters for bumping versions. Last PR released a month ago, the updaters worked #119

/cc @chingor13

KRM releaser does not seem to pick up pkg contents for lz-lite

Release-please KRM updater does not seem to pick up landing-zone-lite package contents and hence not updating attribution. This maybe related to googleapis/release-please#1164