tektoncd / chains Goto Github PK
View Code? Open in Web Editor NEWSupply Chain Security in Tekton Pipelines
License: Apache License 2.0
Supply Chain Security in Tekton Pipelines
License: Apache License 2.0
https://github.com/sigstore/cosign/blob/main/SPEC.md
When we sign OCI artifacts with "simplesigning" format we should make sure it works with this spec. I think it does already, should just double check and add a test.
This is about to land in cosign: sigstore/cosign#335
Fulcio can issue certs for OIDC accounts. The prod instance supports GCP ones, local instances can do any.
We can get a new cert for every signature, or get one for 5-10m and rotate before it expires.
We'll need to add on other "verification bundles" into the signatures/payloads for all the storage backends.
I think some sort of simple hello world walk through / example would be useful, so that folks new to chains can follow something along and get an idea of how everything rigs together.
The current DEVELOPER.md
lists how to load with ko. It would be nice to then point to a simple tutorial of how to setup secrets and then get signing to occur against a pipeline or task. As per title, it could be as simple as an echo output.
I don't think this is enforced right now because some of our files have headers and some don't ~
/*
Copyright 2021 The Tekton Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
An easy way to start this would be to turn on experimental mode for cosign, which will upload OCI signatures to the tlog for us.
Eventually, we could store ITE6 attestations in rekor as well!
We add a UUID to annotations for payload-taskrun and signature-taskrun
"annotations": {
"chains.tekton.dev/payload-taskrun-8c0e9bbc-192b-4ea3-8a6d-26d2ac1e0024": "eyJjb25kaXRpb25zIjpbeyJ0eXBlIjoiU3VjY2VlZGVkIiwic3RhdHVzIjoiVHJ1ZSIsImxhc3RUcmFuc2l0aW9uVGltZSI6IjIwMjEtMDUtMjFUMDk6NDI6MTFaIiwicmVhc29uIjoiU3VjY2VlZGVkIiwibWVzc2FnZSI6IkFsbCBTdGVwcyBoYXZlIGNvbXBsZXRlZCBleGVjdXRpbmcifV0sInBvZE5hbWUiOiJzYW1wbGUtcGlwZWxpbmUtcnVuLWVjaG8tc29tZXRoaW5nLWJqOHI3LXBvZC1zd3J3cyIsInN0YXJ0VGltZSI6IjIwMjEtMDUtMjFUMDk6NDI6MDRaIiwiY29tcGxldGlvblRpbWUiOiIyMDIxLTA1LTIxVDA5OjQyOjExWiIsInN0ZXBzIjpbeyJ0ZXJtaW5hdGVkIjp7ImV4aXRDb2RlIjowLCJyZWFzb24iOiJDb21wbGV0ZWQiLCJzdGFydGVkQXQiOiIyMDIxLTA1LTIxVDA5OjQyOjEwWiIsImZpbmlzaGVkQXQiOiIyMDIxLTA1LTIxVDA5OjQyOjEwWiIsImNvbnRhaW5lcklEIjoiY29udGFpbmVyZDovL2JkNzBkNDBkMGU4NThkNjc0ZDM3ZjIyMDg3NGIzODI3NzExZmE3YzEyYzBmM2I0Yjk0YTE4YTkwNDQxZTc2NjYifSwibmFtZSI6ImVjaG8iLCJjb250YWluZXIiOiJzdGVwLWVjaG8iLCJpbWFnZUlEIjoicmVnaXN0cnkuYWNjZXNzLnJlZGhhdC5jb20vdWJpOC1taW5pbWFsQHNoYTI1NjpmYzc1NTMyYTIwYzFlN2ViMDUxMmEwM2ZlYWM0NjU1NDI3OGJjZTk0NmNmNDU0YTc4ZTExNDMzZTM5YTY2ZDJkIn1dLCJ0YXNrU3BlYyI6eyJzdGVwcyI6W3sibmFtZSI6ImVjaG8iLCJpbWFnZSI6InJlZ2lzdHJ5LmFjY2Vzcy5yZWRoYXQuY29tL3ViaTgtbWluaW1hbCIsImNvbW1hbmQiOlsiZWNobyJdLCJhcmdzIjpbImhlbGxvIHdvcmxkIl0sInJlc291cmNlcyI6e319XX19",
"chains.tekton.dev/signature-taskrun-8c0e9bbc-192b-4ea3-8a6d-26d2ac1e0024": "MEQCIHOOgvkybfaea7moBVHfhyH+BZTBVtdDkxrqiBghWkWFAiBOLuNtvR6nML7TsgddcfKRB+fnbpBpjcr+NLn/vGrFrA==",
"chains.tekton.dev/signed": "true",
"pipeline.tekton.dev/release": "devel"
},
I guess there is a reason for this? I ask as it might it more cumbersome for folks to parse out the values as they will need to find the UUID or use some sort of regex or wildcard to ignore the appended uuid.
We should reorganize the readme a bit to make this more approachable. Some ideas:
They're signed, so maybe we can stick them in the same .sig image with a different media type?
Let's release v0.1.0 of chains! Things we should do before then:
And it would be nice to have:
Remove PGP? We added it as the first signer, because it's all that was really in use. Since, it's been deprecated from the go crypto library, and we have other options.
If no one is using PGP, we should remove it.
ref golang/go#44226
The fulcio one most likely can't run in our integration test cluster because I'm guessing Workload Identity isn't set up
It could still be nice to have the test though, and we could manually run it before releases or something
We can just setup registry:2
or something from Dockerhub in a local service as part of the e2e setup, then push there during tests.
This may well not be a bug, but figured I would use this to track my findings.
I am following the readme steps, but there are no signing annotations such as payload / signature. I have tried this with gpg, cosign and x509.
Steps followed
gpg --gen-key
gpg --export-secret-key --armor $keyname > pgp.private-key
gpg --export --armor $keyname > pgp.public-key
kubectl create secret generic signing-secrets -n tekton-chains --from-file=pgp.passphrase --from-file=pgp.private-key --from-file=pgp.public-key
kubectl create -f examples/task-output-image.yaml
NAME SUCCEEDED REASON STARTTIME COMPLETIONTIME
build-push-run-output-image-fmjmx True Succeeded 64m 64m
tkn taskrun list
NAME STARTED DURATION STATUS
build-push-run-output-image-fmjmx 1 hour ago 23 seconds Succeeded
Resulting taskrun data, note "chains.tekton.dev/signed": "true"
, yet no signature or payload annotations (its the same with cosign keys, x509)
kubectl get taskrun build-push-run-output-image-fmjmx -o=json | jq
{
"apiVersion": "tekton.dev/v1beta1",
"kind": "TaskRun",
"metadata": {
"annotations": {
"chains.tekton.dev/signed": "true",
"pipeline.tekton.dev/release": "devel"
},
"creationTimestamp": "2021-05-19T09:57:49Z",
"generateName": "build-push-run-output-image-",
"generation": 1,
"labels": {
"app.kubernetes.io/managed-by": "tekton-pipelines"
},
"name": "build-push-run-output-image-fmjmx",
"namespace": "default",
"resourceVersion": "134736",
"uid": "d2f71d21-73fa-4957-a08c-88a1729d3c84"
},
"spec": {
"resources": {
"inputs": [
{
"name": "sourcerepo",
"resourceSpec": {
"params": [
{
"name": "revision",
"value": "v0.32.0"
},
{
"name": "url",
"value": "https://github.com/GoogleContainerTools/skaffold"
}
],
"type": "git"
}
}
],
"outputs": [
{
"name": "builtImage",
"resourceSpec": {
"params": [
{
"name": "url",
"value": "gcr.io/foo/bar"
}
],
"type": "image"
}
}
]
},
"serviceAccountName": "default",
"taskSpec": {
"resources": {
"inputs": [
{
"name": "sourcerepo",
"type": "git"
}
],
"outputs": [
{
"name": "builtImage",
"targetPath": "/workspace/sourcerepo",
"type": "image"
}
]
},
"steps": [
{
"image": "busybox",
"name": "build-and-push",
"resources": {},
"script": "set -e\ncat <<EOF > $(inputs.resources.sourcerepo.path)/index.json\n{\n\"schemaVersion\": 2,\n\"manifests\": [\n {\n \"mediaType\": \"application/vnd.oci.image.index.v1+json\",\n \"size\": 314,\n \"digest\": \"sha256:05f95b26ed10668b7183c1e2da98610e91372fa9f510046d4ce5812addad86b5\"\n }\n]\n}\n"
},
{
"image": "busybox",
"name": "echo",
"resources": {},
"script": "cat $(inputs.resources.sourcerepo.path)/index.json"
}
]
},
"timeout": "1h0m0s"
},
"status": {
"completionTime": "2021-05-19T09:58:12Z",
"conditions": [
{
"lastTransitionTime": "2021-05-19T09:58:12Z",
"message": "All Steps have completed executing",
"reason": "Succeeded",
"status": "True",
"type": "Succeeded"
}
],
"podName": "build-push-run-output-image-fmjmx-pod-7f7br",
"resourcesResult": [
{
"key": "commit",
"resourceName": "sourcerepo",
"resourceRef": {
"name": "sourcerepo"
},
"value": "6ed7aad5e8a36052ee5f6079fc91368e362121f7"
},
{
"key": "url",
"resourceName": "sourcerepo",
"resourceRef": {
"name": "sourcerepo"
},
"value": "https://github.com/GoogleContainerTools/skaffold"
},
{
"key": "digest",
"resourceName": "builtImage",
"resourceRef": {
"name": "builtImage"
},
"value": "sha256:05f95b26ed10668b7183c1e2da98610e91372fa9f510046d4ce5812addad86b5"
},
{
"key": "url",
"resourceName": "builtImage",
"resourceRef": {
"name": "builtImage"
},
"value": "gcr.io/foo/bar"
}
],
"startTime": "2021-05-19T09:57:49Z",
"steps": [
{
"container": "step-create-dir-builtimage-fhhzh",
"imageID": "gcr.io/distroless/base@sha256:aa4fd987555ea10e1a4ec8765da8158b5ffdfef1e72da512c7ede509bc9966c4",
"name": "create-dir-builtimage-fhhzh",
"terminated": {
"containerID": "containerd://7bebd525bd9369eed146acc55179d856a1ee05652ad236945b8a8a2281b2b0b7",
"exitCode": 0,
"finishedAt": "2021-05-19T09:58:01Z",
"reason": "Completed",
"startedAt": "2021-05-19T09:58:01Z"
}
},
{
"container": "step-git-source-sourcerepo-7tvf6",
"imageID": "localhost:5000/mypipeline/git-init-4874978a9786b6625dd8b6ef2a21aa70@sha256:b873705b6716384afd08e485dffd112a77d038e06cda95e64d759273d11e1f7f",
"name": "git-source-sourcerepo-7tvf6",
"terminated": {
"containerID": "containerd://d1a09b1b3daebd019834bc22152848211f37a3833530da3e0944c1eb5b5fbea6",
"exitCode": 0,
"finishedAt": "2021-05-19T09:58:10Z",
"message": "[{\"key\":\"commit\",\"value\":\"6ed7aad5e8a36052ee5f6079fc91368e362121f7\",\"resourceName\":\"sourcerepo\",\"resourceRef\":{\"name\":\"sourcerepo\"}},{\"key\":\"url\",\"value\":\"https://github.com/GoogleContainerTools/skaffold\",\"resourceName\":\"sourcerepo\",\"resourceRef\":{\"name\":\"sourcerepo\"}}]",
"reason": "Completed",
"startedAt": "2021-05-19T09:58:01Z"
}
},
{
"container": "step-build-and-push",
"imageID": "docker.io/library/busybox@sha256:b5fc1d7b2e4ea86a06b0cf88de915a2c43a99a00b6b3c0af731e5f4c07ae8eff",
"name": "build-and-push",
"terminated": {
"containerID": "containerd://fbf8538daf6a215cfbabd1109a47935a971c03766608dbcebde250f95707de1a",
"exitCode": 0,
"finishedAt": "2021-05-19T09:58:11Z",
"reason": "Completed",
"startedAt": "2021-05-19T09:58:11Z"
}
},
{
"container": "step-echo",
"imageID": "docker.io/library/busybox@sha256:b5fc1d7b2e4ea86a06b0cf88de915a2c43a99a00b6b3c0af731e5f4c07ae8eff",
"name": "echo",
"terminated": {
"containerID": "containerd://5451255769fcb3973e6c1893f6f50d9064057859a0c1e397787a119175accf15",
"exitCode": 0,
"finishedAt": "2021-05-19T09:58:11Z",
"reason": "Completed",
"startedAt": "2021-05-19T09:58:11Z"
}
},
{
"container": "step-image-digest-exporter-h6hxl",
"imageID": "localhost:5000/mypipeline/imagedigestexporter-6e7c518e6125f31761ebe0b96cc63971@sha256:70b6715e478899bcc4a2c2c65db2fad26413fc1865bce368c48930cdc9c05eb9",
"name": "image-digest-exporter-h6hxl",
"terminated": {
"containerID": "containerd://b723d8ee2bc4c909668c00025eda98c7ca39d665a455c1ce38bccfa6b09fb993",
"exitCode": 0,
"finishedAt": "2021-05-19T09:58:12Z",
"message": "[{\"key\":\"digest\",\"value\":\"sha256:05f95b26ed10668b7183c1e2da98610e91372fa9f510046d4ce5812addad86b5\",\"resourceName\":\"builtImage\",\"resourceRef\":{\"name\":\"builtImage\"}},{\"key\":\"url\",\"value\":\"gcr.io/foo/bar\",\"resourceName\":\"builtImage\",\"resourceRef\":{\"name\":\"builtImage\"}}]",
"reason": "Completed",
"startedAt": "2021-05-19T09:58:12Z"
}
}
],
"taskSpec": {
"resources": {
"inputs": [
{
"name": "sourcerepo",
"type": "git"
}
],
"outputs": [
{
"name": "builtImage",
"targetPath": "/workspace/sourcerepo",
"type": "image"
}
]
},
"steps": [
{
"image": "busybox",
"name": "build-and-push",
"resources": {},
"script": "set -e\ncat <<EOF > $(inputs.resources.sourcerepo.path)/index.json\n{\n\"schemaVersion\": 2,\n\"manifests\": [\n {\n \"mediaType\": \"application/vnd.oci.image.index.v1+json\",\n \"size\": 314,\n \"digest\": \"sha256:05f95b26ed10668b7183c1e2da98610e91372fa9f510046d4ce5812addad86b5\"\n }\n]\n}\n"
},
{
"image": "busybox",
"name": "echo",
"resources": {},
"script": "cat $(inputs.resources.sourcerepo.path)/index.json"
}
]
}
}
}
If we're storing multiple payloads on multiple backends, we'll retry if any of the storage uploads failed. This means we may retry storing something that was successfully uploaded the first time, which some of our backends don't like.
We could either make all the operations safe to retry, or be more specific about what we retry.
we get this error:
{"level":"info","ts":"2021-08-12T15:12:00.776Z","logger":"watcher.event-broadcaster","caller":"record/event.go:282","msg":"Event(v1.ObjectReference{Kind:\"TaskRun\", Namespace:\"default\", Name:\"build-push-run-output-image-l4mjf\", UID:\"8628d8a4-2ca6-401f-a15a-c310d2cbd562\", APIVersion:\"tekton.dev/v1beta1\", ResourceVersion:\"91241155\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' 1 error occurred:\n\t* getting digest: digest must be between 71 and 71 runes in length: \n\n"}
Placeholder issue for in-toto integration tasks
@SantiagoTorres , hopefully you can see this.
Will need their own issues.
PR #22 will add a config store we can use to configure the chains system. Here's a rough sketch for what that config might look like.
Roughly in chains, we have a few subsystems:
Some storage systems and metadata formats may cross artifact types, but others won't. Here's my proposal for how to organize config, starting with things to watch and sign.
Things to sign will include taskruns, oci images, binaries, wheels, etc.
taskruns.formats.enabled-formats=in-toto,tekton,etc.
taskruns.signing.enabled-strategies=pgp,x509,etc.
taskruns.storage.backends=tekton,gcs,oci,etc.
We can then repeat this for the other artifact types:
oci.formats.enabled-formats=in-toto
oci.signing.enabled-strategies=pgp
oci.storage.backends=oci
to indicate we want to sign OCI images using PGP, with metadata in the in-toto format, and store that in an OCI registry.
Some of these may require their own custom sections as well:
storage.gcs.location=gs://foo/bar
would indicate where we want to store signatures in GCS.
For OCI, we could either store signatures next to the image we sign (notary v2 style), or we'd want to specify a repository to store signatures in (for other types):
storage.oci.repository=gcr.io/foo/bar
Deleting a namespace containing taskruns completes successfully.
I am seeing the chains finalization failing because the service account gets cleaned up, which results in a reconciliation error, and the TaskRun sits around forever (keeping the namespace forever).
I noticed this because I was running the pipelines e2e tests, which clean a lot of things up by deleting namespaces, and when I dug into why so much stuff was being left around, it looks a lot like this.
Perhaps the finalization path should log, but not return errors? 🤷
We generate package URLs for the intoto formatter:
Instead of creating the string ourselves, it would be nice if we could use the library to do it for us: https://github.com/package-url/packageurl-go
I briefly tried to add it in myself, but the library returns a percent-encoded string, and I'm not totally sure if that's what we're supposed to be using (right now we aren't percent-encoding strings)
We should probably clean up the configmap code a bit, the defaults are set in yaml which means they get overwritten every time you deploy.
We can probably do this better with actual defaults, and maybe even use reflection!
See here: in-toto/in-toto-golang#101
When generating ITE-6 payloads we can use this format (and the JSON wrapper) for signatures.
It might mean we have to change the interfaces around a bit, but now's the time!
This seems similar to tektoncd/plumbing#241, which the tektoncd/pipeline repo faced in early 2020.
Seems like the fix was adding more nodes to the prow cluster and upgrading golangci-lint to a version which used less memory (1.27).
Our current version already incorporates those changes (1.41) so we could look into adding more nodes to the prow cluster.
For now, I'll try increasing the timeout to 10m and see if that resolves things.
We probably want some docs around :
We should be able to copy cosign for oci repository replacement for storing image signatures
ref: https://github.com/sigstore/cosign/blob/main/cmd/cosign/cli/sign.go#L305
Right now chains supports "Tekton" as a storage backend, which means signatures and payloads are stored in Tekton objects via annotations. We would also like to store these objects in an OCI registry.
Notary v2 work is ongoing, which should make it possible to store signatures for container images in a native manner, but that is still a long way off from being supported widely. Additionally, this would be limited to signatures for artifacts also stored in a registry.
We have a few options:
I propose we start with our own storage system on OCI. At a high level it'll look like this:
cc @jonjohnsonjr is there anything silly I'm missing here?
Chains currently watches the outputs (resource or task) from TaskRuns, and signs these. Using a separate system to capture the final signed payload is an intentional decision to avoid the need to make cluster-level secrets accessible to the build process. This does require placing some trust in the Kubernetes control plane/data layer. Anyone with edit permission on pods in that namespace could forge or falsify taskrun outputs.
We could tighten this up a bit by adding another level of security, allowing Tasks to sign their own outputs, then validating those signatures before accepting the final payload.
This could look like:
All TaskRuns are signed.
TaskRuns that are quickly cleaned up might not get signed.
I see this sometimes when running tasks with mink run task
, which pretty aggressively cleans things up upon completion.
I think the fix is to have chains attach a finalizer to the TaskRun, which entails:
controller.Options
here:
. We want to customize the name in case other controllers start to finalize TaskRun
and to clearly identify that it is chains that hasn't processed the resource.FinalizeKind
, which can probably just call ReconcileKind
. You can see an example of this here: https://github.com/knative/serving/blob/c6d95c0cfcb27b6cb8c7cc82bf30e3de661a67f5/pkg/reconciler/labeler/labeler.go#L40Even with these two things, there is a chance that a TaskRun won't be signed, if the TaskRun completes and is removed before the chains controller can even process the resource a single time, but this seems fairly unlikely to happen in practice.
This way the controller can push signatures to a registry! I got this to work by creating a dockerregistry secret with credentials, and applying it to the tekton-chains-controller
service account as an imagePullSecret
$ kubectl create secret docker-registry registry-credentials \
--docker-server=gcr.io \
--docker-username=_json_key \
[email protected] \
--docker-password="$(cat credentials.json)"
$ kubectl patch serviceaccount tekton-chains-controller \
-p "{\"imagePullSecrets\": [{\"name\": \"registry-credentials\"}]}" -n tekton-chains
Right now signatures and payloads are stored in annotations on the TaskRuns:
chains.tekton.dev/payload
chains.tekton.dev/signature
But, as we start storing multiple types of payloads (one for the overall TaskRun, one for each image/binary produced, etc.), we need a way to store multiple. Some ideas:
Kubectl apply does something similar:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"chains","app.kubernetes.io/name":"tekton-pipelines","pipeline.tekton.dev/release":"devel","version":"devel"},"name":"tekton-chains-controller","namespace":"tekton-pipelines"},"spec":{"replicas":1,"selector":{"matchLabels":{"app":"tekton-chains-controller"}},"template":{"metadata":{"annotations":{"cluster-autoscaler.kubernetes.io/safe-to-evict":"false"},"labels":{"app":"tekton-chains-controller","app.kubernetes.io/component":"controller","app.kubernetes.io/name":"tekton-pipelines","version":"devel"}},"spec":{"containers":[{"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"METRICS_DOMAIN","value":"tekton.dev/chains"}],"image":"gcr.io/dlorenc-vmtest2/controller-92006fd957c0afd31de6a40b3e33b39f@sha256:0924a9571260ac4892f2c9a52bfd1c1775683cddd7cf1725ca8ec816bdb32f40","name":"tekton-chains-controller","volumeMounts":[{"mountPath":"/etc/config-logging","name":"config-logging"},{"mountPath":"/etc/signing-secrets","name":"signing-secrets"}]}],"serviceAccountName":"tekton-pipelines-controller","volumes":[{"configMap":{"name":"config-logging"},"name":"config-logging"},{"name":"signing-secrets","secret":{"secretName":"signing-secrets"}}]}}}}
This could look like:
chains.tekton.dev/payloads: |
{
"artifacts": [
{
"uri": "tekton://$namespace/$name/$uid",
"payload": "$payload",
},
{
"uri": "oci://gcr.io/foo/bar",
"payload": "$payload",
},
]
Problems: The modification operation here is "read-modify-write", so we risk losing writes or races.
Instead of one with all payloads, each artifact would get its own. Something like:
chains.tekton.dev/taskrun-payload
chains.tekton.dev/image1-payload
chains.tekton.dev/binary1-payload
The logic to retrieve all and validate them all is a little tougher - we need to loop through all annotations matching a pattern.
But, the write operation is merge-safe.
I'm leaning toward #2.
Right now, we're skipping these tests because we can't access GCR. Some options we have to fix this:
Right now we only sign TaskRuns and OCI Images, but it would be cool to give users a way to sign other types of results (binaries, files, whatever)
This would probably be a type of x509 signing, which supports the COSIGN ENCRYPTED PRIVATE KEY type
Also maybe run it in a different namespace from the pipelines controller. Right now we piggy-back off both.
This should be done to limit ACLs on any signing keys better.
main...solarwinds:ite6 is a draft branch.
The ITE-6 code is here: https://github.com/in-toto/in-toto-golang/tree/ite-6
Right now, we will continuously retry if some part of the reconcile loops fails. We could retry up to 3 times, and track the number of retries via annotations on the TaskRun:
annotations:
chains.tekton.dev/retries: 2
If all three retries fail, we can set the signed annotation on the TaskRun to “failed”:
annotations:
chains.tekton.dev/signed: failed
If we set it to "failed", we should leave the "retry" annotation so that it's clear all three retries occurred and failed.
We should come up with some tests for all the tasks in the catalog, or the pipelines examples: https://github.com/tektoncd/pipeline/tree/main/examples
We've started storing attestations for images in cosign using a ".att" suffix. We should figure out how to do that here in chains (right now we don't support storing attestations in OCI). I think the biggest challenge will be that the storage backends don't know what they're storing (signatures vs. attestations), and we hardcoded the ".sig" suffix.
cc @priyawadhwa any ideas?
we probably need to:
maybe more?
I'm talking about this code in particular:
chains/pkg/chains/storage/oci/oci.go
Lines 50 to 51 in 288242f
There are two things about this that are bugging me:
NewInCluster
should be avoided, and this should probably be calling New
with a kube client passed through (this will also make testing easier because you can pass a fake client though)."tekton-chains-controller"
) feels wrong to me for a variety of reasons (what if I want to reconfigure things to run differently? Why is this using the controller's SA vs. the task's SA?)Figured I'd open an issue for discussion rather than just slinging a PR for this one.
We need to update the release Pipeline to store the Git commit and url as results, so that Chains picks it up and includes it in the final provenance.
Right now, the git repo/commit isn't in there
$ rekor-cli get --uuid 3873a54462deab6320d1cac993b31b36bb28ff5c2f0d16993909b61907235ec6 --format json | jq -r .Attestation | base64 --decode | jq
{
"_type": "publish-chains-release",
"predicateType": "https://tekton.dev/chains/provenance",
"subject": [
{
"name": "gcr.io/tekton-releases/github.com/tektoncd/chains/cmd/controller",
"digest": {
"sha256": "1189a2207be3e93e91aaeff323dc2804576f188527afe3cc2e9a9a0c688344df"
}
}
],
"predicate": {
"invocation": {
"parameters": [
"package={string github.com/tektoncd/chains []}",
"versionTag={string v0.3.0 []}",
"imageRegistry={string gcr.io []}",
"imageRegistryPath={string tekton-releases []}",
"releaseAsLatest={string true []}",
"platforms={string linux/amd64,linux/arm64 []}",
"serviceAccountPath={string release.json []}",
"package=github.com/tektoncd/chains",
"images=controller",
"imageRegistry=gcr.io",
"imageRegistryRegions=us eu asia",
"releaseAsLatest=true",
"platforms=linux/amd64,linux/arm64,linux/s390x,linux/ppc64le"
],
"recipe_uri": "task://publish-chains-release",
"event_id": "c26f0ca6-1a00-4313-ac6b-ee098ad859da",
"builder.id": "tekton-chains"
},
"recipe": {
"steps": [
{
"entryPoint": "#!/busybox/sh\nset -ex\n\n# Login to the container registry\nDOCKER_CONFIG=$(cat ${CONTAINER_REGISTY_CREDENTIALS} | \\\n crane auth login -u _json_key --password-stdin $(params.imageRegistry) 2>&1 | \\\n sed 's,^.*logged in via \\(.*\\)$,\\1,g')\n\n# Auth with account credentials for all regions.\nfor region in ${REGIONS}\ndo\n HOSTNAME=${region}.$(params.imageRegistry)\n cat ${CONTAINER_REGISTY_CREDENTIALS} | crane auth login -u _json_key --password-stdin ${HOSTNAME}\ndone\ncp ${DOCKER_CONFIG} /workspace/docker-config.json\n",
"arguments": null,
"environment": {
"container": "container-registy-auth",
"image": "docker-pullable://gcr.io/go-containerregistry/crane@sha256:3095b8be43318d89e593a7d067430b79070cd9da32b4c9484438cef117a31a98"
},
"annotations": null
},
{
"entryPoint": "#!/usr/bin/env sh\nset -ex\n\n# Setup docker-auth\nDOCKER_CONFIG=~/.docker\nmkdir -p ${DOCKER_CONFIG}\ncp /workspace/docker-config.json ${DOCKER_CONFIG}/config.json\n\n# Change to directory with our .ko.yaml\ncd ${PROJECT_ROOT}\n\n# For each cmd/* directory, include a full gzipped tar of all source in\n# vendor/. This is overkill. Some deps' licenses require the source to be\n# included in the container image when they're used as a dependency.\n# Rather than trying to determine which deps have this requirement (an(params.imageRegistryd\n# probably get it wrong), we'll just targz up the whole vendor tree and\n# include it. As of 9/20/2019, this amounts to about 11MB of additional\n# data in each image.\nTMPDIR=$(mktemp -d)\ntar cfz ${TMPDIR}/source.tar.gz vendor/\nfor d in cmd/*; do\n if [ -d ${d}/kodata/ ]; then\n ln -s ${TMPDIR}/source.tar.gz ${d}/kodata/\n fi\ndone\n\n# Rewrite \"devel\" to params.versionTag\nsed -i -e 's/\\(chains.tekton.dev\\/release\\): \"devel\"/\\1: \"$(params.versionTag)\"/g' -e 's/\\(app.kubernetes.io\\/version\\): \"devel\"/\\1: \"$(params.versionTag)\"/g' -e 's/\\(version\\): \"devel\"/\\1: \"$(params.versionTag)\"/g' ${PROJECT_ROOT}/config/*.yaml\n\n# Publish images and create release.yaml\nmkdir -p $OUTPUT_RELEASE_DIR\n\nko resolve --platform=$(params.platforms) --preserve-import-paths -t $(params.versionTag) -f ${PROJECT_ROOT}/config/ > $OUTPUT_RELEASE_DIR/release.yaml\n\n# Publish images and create release.notags.yaml\n# This is useful if your container runtime doesn't support the `image-reference:tag@digest` notation\n# This is currently the case for `cri-o` (and most likely others)\nko resolve --platform=$(params.platforms) --preserve-import-paths -t $(params.versionTag) -f ${PROJECT_ROOT}/config/ > $OUTPUT_RELEASE_DIR/release.notags.yaml\n",
"arguments": null,
"environment": {
"container": "run-ko",
"image": "docker-pullable://gcr.io/tekton-releases/dogfooding/ko@sha256:ff918ec2c8bbe416d5a9b6f9d25dfe9012dce673922fe7b2d5d69a99b02df0ac"
},
"annotations": null
},
{
"entryPoint": "set -ex\n\nIMAGES_PATH=${CONTAINER_REGISTRY}/$(params.package)\n\nfor cmd in $(params.images)\ndo\n IMAGES=\"${IMAGES} ${IMAGES_PATH}/cmd/${cmd}:$(params.versionTag)\"\ndone\n\n# Parse the built images from the release.yaml generated by ko\nkoparse \\\n --path $OUTPUT_RELEASE_DIR/release.yaml \\\n --base ${IMAGES_PATH} --images ${IMAGES} > /workspace/built_images\n",
"arguments": null,
"environment": {
"container": "koparse",
"image": "docker-pullable://gcr.io/tekton-releases/dogfooding/koparse@sha256:5945f709f5533347e2fac2f7e757a2acde2ce25418a7193489bf49027aa0497f"
},
"annotations": null
},
{
"entryPoint": "#!/busybox/sh\nset -ex\n\n# Setup docker-auth\nDOCKER_CONFIG=~/.docker\nmkdir -p ${DOCKER_CONFIG}\ncp /workspace/docker-config.json ${DOCKER_CONFIG}/config.json\n\nREGIONS=\"us eu asia\"\n\n# Tag the images and put them in all the regions\nfor IMAGE in $(cat /workspace/built_images)\ndo\n IMAGE_WITHOUT_SHA=${IMAGE%%@*}\n IMAGE_WITHOUT_SHA_AND_TAG=${IMAGE_WITHOUT_SHA%%:*}\n IMAGE_WITH_SHA=${IMAGE_WITHOUT_SHA_AND_TAG}@${IMAGE##*@}\n\n echo $IMAGE_WITH_SHA, >> $(results.IMAGES.path)\n\n if [[ \"$(params.releaseAsLatest)\" == \"true\" ]]\n then\n crane cp ${IMAGE_WITH_SHA} ${IMAGE_WITHOUT_SHA_AND_TAG}:latest\n fi\n\n for REGION in ${REGIONS}\n do\n if [[ \"$(params.releaseAsLatest)\" == \"true\" ]]\n then\n for TAG in \"latest\" $(params.versionTag)\n do\n crane cp ${IMAGE_WITH_SHA} ${REGION}.${IMAGE_WITHOUT_SHA_AND_TAG}:$TAG\n done\n else\n TAG=\"$(params.versionTag)\"\n crane cp ${IMAGE_WITH_SHA} ${REGION}.${IMAGE_WITHOUT_SHA_AND_TAG}:$TAG\n echo ${REGION}.$IMAGE_WITH_SHA, >> $(results.IMAGES.path)\n fi\n done\ndone\n",
"arguments": null,
"environment": {
"container": "tag-images",
"image": "docker-pullable://gcr.io/go-containerregistry/crane@sha256:3095b8be43318d89e593a7d067430b79070cd9da32b4c9484438cef117a31a98"
},
"annotations": null
}
]
},
"metadata": {
"buildStartedOn": "2021-07-28T15:23:18Z",
"buildFinishedOn": "2021-07-28T15:29:12Z",
"reproducible": false
}
}
}
We should consider making chains configurable so it can use a different set of criteria for making decisions based on some characteristics of the TaskRun.
A concrete example here is the Tekton CI set-up. We store everything right now in rekor, but we probably only care about release builds.
Right now, we parse through TaskRun annotations to set this field. However, annotations aren't copied over into the TaskRun spec from the Task itself, so this often doesn't work.
We'll need a different way for users to let Chains know if a build is reproducible.
I'm imagining a Tekton Cat but with the chains necklace on, hanging out with the Falco logo. The cat will be holding some kind of shield with the SPIFFE logo on it, and we can throw in some sigstore/cosign logos too when we get them.
Ref cncf/tag-security#625 (comment)
But anyway - the real is to try to put together an end to end hardened build system using Falco policies to detect runtime issues. Use something like SPIRE to tie builds to specific machines, and some kind of TPM attestations for integrity of the build system itself and the individual build instances.
In-toto, sigstore and hermekton can provide a secure end to end provenance of build artifacts, which could include these build system and build node attestations all the way back to a hardware root of trust.
This will probably be something similar to what triggers does, described here: https://github.com/tektoncd/triggers/blob/main/tekton/release-cheat-sheet.md
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.