Automated message signing and packet encryption key rotation about prio-server

[tgeoghegan]

Further, we should run the key generation and rotation tasks in the Kubernetes cluster as a cronjob, so that secrets never need to be exposed to our operator computers. This will involve building a Docker image for the key rotator and putting a kubectl run invocation in a makefile, I think.

[tgeoghegan]

Retitled; this is about validating and finishing our implementation of key rotation (deploy-operator, manifest-updater, etc.), and then deploying it in all the prio-server deployments (ISRG/prod-us, ISRG/prod-intl, NCI/prod-us, Google/prod-intl).

[tgeoghegan]

When we do this, we should also consider automating key backups (relatedly, see #935). For operational smoothness, it'd be nice if key backups were automated. On the other hand, if we empower some Kubernetes job to do this, then we have to be careful to configure ACLs on the backups so that they cannot be deleted or tainted with by a bug somewhere in prio-server. Maybe the versioning features offered by the each cloud platform's secrets storage products suffices.

[branlwyd]

Some thoughts on validations that the key rotation infrastructure will need to implement:

• Keys must either have no versions, or exactly one primary version.
• Post-rotation, keys must have at least one version.
• When being used to update keys in a manifest, keys must have at least one version.
• When being used to update keys in a manifest, keys must have versions with distinct creation times.
• When being used to update keys in a manifest, existing key versions' public portions should match with the public keys written to the manifest, matching by creation time/key ID (both of which would provide the same matching).
• Post-update, manifests must have exactly one packet encryption key version.
• Post-update, manifests must have at least one batch signing key version.
• Post-update, manifests' non-key data must match the pre-update manifest data exactly.
• Post-update, manifests' key data for key versions that exist both pre- & post-update must match exactly.

[branlwyd]

From discussion in #1114:

Specifically, if a packet encryption key got so far ahead of its related manifests that the key did not include the version advertised as primary in the manifest, we would be asking clients to encrypt with a key we no longer consider valid for decryption. And if a batch signing key got so far ahead of its related manifests that the primary key was not included in the manifest, we would be trying to sign data with a key that we do not recognize as valid for verification.

Both of these conditions can be detected by the key-rotator. We should include in our validations not writing keys that, if the key-rotator fails to write updated manifests, would leave the system in a non-working configuration.