jlevesy / kudo

sudo for Kubernetes

kudo: sudo for Kubernetes

What is Kudo

Kudo is a Kubernetes controller that allows individual users to temporarily escalate their permissions while still maintaining security and crystal clear auditability.

It comes in complement of existing access control systems (Kubernetes RBAC, GCP IAM), and relies on them to temporarily grant or reclaim permissions. In the context of Kubernetes, Kudo temporarily creates a RoleBinding or a ClusterRoleBinding between an existing role and the escalation requestor.

To gain a better understanding of what Kudo is, you can refer to the Kudo website

Project Status

This is a toy project at the moment, please do not try to use it in that state as It can be dangerous from a security standpoint and I don't provide support in any case.

Though, if you like the idea, please let me know!

Contributing To Kudo

Roadmap

The kudo project pretty much carries all what I have in mind for kudo. Feel free to pick a task in the TODO column.

Development Environment

Tooling

Here's a list of tools you need to have installed to run your development environment.

• go1.19
• k3d
• ko
• helm
• kubectl
• OpenSSL
• (For the documentation) hugo

Configuration

You need to have both kudo-registry.localhost and kudo-e2e-registry.localhost configured to resolve to 127.0.0.1 in your development environment.

Using the Development Environment

The following command line spins up a k3d cluster, provision necessary resources and install your current tree version of kudo in the cluster.

make run_dev

To simulate an escalation. This will switch your current kubectl context to the kudo test user, create the permission, then switch back to the admin context.

make escalate_dev

To display the controller logs

make logs_dev

When you're done:

make stop_dev

And if you're in need

make help

Running the Test Suite

• make unit_tests runs the unit test suite, ie all the tests defined in package.
• make e2e_tests runs the end to end test suite, which simulate real kudo use cases. The test suite lives under the ./e2e directory.

Serving The Documentation

• make serve_docs starts a local webserver. You can then go to http://localhost:1313/kudo to check your local doc.

kudo's Issues

Support Events

What needs to be done?

For observability, we need to make kudo emit events for major events regarding permissions.
For example State transtitions for escalations. Kudo should log an event explaining why the transition happened. Depending on what we can achieve here, we could remove the stateDetails field.

Acceptance Criteria

• Kudo logs an event on the escalation each time it changes state.

Refactor Plugin to allow multiple entrypoints in a single binary

What needs to be done?

It looks like the kudo plugin will receive multiple sub commands

For example:

• kudo escalate already exists
• kudo list-policies list existing policies available for escalation (alias to get policies)?
• kudo list-pending-approvals list the pending escalations that potentially requires your approval
• kudo approve approve an ongoing escalation

The problem we have at the moment, is that we have one binary per subcomand, which is super sub optimal. It would be great to only distribute (and release!) a single binary

I wonder if it would be possible to only have one binary and then rely on the binary name to actually.

Looks like krew does it here?

Acceptance Criteria

• we Have a way of having one binary for all the potential kudo subcommands.

Proper Config File

What needs to be done?

Kudo at the moment hardcodes stuff does not offer any opportunity to configure its internal behavior.

It would be nice to introduce a config map, instead of flags. For instance to allow the following properties to be configured

• resyncInterval
• retryInterval
• threadiness
• audit Sinks

resyncInterval: 30s
retryInterval: 10s
threadiness: 10
auditSinks:
   - kind: K8sEvents
     namespace: ""

Documentation should be updated to reflect that changes, and the chart should accept configuring this file based on the values file

Acceptance Criteria

• Kudo accepts a config file
• Helm chart allows to configure this file
• Documentation has been updated to reflect this change

Allow requestor to Indicate target namespace

What needs to be done?

At the moment, our current design for the K8sRoleBinding grant kind is to have the target namespace specified in the EscalationPolicy directly.

While this can be applicable in some cases, it can lead to the duplication of the same policy (for example let's consider a policy that allows to gain the port-forward role in differente namespace, we would have N policies for N namespace port-forward-namespace-x and port-forward-namespace-y which are probably the same policies, but applied in different namespace).

That is not great, and we want to change this by allowing the user to hint the destinationNamespace when asking for a grant

kubectl kudo escalate port-forward "need access" --namespace=namespace-z

Doing this, the previous case could be solved with a single policy, which is way better. But this does not come from free:

• The webhook needs to validate the incoming escalation against all the grants defined in the policy, if the request doesn't have a namespace, then the webhook should reject the request
• We should think of a way to restrain policies, otherwise everyone will be able to use a policy anywhere. So by default the request is denied unless the namespace is listed in the grant namespaceAllowList, a wildcard might be supported here?

Acceptance Criteria

• Requestors are now able to pick a namespace
• Escalations are denied if if does not contain sufficient information
• Escalations are denied if the target namespace is not in the allow list
• e2e tests
• Doc

End to End test suite

What do we need to do?

The goal of this task is to provide an end to end test suite in complement of the unit test suite for Kudo. Unit test suite don't allow us to properly test the combination of admission webhooks + controller.

What I have in mind for this is a go package ./integration that carries the test suite for Kudo. Then one standard go test per use case

This implies a bunch of tooling, that needs to be done on a pre test suite and pre test hooks

• Boot a local image registry container in the local docker daemon
• Build and push the current controller image to this local registry, ideally by embedding code from https://github.com/google/ko
• Boot a k3d cluster from the test suite as well, and trust the previously created local registry.
• Install Kudo on the newly created cluster using the helm chart in the provisioned library. Helm v3 seems to be usable as a go-sdk
• Provide synchronization primitives so that we start the test when kudo is ready

Acceptance criteria

• We have an integration test suite which test the main supported use case at the moment
• Writing an integration test is easy, and documented
• Test are "fast enough"
• Test suite can be run having only docker and go on the local machine
• Test suite is run on the CI worklow

Configurable expiration delay

What needs to be done?

Similar to #26 with namespaces, we would like to allow the user to tell how much time they need the escalation. This implies to change the target.duration field of the EscalationPolicy to target.maxDuration (to control the maximum duration) and target.defaultDuration to provide a default value.

Acceptance Criteria

• kubectl kudo escalate accepts an optional --duration argument, that controls for how much time the escalation is granted.

Harden EscalationPolicy / Escalation link

What needs to be done?

At the moment the Escalation references an EscalationPolicy by its name. Which means that if the EscalationPolicy content is changed for instance to add grants, those new grants are going to be applied to the escalation in the next sync. Which can lead to perm escalation if one gains the right to edit policies, it also leaks existing bindings.

How to fix this?

I would go to store the policy resourceVersion in the escalation status and check that the policy resource version is the same than the one stored. If we spot a difference, then we deny the escalation.

Acceptance Criteria

• An end to end test proves that an escalation policy is changed, then all the referring active escalations are denied

Harden CRDs definition with OpenAPI schema validation

What needs to be done ?

Our current CRDs don't leverage properly the k8s schema validation.

We should review our model and apply as much as data-validation as possible.

Acceptance Criteria

• CRD Schema validation is used correctly

Highly available controller

What needs to be done?

The kudo controller app runs currently both the controller loops as wll as the webhook handler, and this constitutes a spof for webhook admission if we only have one replica.

While I not super keen to split this into two different deployment (but that's up for debate), it would be interesting to leverage k8s leader election capabilities so we could have multiple replicas of kudo running, but only one executing the control loop.

Acceptance Criteria

• You can scale the kudo deployment to as many replica as you wish, yet only one runs the control loop
• Helm chart deploys 2 replias by default
• e2e test suites are adapted

Allow to grant a k8s role binding on a service account

What needs to be done?

A cool feature would be to define a potential grantee in an escalation grant directly, for example in the K8sRoleBinding grant we could write something like this

# some-policy.yaml
target:
   grants:
    - kind: KubernetesRoleBinding
      defaultNamespace: some-app
      allowedNamespaces:
        - some-app
        - some-other-app
      grantee:  # <- this !
        kind: ServiceAccount
        name: super-app
      roleRef:
        kind: ClusterRole
        name: config-viewer
        apiGroup: rbac.authorization.k8s.io

In that case, kudo will create the role binding not for the requestor, but for the grantee defined by the policy in the namespace picked by the policy.

This could open up a bunch of cool use cases like granting temporarily more permissions to a workload.

Kudos to @dduportal for the idead 🎉

Acceptance Criteria

• Kudo allows granting permissions to a third party grantee defined in policy
• an e2e test has been written to prove this

Suport provisioning cluster role bindings

What needs to be done?

At the moment, kudo only supports provisioning RoleBindings (via the k8sRoleBindingGranter)

The goal of this tasks is to implement a k8sClusterRoleBindingGranter which works in a similar fashion but provisions ClusterRoleBindings.

Acceptance Criteria

• Kudo can understand the K8sClusterRoleBindings grant kind
• An e2e test has been written to prove this.

Prevent escalation updates or deletes

What needs to be done?

Currently any user can update any property of an already existing escalation, this might be problematic and lead to abusing escalation. What we want here, is to forbid this.

How to fix this?

Enable our AdmissionMutationWebhook for operations other than CREATE, and deny them if the user that triggered it is not kudo service account.

Acceptance Criteria

• As an user, I am not allowed to update any escalations
• An e2e test proves this

Improve `kubectl get` output for our CRDs

What needs to be done?

We need to improve output of kubectl get escalations and kubectl get escalationpolicies to expiose relevant information.

• For escalation: status.State status.StateDetails and maybe a counter showing how many grants are Created comapted to the total grants.
• For escalation policies, eeeh. This needs to be given a bit of thinking. Maybe how may escalalations are active?

This could be done using additional printer colums in resources CRD.

• kubectl get escalations shows meaningful output
• kubectl get escalationpolicies shows meaningful output

Fix dev env to use a local registry

What needs to be done?

Our current Makefile set up for the dev env (make run_dev) works as the following:

• Spins up a k3d cluster
• Calls helm template | ko apply -f - which builds and pushes an image to ghcr.io/jlevesy/kudo/controller and deploy the templated chart.

While it is somehow efficient, it has two major drawbacks:

• We need to be connected to the interwebs to run the dev_env
• And maybe the most important one: I only can use it, because I have push permissions on jlevesy/kudo 🙃

My goal here is using a local registry instead.

An other alternative I already explored is using k3d images push buuut this doesn't work on Apple silicon (containerd breaks somehow.)

Acceptance criteria

• Anyone can run make run_dev with the right tools installed

Set up documentation website

What needs to be done?

We need a documentation website that should carry minimal information on how to use kudo.

• Ideally, it would live in-tree (no documentation website repo)
• I would favor using hugo and docsy
• It'll be hosted on githubpages, under jlevesy.github.io/kudo
• Deployed automatically at each merge on main

At this stage, kudo is highly unstable so I wouldn't dive into too much details at the moment. But at least:

• an intro on what kudo actually does
• a getting stated guide, (even if installation is left to define at the moment) with a quick example demonstrating the use case.
• its main concepts (escalation, policies, challenges, audit sinks),

Acceptance Criteria

• jlevesy.github.io/kudo serves a basic documentation website
• This website is deployed automatically on each merge on main

Snappy Reclaims

What needs to be done?

At the moment, we're relying on the default resync interval to update escalations states. As it is quite expensive to trigger a full resync, one clever improvement would be to reque an object when in is supposed to have expired to triger a resync.

Acceptance Criteria

• Reclaims are done closer from expiration

Refactor `EscalationGrantRef` to easilly handle more grant kinds

What needs to be done?

At the moment the EscalationGrantRef is a simple struct that stores information referencing a resource managed by kudo.

The problem is that it'll grow as we add support for new escalation grants and will end up if a gigantic bag of data.

So our goal is, instead of having this:

type EscalationGrantRef struct {
	Kind            string      `json:"kind"`
	Name            string      `json:"name"`
	Namespace       string      `json:"namespace"`
	UID             types.UID   `json:"uid"`
	ResourceVersion string      `json:"resourceVersion"`
	Status          GrantStatus `json:"status"`
}

We'll go to this and introduce one ref type per grant kind.

type EscalationGrantRef struct {
	Kind            string      `json:"kind"`
	Status          GrantStatus `json:"status"`

       K8sRoleBinding *K8sRoleBindingGrantRef `json:"k8sRoleBinding"`
}

type K8sRoleBindingGrantRef struct {
	Name            string      `json:"name"`
	Namespace       string      `json:"namespace"`
	UID             types.UID   `json:"uid"`
	ResourceVersion string      `json:"resourceVersion"`
}

Acceptance Criteria

• CRDs are updated
• Escalation grant refs have now one specific structure per escalation grant

Add kubectl plugin

What needs to be done?

The goal of this task is to develop a kubectl plugin that allows an user to trigger an escalation in a one liner.

For example kubectl kudo escalate --policy=policy-name creates an escalation in the current cluster. That would be a first step.

Acceptance Criteria

• Plugin has been implemented
• Plugin is developed in a way that allows testing them in the e2e test suite

Set up issues / PR template

What needs to be done?

As a good open source project, we should probably have issues templates for:

• PRs
• Feature request
• Bugs

As well as some automation for stales issues and PRs

Acceptance criteria

• We have a decent issue pipline