kubernetes-retired / multi-tenancy

A working place for multi-tenancy related proposals and prototypes.

Kubernetes Working Group for Multi-Tenancy

This is a working place for multi-tenancy related proposals and prototypes. To join our biweekly meetings, Slack, mailing list, please visit our community page.

Projects

The multi-tenancy working group is in charge of the following projects:

• Benchmarks: a set of benchmarks (i.e., compliance tests) to determine if your clusters are well-configured for multitenancy.
• Hierararchical namespaces (aka HNC): allows namespaces to own each other, policy propagation between related namespaces, and delegated namespace creation.
  • HNC was previously housed in this repo but graduated in May 2021.
• Virtual clusters: run multiple virtualized cluster on a single underlying cluster, allowing for hard(er) multitenancy.
  • VirtualCluster was previously housed in this repo but graduated in May 2021.

Past projects also include:

• [DEPRECATED] Tenant Operator: an opinionated solution to manage tenants within a cluster.
  • The Tenant Operator has been replaced by HNC and VC and is no longer being actively developed.

Resources

The docs directory contains any documents written in markdown. Some draft docs which need collaboration are Google docs for better collaboration experience. The links file contains links to all presentations, wg-multitenancy minutes, and other docs not directly related to the projects above.

Join this repo

File a request at https://github.com/kubernetes/org to be added to @kubernetes-sigs, using the Template.

Once you've been a member, when you are ready to become a reviewer of other people's code, file a PR on our OWNERS file and an approver will need to approve you.

Once you've been a reviewer, you can request to become an approver by filling a PR on our OWNERS file and another approver will need to approve you.

[Deprecated] PoC directory

The poc directory contains deprecated proof-of-concept code which is not supported.

Code of conduct

Participation in the Kubernetes community is governed by the Kubernetes Code of Conduct.

multi-tenancy's Issues

Final review of multi-tenancy proposal

Draft a proposal of the work the multi-tenancy working group is going to tackle, and present it to the working group for feedback and approval.

https://docs.google.com/document/d/1U8RQQmTUjxgMZY05HG2f7b3KsB94BhK4Ko6aWbLNXcc/edit#heading=h.usecunpmfu3a

HNC: ensure hierarchy doesn't violate required children

Once #90 is complete, we should add a validating webhook to ensure that required children are not reassigned to the wrong parents.

/good-first-issue

Populate namespace with policies defined in NamespaceTemplate

Create policy objects defined in NamespaceTemplate into namespaces created after Tenant CR.

Review soft-multitenancy profile

https://github.com/kubernetes-sigs/multi-tenancy/blob/master/docs/profiles/profile-soft-multitenancy-s1.md

Please read, review, and update.

HNC: Refactor hierarchy controller into smaller methods

The hierarchy controller has the following rough structure:

• reconcile loads the singleton and namespace from the apiserver, determines if updates should be written back to the apiserver, calls updateTree, and calls updateObjects if we're writing updates.
• updateTree makes a copy of the singleton and namespace, calls syncWithForest, and (if we're updating) writes the objects back to the apiserver.
• syncWithForest is very long, but roughly does the following things, mostly inline:
  • Locks the forest
  • If the namespace is being reconciled for the first time, enqueues all relatives
  • If the namespace is a required child, ensure the parent is set correctly
  • Ensures the parent exists
  • If the parent has changed, ensures the new one is legal and enqueues the old/new parent
  • Determine if any required children are missing and handle them
  • Set the conditions

There are two problems with this. Firstly, syncWithForest is far too long at about 80 lines and growing continuously. Fortunately, once #76 is done, there will be no local variables that must be maintained throughout the function. Therefore, it will be easy to break up. I recommend that no one function be over 30 lines if possible; it should look roughly like the sublist of tasks shown above.

Secondly, the boundary between reconcile and updateTree is very fuzzy - the former reads the objects, but the latter writes them? What gives? We should probably merge the two of them together, and factor out anything that would push the combined function over 30 lines. For example, we could load both objects in one function, copy them both in another, and write them back in a third; the logic in reconcile to determine whether we should write back to the apiserver can probably also be factored out. Therefore, the revamped reconcile should probably look something like:

• load the singleton and namespace
• copy the singleton and namespace
• call syncWithForest
• early-exit if we shouldn't write back to the apiserver because something's being deleted
• write the singleton and namespace back to the apiserver
• call updateObjects

This should result in a far easier-to-understand and easier-to-extend flow.

HNC: in-memory tree may be out-of-sync with version on apiserver

See #58 (review)

Auto-managed RBAC for Tenant CR

When a Tenant CR is created, the Tenant Admins listed in the CR will be granted get, update, patch, delete permissions.

Baseline security profile today in a soft multitenant cluster

Baseline security profile today in a soft multitenant cluster

Using the current release of Kubernetes, assuming a Kubernetes cluster is shared between multiple teams (i.e. soft multitenancy), document how to configure a cluster for soft multitenancy.

• N tenants per cluster, 1 Kubernetes namespace per tenant, no changes to Kubernetes upstream other than configuration
• See @easeway's KEP for some good concrete policies towards the end of the doc https://docs.google.com/document/d/1lFiRNDWgyoZWQfvQV2UYsH80CKW90jwSeBzWyssTf7g/edit#heading=h.xgjl2srtytjt

Clarification: Regarding TenantTemplate CR in the documentation

In the document Kubernetes Tenant CRD

Advanced (Full) Version Workflow:

• A cluster-admin creates and defines one or more TenantTemplate-CR and NamespaceTemplate-CR objects in a cluster.
• A TenantTemplate-CR instance may itself refer to one or more NamespaceTemplate-CR objects.
• A tenant-instance-admin will be able to view the available list of TenantTemplate-CR instances that are made readable to him/ her by the cluster-admin.
• A tenant-instance-admin can then synchronously create one or more Tenant-CR objects using one of the TenantTemplate-CR objects available to him. This is referred to as the synchronous model of tenant creation.

Is there any example for the tenantTemplate-CR ? How the cloud admin use it to create one or more tenant-CR ?

clarify incubator directory intent

The readme calls out the poc folder as containing explicitly unsupported code. What is the disposition of incubator directory? Nesting code inside that folder seems like it will be problematic/disruptive to unnest if those projects persist/grow. Should we move those to the poc folder or move them to their own repos?

Give security researchers a secured multitenant cluster to work against

Give security researchers a reference cluster to work against.

• We will use the results of #5 to give security researchers a reference cluster they can work against. No one working on the Kubernetes’ bug bounty had time to work on that configuration, so it would benefit the project if the work coming out of this group could be used and formally made part of the Kubernetes bug bounty project.
• Having a concrete reference architecture that a researcher could bring up and test against a defined threat model would make their job and our job of assessing the vulnerability easier.
• The m10 wg can then use their findings to improve our own work

Baseline security profile today in a single tenant cluster

Baseline security profile today in a single tenant cluster

Using the current release of Kubernetes (1.13), assuming an entire Kubernetes cluster is used by a single team (i.e. no multitenancy), document the baseline recommended security profile to run a “secure” single tenant cluster.

• 1 tenant per cluster, 1 Kubernetes cluster per tenant, no changes to Kubernetes upstream other than configuration

HNC: Add proper conditions to Hierarchy status

Currently, the conditions in the HNC status are simply a set of strings. In addition, conditions don't propagate between namespaces. This is a proposal to make them more structured, machine-readable, and useful.

Error codes and references

We should expand the Condition struct to include the following new fields:

• code: a machine-readable string value (values shown below) summarizing the condition.
• affects: a list of group-version-kind-namespace-name that uniquely identifies the object(s) affected by the condition.

Initial list of conditions.

Some condition codes will start with the prefix CRIT_ which indicates that this is a critical error; until the error is resolved, HNC will no longer make any changes in this namespace (or its descendants).

I would recommend the following initial set of codes (clients should expect that the list can gro over time):

• CRIT_PARENT_MISSING: The specified parent is missing.
  • affected and msg are empty.
• CRIT_PARENT_INVALID: The specified parent is invalid (ie would cause a cycle).
  • affected is empty and msg describes the cycle.
• CRIT_ANCESTOR: A critical error exists in an ancestor namespace, so this namespace is no longer being updated.
  • affected[0].name identifies the most recent ancestor with a critical condition. msg is empty.
• OBJECT_OVERRIDDEN: An object in this namespace has been overridden from its parent and will no longer be updated.
  • affected[x].{group,version,kind,namespace,name} is set for every such object. msg may summarize the errors in a human-readable form.
• OBJECT_DESCENDANT_OVERRIDDEN: An object in this namespace is no longer being propagated because a propagated copy has been modified.
  • affected[x].{group,version,kind,namespace,name} identifies the modified copy, and msg may summarize the errors.
  • Note that OBJECT_OVERRIDDEN will be set for the same object in its own namespace
  • This condition should be set in the namespace that contains the source object. If the source is in a grandparent, and the modification is in the child, DESCENDANT_OVERRIDDEN will not be set on the intermediate namespace.

Implementation

pkg/forest should be modified to allow for conditions to be quickly set and unset. I think the best way to do it is as follows: forest.Namespace gets a new Conditions map[string][]Code field. The value ([]Code) is a list of typedef'd strings that can take on the values shown above. The key (the string) is one of the following:

• Empty for conditions affecting this namespace (ie CRIT_PARENT_MISSING, CRIT_PARENT_INVALID, `DESCENDANT_OVERRIDDEN)
• A namespace name if referring to another namespace (ie CRIT_ANCESTOR)
• A string of the form group/version/kind/namespace/name for the OBJECT_ conditions.

This format allows the reconciler for any object to easily clear any conditions that it is causing in any other namespace. Then, the namespace reconciler can reassemble the errors into the per-code format shown above.

Get official container registry from #wg-k8s-infra

We need to move from docker hub to an official k8s sig registry. Info from @kfox1111 here:

#42 (comment)

Tenant Operator does not work using the Readme

It does not work, the t1adminnamespace or the t1-ns1 doesnot get created.

root@wg-multitenancy:~# kubectl apply -f multi-tenancy/tenant/config/crds/tenancy_v1alpha1_tenant

tenancy_v1alpha1_tenantnamespace.yaml tenancy_v1alpha1_tenant.yaml

root@wg-multitenancy:~# kubectl apply -f multi-tenancy/tenant/config/crds/tenancy_v1alpha1_tenant.yaml

customresourcedefinition.apiextensions.k8s.io/tenants.tenancy.x-k8s.io created

root@wg-multitenancy:~# kubectl apply -f multi-tenancy/tenant/config/crds/tenancy_v1alpha1_tenantnamespace.yaml

customresourcedefinition.apiextensions.k8s.io/tenantnamespaces.tenancy.x-k8s.io created
root@wg-multitenancy:~# kubectl apply -f multi-tenancy/tenant/config/manager/
all_in_one.yaml manager.yaml

root@wg-multitenancy:~# kubectl apply -f multi-tenancy/tenant/config/manager/all_in_one.yaml

namespace/tenant-system created
clusterrole.rbac.authorization.k8s.io/manager-role created
clusterrolebinding.rbac.authorization.k8s.io/manager-rolebinding created
validatingwebhookconfiguration.admissionregistration.k8s.io/tenant-validating-webhook-cfg created
service/controller-manager-service created
secret/webhook-server-secret created
statefulset.apps/controller-manager created

root@wg-multitenancy:~# kubectl get crd

NAME CREATED AT
tenantnamespaces.tenancy.x-k8s.io 2019-09-20T21:54:48Z
tenants.tenancy.x-k8s.io 2019-09-20T21:54:42Z
root@wg-multitenancy:#
root@wg-multitenancy:# kubectl get statefulset
No resources found in default namespace.
root@wg-multitenancy:~# kubectl get statefulset -n tenant-system
NAME READY AGE
controller-manager 1/1 25s

root@wg-multitenancy:~# kubectl apply -f multi-tenancy/tenant/config/samples/tenancy_v1alpha1_tenant

tenancy_v1alpha1_tenantnamespace.yaml tenancy_v1alpha1_tenant.yaml

root@wg-multitenancy:~# kubectl apply -f multi-tenancy/tenant/config/samples/tenancy_v1alpha1_tenant.yaml

tenant.tenancy.x-k8s.io/tenant-sample created

root@wg-multitenancy:~# kubectl get tenant
NAME AGE
tenant-sample 6s

root@wg-multitenancy:~# kubectl get ns

NAME STATUS AGE
default Active 3m44s
kube-node-lease Active 3m47s
kube-public Active 3m47s
kube-system Active 3m47s
tenant-system Active 61s

root@wg-multitenancy:~# kubectl apply -f multi-tenancy/tenant/config/samples/tenancy_v1alpha1_tenantnamespace.yaml

Error from server (NotFound): error when creating "multi-tenancy/tenant/config/samples/tenancy_v1alpha1_tenantnamespace.yaml": namespaces "tenant1admin" not found

root@wg-multitenancy:~# kubectl get ns

NAME STATUS AGE
default Active 4m20s
kube-node-lease Active 4m23s
kube-public Active 4m23s
kube-system Active 4m23s
tenant-system Active 97s

also directly using the link does not work, I need to git clone and then use it..
kubectl apply -f https://github.com/kubernetes-sigs/multi-tenancy/blob/master/tenant/config/crds/tenancy_v1alpha1_tenant.yaml
error: error parsing https://github.com/kubernetes-sigs/multi-tenancy/blob/master/tenant/config/crds/tenancy_v1alpha1_tenant.yaml: error converting YAML to JSON: yaml: line 618: mapping values are not allowed in this context

HNC: stop modifying source files whenever we call `make docker-build`

See the following fairly terrifying lines in Makefile (added by kubebuilder when it first scaffolds the project):

        docker build . -t ${IMG}
        @echo "updating kustomize image patch file for manager resource"
        sed -i'' -e 's@image: .*@image: '"${IMG}"'@' ./config/default/manager_image_patch.yaml

This means that every time we run make docker-build, our source files get modified! We should not do this. I'm not sure what we should do. Maybe have a non-default kustomization that we can .gitignore?

HNC: finalize the initial kubectl plugin

The initial kubectl-hnc plugin contains the following commands:

• set-parent <child> <parent>: sets the parent. Alias: set
• unset-parent <child>: unsets the parent. Alias: unset
• show <namespace>: shows information about a single namespace
• tree <namespace>: like the Linux tree command for a namespace.
• subnamespace <parent> <child>: creates a new child under the given parent (one #90 is complete). Alias: sub

Questions:

1. Is this a good list of commands for the initial release of the HNC?
2. What else do we need to do to make this plugin usable? Currently, make kubectl installs the binary into /bin, and . devenv adds /bin to your path, thus "installing" the plugin. How are kubectl plugins usually distributed?

HNC: Self-service namespace creation

Allow tenant admins who do not have cluster-wide namespace creation privileges to still create subnamespaces. Webhooks will be addressed by subsequent issues; this issue only tracks the core functionality (the reconciler).

These changes are already underway with #84; the remaining commits in https://github.com/adrianludwin/multi-tenancy/tree/self-serve will follow once this is merged.

Auto-create namespaces when Tenant CR is created

When a Tenant CR is created, the namespaces listed is automatically created.

HNC: prevent users from applying the inheritedFrom label

hnc.x-k8s.io/inheritedFrom is used by the controller to determine which objects to manage. We should add a webhook to prevent users from applying it directly.

HNC: Finalize the initial list of HNC-controlled objects

Currently, the HNC will propagate Roles, RoleBindings and Secrets. I'd like to propose the following additional Kinds that get propagated in HNC 0.1:

• ConfigMap
• NetworkPolicy
• PodSecurityPolicy
• ResourceQuota
• LimitRange

That is, all the common policy objects that an admin might want to set and then forbid subadmins from modifying.

In HNC 0.2 (aka Phase 2 in the design doc), we can look at making this set configurable.

Use kubebuilder to write tenant-controller poc

Adopt kubebuilder to help write tenant-controller poc.

HNC: add the `tree` label to all namespaces with explicit hierarchies

As described in the design doc, we'll add a <namespace>.tree.hnc.x-k8s.io/depth label to all namespaces with nontrivial hierarchies (i.e. one or more children or any parent).

Can we update the multi-tenancy/docs/profiles/profile-soft-multitenancy-s1.md

Can we update the profile-soft-multitenancy-s1.md to specify the actions each persona will be executing as below :

Clarification: How the workloads are assigned for the tenant ?

I am running the poc in the lab. Here is my understanding of the work regarding the tenant. Please help me to clarifies it.

Ownership
For resources created for a tenant, the OwnerReferences in the metadata points to the Tenant CR for maintaining the ownership. Currently, only namespaces are considered to adopt this reference. Additionally, a label is introduced on all resources owned by a tenant:

multi-tenanct.k8s.io/tenant=tenant-name

The Flow

• Cluster admin creates a cluster;
• Cluster admin creates all cluster-scoped policy objects in the cluster (e.g. ClusterRole, ClusterRoleBinding, PodSecurityPolicy etc.);
• Cluster admin creates NamespaceTemplate objects for per-tenant policies (e.g. Role, RoleBinding, NetworkPolicy, ResourceQuota etc.);
• Tenant Admin creates Tenant CR;
• The controller for Tenant CRD automatically creates the desired namespaces in Tenant object, populates the policy objects defined in NamespaceTemplate objects into the namespaces and also the access control resources (e.g. with RBAC: Role, RoleBinding) for admin identities.

Let's take tenant CRD is as below

---
apiVersion: tenants.k8s.io/v1alpha1
kind: Tenant
metadata:
  name: tenant-a 
spec:
  namespaces:
      - name: ns-1
      - name: ns-2

This creates namespace as below:

# kubectl get ns
tenant-a-ns-1   Active   7d
tenant-a-ns-2   Active   7d

and pod spec will be something like this below:

apiVersion: v1
kind: Pod
metadata:
  name: dummy-pod
  annotations:
    multi-tenanct.k8s.io/tenant=tenant-a
spec:
  containers:
    - name: demo-container-1
      image: karlherler/pause 
      resources:
        limits:
          dummy/dummyDev: 2

How I can make sure that the workload dummy-pod is in tenant-a-ns1 or tenant-a-ns2? Am I missing some details here?

Tenant CRD Status subresource

Today Tenant CRD has Status but the content is not actually populated. We need to populate the content to reflect:

• The status of Tenant: Pending/Ready
• Any error conditions: e.g. some namespaces are not successfully created
• Any thing requested in the CR can't be satisfied/approved.

HNC : New repository

hi @adrianludwin ,

The Hierarchical Namespace Controller is pretty interesting. Even if this project is a pre-alpha and also an incubation of the SIGs multi-tenancy... I think hnc is a project on its own (but it's a point of view).

I wonder if we can have a new repository for the HNC to simplify contribution and issue reporting?

Report Event during Tenant activities

For create/update/delete activities on Tenant CR, emit Event objects for any success/failures.

tenancy crd client code

Is there we can upstream the tenant crd client code using https://blog.openshift.com/kubernetes-deep-dive-code-generation-customresources ?

HNC: Configure the list of HNC-controlled objects at the cluster level

Issue #91 hardcodes the initial list of GVKs controlled by HNC. We should make this configurable.

HNC: Implement RBAC rules in a webhook

Follow the restrictions in the design doc about changing the parent of a namespace.

HNC: allow kubectl-hnc to get the kubeconfig passed in by argument

See the TODO in pkg/kubectl/root.go.

HNC: parallelize object reconciliation

Just like #64 allowed multiple namespaces to be reconciled in parallel (pending #102) and, more importantly, to be requeued in cases of error, we should make the same change to the object reconciler.

Tide and GitHub settings for this repo are conflicting.

https://prow.k8s.io/tide-history?repo=kubernetes-sigs%2Fmulti-tenancy

Tide needs to be configured to use the 'rebase' merge method for this repo or the repo needs to allow merge commits: Merge commits are not allowed on this repository.

Please change the GitHub settings for the repo to allow merge commits or configure Tide to use the rebase merge method ASAP to allow PRs to merge and avoid wasting GitHub API ratelimit.

HNC: publish basic metrics

See https://kubebuilder.io/reference/metrics.html. We need to also write a mini-design doc (possibly write in this issue?) stating which metrics we need to collect.

/assign @lafh

Define NamespaceTemplate CRD

To support policy based namespace population, define NamespaceTemplate CRD to contain policies to be auto-created in tenant's namespace.

HNC: kubectl-hnc should not allow duplicate requiredChildren

See comment in pkg/kubectl/sub.go.

/good-first-issue

HNC: configure Prow for presubmit testing

/assign @danielSbastos

Tenant Controller prototype code skeleton

Auto-populate minimal RBAC in namespaces created after Tenant CR

A RoleBinding is created to bind Tenant Admins defined in Tenant CR to ClusterRole admin.

HNC: remove the SA exemption in the validating webhook

See: #78 (comment)

The HNC SA can make any change so that it can update the singleton's status fields, even if the spec is illegal. I'm pretty sure there's a way to update only the status, which would allow us to remove this exemption.

HNC: prevent required children from being removed once added

Per the Phase 1 UJ, required children should be "permanent" - that is, once a namespace has been created as a child namespace, it cannot be removed (until we add Phase 2 features).

This might be a bit tricky since we basically need to compare the old and proposed version of the object in the validating webhook, which we haven't done yet.

This is a continuation of #90

HNC: forbid propagated objects from being modified

Thanks to #65 , we can now report when a user has incorrectly modified a propagated object, but ideally we should prevent it instead. So we should add a check in pkg/validators to ensure that propagated objects are never modified by anyone other than the HNC itself.

HNC: figure out what to do with kustomize and allow one-line installation

It would be nice to be able to install the HNC by saying kubectl apply -f <some single file>, but for reasons I don't fully understand, kubebuilder creates a kustomization structure with only a single configuration (/config/default). We should figure out why this is, if we should take advantage of it somehow, maybe to help solve #112.

/assign @lafh

since she has experience with kustomize.

HNC: Change singleton names from hier/Hierarchy to hierarchy/HierarchyConfiguration

Currently, the HNC defines the hierarchy using a Kind called Hierarchy (kubectl short name: hierarchy/hierarchies) and a per-namespace singleton called hier. However, these names have a few problems:

• hier is hard to say and looks kind of dumb. I wanted it to be short so we could easily type it into kubectl, but the new kubectl plugin (kubectl hnc) makes this less relevant than I initially thought.
• hierarchy is a pretty generic name to reserve for the HNC in kubectl.

Here are the options I'm considering:

1. Singleton name hierarchy, Kind HierarchyConfiguration/hierarchyconfig/hierarchyconfigs
2. Singleton name tree, Kind TreeConfiguration/treeconfig/treeconfigs

Which would you prefer? Or do you have another preference?

Update: we are going with option #⁠1

Kindly specify tenant poc section -PoC Development Guide is deprecated

Kindly specify tenant poc section -PoC Development Guide is deprecated

HNC: allow multiple worker threads

I believe the number of worker threads is currently set to 1 by default, but we should be able to handle more.

HNC: handle object propagation if the in-memory forest is not yet synced

See TODO in pkg/controllers/object_controller.go, method hasCorrectAncestry. This method can fail incorrectly the source, dest or intermediate namespaces haven't been synced yet, which can easily happen during controller startup. This could lead to objects being incorrectly deleted until the controller stabilizes. We should try to find a way to avoid this if possible.

HNC: remove the long delays in object_controller_test.go

The object propagation tests have 5s delays, and even then they sometimes fail. This doesn't really make sense; they should be passing reliably. We should figure out what's causing the failures in these (very small) test cases and fix them.

HNC: don't propagate objects with initializers or finalizers

We wipe away the initializers and finalizers on any propagated objects. Instead, we should probably refuse to propagate any object that has an initializer or finalizer. See #201 for a continuation of this feature.