Consolidate namespaces so they are all semi-sane about serving

[mattmoor]

There's an interesting problem with our current namespace setup, which we should address before things progress too far along the current course.

What happens today

Users create Elafros resources in a particular namespace (e.g. default) and the controller creates for it a new namespace (e.g. ela-default) in which it puts resources.

The problem

The problem with this is that we want (and in some cases need) to allows users to take advantage of Kubernetes features that don't straddle namespaces well.

In particular, if a user wants to use imagePullSecrets to pull their private serverless container, this is a LocalObjectReference in Kubernetes. In other words, Pods will look for this Secret object in the same namespace in which they exist.

However, in the standard flow where we are creating Pods in a namespace that we create and manage for them, there is an ordering problem. The namespace is created after the user initiates a deployment, which doesn't give them an opportunity to create the pull secrets to be used in this namespace.

This problem is echoed in other less-essential same-namespace references, such as ConfigMaps or other kinds of Secrets.

Breaking encapsulation

The original intent of having a separate namespace (IIRC) was to encapsulate and hide some of the implementation details of how the Elafros abstractions are materialized. We didn't want to pollute kubectl get deployments, which the user may be using directly, with our generated deployments, but if the user wanted to "break glass" they could see these resources in kubectl -n ela-default get deployments.

To workaround the problem outlined above, we are essentially requiring the user to break encapsulation already for (what I expect to be) a relatively common case. They'd have to pre-create our Elafros namespace and populate it with the sort of local objects that they expect to be able to access.

Simplifying the model

I think given this, we should simplify and revert to a model where we simply create user resources within the namespace containing the Elafros resources. This enables a more natural and familiar flow for referencing local objects. This is also simply more consistent with how other high-level resources (e.g. ReplicaSet) create low-level resources (e.g. Pod).

Regarding the resource pollution concern, we can simply encourage users to make use of separate namespaces on their own, if this even becomes a concern.

Thoughts? @vaikas-google @evankanderson @grantr @dewitt

[mattmoor]

FWIW, this is what we do in build already, so some resources will appear in the user's namespace. We should try and align these on a common approach.

[vaikas]

I'm fine with putting them into the same namespace. But if we do that, it will prevent us from putting them into a different namespace and I could see that being a desireable. I guess we could add something like 'targetNamespace' (optional field) so the user has some choice on still keeping the separation if they so want to? My main thinking is that for encapsulation reasons it would be nice for the user to be able to modify their ela resources but not allow them (without explicitly breaking glass) to operate on underlying k8s resources.

[mattmoor]

I wonder if we can/should achieve that same effect with RBAC within the namespace?

[vaikas]

I think we could, it just seems easier to say, this namespace the user can't modify k8s resources on, rather than add rules for some that they can and some that they can't. But like I said, I'm not wedded to the resources being deployed to a different namespace.

[mattmoor]

I think my meta point is that I think we have alternatives for locking down undesirable mutations (which the controller should handle reconciling anyways), but there isn't such a simple alternative for accessing other K8s resources (if we move to a different namespace).