Restrict Service : Archetype to be 1:1 about serving

[evankanderson]

My initial take:

It's easier to relax restrictions later, so I'm sympathetic to attempting to restrict this relationship to 1:1 between Service and Archetype at the moment.

One idea for how to do this (which would also simplify the initial deployment experience to a single API call):

• When an Archetype is created, it implicitly creates an ElaService configured for automatic rollout with the same name, e.g.

POST /api/elafros.dev/v1alpha/namespaces/jimmy/archetypes

apiVersion: elafros.dev/v1alpha
kind: Archetype
metadata:
  name: thumbnailer
...

This would turn around and create an ElaService like the following:

apiVersion: elafros.dev/v1alpha
type: ElaService
metadata:
  name: thumbnailer
spec:
  traffic:
  - archetypeName: thumbnailer
    percent: 100

We could even set metadata.ownerReference on either the Archetype or the ElaService if we want to perform automatic garbage collection when the other is deleted.

Two concerns about this solution:

1. If we discover that 1:N or N:N are useful/requested by many customers, how do we signal to avoid the automatic ElaService creation and/or metadata.ownerReference management?
2. If we use metadata.ownerReference, which way should the ownership go? If we think that 1:N from ElaService to Archetype is the most common pattern (e.g. an experimental/rewrite release train and the main production train off the same DNS name) and we think that ElaService is the top-level concept that customers will think of, then it would make sense to make the Service own the Archetype. (Obviously, in complex scenarios, customers could simply remove the ownerReferences completely to avoid garbage collection.)

[dewitt]

If we wanted to go all in on the 1:1 relationship, we could address the naming problem of 'Archetype' (see issue #65) at the same time by inlining it into ElaService entirely.

apiVersion: elafros.dev/v1alpha
type: ElaService
metadata:
  name: thumbnailer
spec:
  template:
    serviceType: container
    env:
      - name: TARGET
        value: thumb_v1
    containerSpec:
      image: docker.io/elafros/thumbnailer:latest
  traffic:
    percent: 100

[mattmoor]

I like the separation we have, and leaving the door open to relaxing this later.

What we have supports a lot of interesting scenarios, without significantly complicating the 99% case (IMO).

I think that UX changes a lot over time, based on how folks use the product, so I'd rather not tie ourselves too closely to a particular model.

[qelo]

+1 to @dewitt, that would be the most elegant way to maintain 1:1 and would avoid all the confusion coming from a separate object

Would be harder to relax back to n-n in the future, but: what if we embraced ElaService composition instead?

Like: You wanted two Archetypes? Just have two sub-services and route between them using other Ela primitives (we can design those separately, like a service that just splits traffic between other services).

I think service composition is a very powerful model, because we still allow for very sophisticated configurations/graphs, while the mental model of basic building blocks remains simple.

Otherwise, I like the approach that @evankanderson is exploring, but we will run into the number of problems:

1. "ElaService is the top-level concept that customers will think of" -- agree
2. We end up with this awkwardness that you come with the intent "I'd like to create a new service", but you are doing /archetypes POST call for that.
3. More preferred would be if ElaService created an Archetype automatically, but then where does initial Archetype spec fit in that call?
4. Actually we risk problem (3) also in other direction: if we ever add some more settings to ElaService, it might be a problem to set them during Archetype creation (you would need to update ElaService just after creation -- not so nice)

[vaikas]

I feel like we're having this discussion from scratch all over again, so it would be beneficial to have a pointer to the documents that outlined 'alternatives' considered. For example, we had one version of the model where service inlined the template and it became very unwieldy and got a lot of pushback for being one single huge object (which I agree with). Are we also proposing on not allowing a specific revision to not be pointed to by the service? I am not convinced that N:M model makes sense. that part I agree with. If we want to restrict a service in a way that only one archetype can be specified in the service, I'm fine with that. My only concern is that there had been existing use cases identified for 1:N. Are we also saying that there can only be a single archetype _or_ single revision pointed to by a service? Or, how's that going to work? My preference would be to rather keep the decoupling of service / archetype as far as resource types goes. Otherwise I feel like we're going to conflate the separation of concerns.

…

On Sat, Feb 3, 2018 at 10:01 AM qelo ***@***.***> wrote: +1 to @dewitt <https://github.com/dewitt>, that would the most elegant way to maintain 1:1 and would avoid all the confusion coming from a separate object Would be harder to relax back to n-n in the future, but: what if we embraced ElaService composition instead? Like: You wanted two Archetypes? Just have two sub-services and route between them using other Ela primitives (we can design those separately, like a service that just splits traffic between other services). I think service composition is a very powerful model, because we still allow for very sophisticated configurations/graphs, while the mental model of basic building blocks remains simple. Otherwise, I like the approach that @evankanderson <https://github.com/evankanderson> is exploring, but we will run into the number of problems: 1. "ElaService is the top-level concept that customers will think of" -- agree 2. We end up with this awkwardness that you come with the intent "I'd like to create a new service", but you are doing /archetypes POST call for that. 3. More preferred would be if ElaService created an Archetype automatically, but then where does initial Archetype spec fit in that call? 4. Actually we risk problem (3) also in other direction: if we ever add some more settings to ElaService, it might be a problem to set them during Archetype creation (you would need to update ElaService just after creation -- not so nice) — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#63 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKwedJd0GcHtvSS_0HSfJqOvkjBbup9Xks5tRJ8BgaJpZM4R39f2> .

[mikehelmick]

The decoupling here between ElaService and Archetype is the least restrictive for future expansion and we should prefer keeping this separation in the API. The original inclusion of everything included in the Archetype in the service definition was indeed complicated, another reason for maintaining the separation.

For the future N:M case, if we go there, garbage collection may or may not be necessary or even desirable. There are configurations where and archetype could become temporarily orphaned. Detecting orphaned ElaService and/or Archetype objects could be easily done and clearly reflected in the status of the object. If we end up going that direction, l can see how we could go to configurable garbage collection for these entities.

The lack of explicit service creation isn't without precedent. Google App Engine services are created by creating the first version of that service. There is a nice parallel here with the ElaService being created by the creation of the Archetype. The user intent is the same. I want to create a new service, but the creation of a service without something to run isn't necessarily meaningful. I don't find creating the ElaService through the creation of the Archetype to be a problem in that respect, it ensures that the service has a way for revisions to be created.

[evankanderson]

Pulling out and cleaning up the doc referenced by @vaikas-google; this is re-treading existing territory, but new people have entered the discussion with new opinions. The double goal here is to satisfy their concerns and to document the decision for future generations. :-)

[evankanderson]

I did a little wire-framing of a possible UI treatment to address some of @qelo 's concerns. The idea is that the basic UI would (like the CLI) rely on a few conventions to cover the 98% case, and then have a more complete and complex UI which would expose all the concepts available for both the 2% case and for users who were curious about the underlying system architecture.

In the basic view, the system would display a list of Services with several workflows available based on the existence of the corresponding Archetype (e.g. "redeploy source", "update settings", etc). If the corresponding (same name) Archetype does not exist, the UI would degrade gracefully by offering a link to the Advanced view. Over time, the "same name" condition could be relaxed in cases where there was a single Archetype with a different name from the service, but this might not be desirable, as it might put an equivalent burden on other tools.

In the Advanced View, each resource would be represented graphically. Services and Archetypes (and their corresponding Revision histories) would be connected via "sends traffic to" arrows to represent arbitrarily complex configurations. While more accurate to the underlying resources, this view would require users to understand all three of the resource concepts in the API. I also forgot to draw the blue "deploy" button on Archetypes in these mocks. (oops!)

Note that both of these designs use a "stacked file" icon to represent the set of Revisions descended from an Archetype (see #65 for possible renaming), as I found that it was somewhat difficult to create individual graphical representations of all existing Revisions. For the Advanced view, it would be possible to have the actual live Revisions present as a third column in the UI; it would be expected that a single Service might link to 10 revisions (possibly via Archetype latestReadyRevisionName reference), but the number would typically be bounded even smaller than that.

[evankanderson]

Some example scenarios for each direction of 1:N and N:1 between Service and Archetype:

1:N

• Experiments (what if we did things async, or doubled available RAM?).
  • With RBAC, it's possible to grant a developer access to update only one Archetype. If you delegate e.g. 1% of traffic to that Archetype, the developer can do 1% experimentation against the environment without the ability to cause further damage.
• More generally, replacing an existing service with a new implementation while both are being built and deployed. (For a cutover where you don't ever need to go back and fix something in the old implementation, you can simply replace the old Archetype with new. We call this "big bang" deployment, for the sound it makes.)

N:1

These are slightly more esoteric; I've done these in previous production environments by slipping a separate load-balancer in front to provide the additional naming.

• Strangler Application: while slipping the proxy in front, it may be desirable to have a second Service which only points at the legacy application.
• Endpoint consolidation: use a single App to handle what used to be 2 or more named endpoints, by redirecting the traffic to the single implementation. This is particularly useful for low-traffic services where it may be hard to update all the callers. An alternate implementation of this is to deploy a forwarder service at N-1 of the other endpoint services (at which point, you get separate monitoring metrics, pay 2x invocations, etc).

[qelo]

The option with ElaTrafficSplit is the one to which I was referring by "service composition". That's my favorite.

I think it much better delivers on our explicit "Define a minimal set of concepts which a customer must understand." requirement. The key difference is that customers following simple usecases don't need to create/understand TrafficSplit at all, while they do have to create/understand Archetype in the current design. At the same time, TrafficSplit still covers complex use cases. @evankanderson I believe all listed N:N use cases are still expressible with TrafficSplit, aren't they?

I think some customers will follow TrafficSplit model anyway (even if we keep Archetypes), by using separate loadbalancing solutions. Separate routing layer would also be required to implement multi-region availability. Having TrafficSplit keeps us consistent with all that.

We would also avoid all the problems around "how to validate 1:1 now, but relax in the future". We can simply not have TrafficSplit for now and add it later.

@evankanderson Re advanced view: that's good ideas, but note that building and maintaining a separate advanced view is very expensive for the UI team, with costs doubled or more, and with high chance that one view gets neglected (especially when split is 98%/2%). I would much prefer if we had one standard view that naturally scales with complexity and can be used for both simple and sophisticated use cases. The UI I envision for TrafficSplit seems closer to that (basically you can reuse entire Services UI to inspect/configure sub-services, you only need something more to configure TrafficSplit objects)

@vaikas-google

Are we also proposing on not allowing a specific revision to be pointed by the service?

I think we are not. Just that the service should only have one list of revisions (one Archetype) and should only point to those (shouldn't point to revisions from other services). There can still be many revisions pointed by the service, e.g. to make more revisions routable, or to implement gradual rollout. I saw a lot of push-back for current/next simplification in past discussions and I don't have any reasons to reopen that.

Re: "having this discussion from scratch all over again" -- I can understand how this can be frustrating.. :( I'm trying to dig through all the past docs and comments to minimize that. I found strong evidence against the very basic model (Service+Revisions, with current/next, without TrafficSplit), but I haven't found much against TrafficSplit. We've got some new evidence too: difficulties in expressing things in Web UI + a lot of new people learning about the API confused by Archetype model. Since API definition has profound long-term consequences, I hope we can still re-evaluate.

[evankanderson]

The TrafficSplit solution has several problems which led us to not pursue it in much detail. IMO, the most substantial problem is related to namespacing. If the Service is addressable, then one of the following must be true:

1. TrafficSplit and Service share a namespace, and it is impossible to create a TrafficSplit with the same name as a Service.
2. TrafficSplit and Service use distinct namespaces, and a different serving name prefix is required for TrafficSplits compared with Services. In turn, this means two sets of wildcard DNS entries, certificates, etc.

For both 1 and 2, the decision to introduce a TrafficSplit is necessarily a visible change to all the consumers of your service, because the DNS name used to reach the service changes. In the case of 1, there are additional complications in attempting to ensure (within the apiserver) that TrafficSplit and Service have non-overlapping namespaces despite being different types. In the case of 2, the naming of the service ends up exposing whether or the service is using TrafficSplit, which seems like it should be an implementation detail hidden from consumers of your service.

After talking with @dewitt, he had an interesting suggestion. Let's say that we don't allow Service to be addressable, and instead use TrafficSplit (or possibly a name like "Router" or "Endpoint") to act as the sole addressing mechanism. We can then drop any percentage/subdomain settings from the ElaService, and have it solely responsible for managing the configuration and deployed code. (This avoids the "two ways to do the same thing" problem mentioned in the doc.) We could rename this construct "Configuration", "Settings", or "Blueprint", to better capture that it acts as "how the service should be," or leave the name as "ElaService" ("Service" conflicts with the k8s concept, which is substantially different).

@qelo What do you think?

[mikehelmick]

We can keep adding layers of abstraction and containers for traffic split all the way up. At some point we have to pick a container for things.

Keeping traffic split at the service level is what I think makes the most sense for a first pass. If we wanted to add a higher level concept of an Endpoint in the future, that could be used to compose services in interesting ways, there's nothing preventing us from doing so, let's just make sure that there is a user need for it.

[qelo]

I assumed TrafficSplit and Service would share the namespace, or rather: there would be a shared namespace for urls and both TrafficSplits and Services would compete for the urls there, even if their object namespaces are separate (I imagine there could be an optional parameter that allows a service to be routable at a different url than its object name). The conflicts in url namespace would be reflected by status.

Replacing the Service with a TrafficSplit without changing the url would indeed be a problem. Similar routing problems can reappear in various forms, e.g. if a customer wants to migrate the service to another project, or customer ends up with Router -> Router -> Service after some migration and wants to clean this up. Customers that have another routing layers (like GCLB) would probably deal with it there. But if we really wanted to allow a invisible transition from Service to TrafficSplit, we would need some smart resolution of conflicts in URL namespace, e.g. TrafficSplit always takes precedence, or both Service and TrafficSplit declare optional priority for URL.

The last suggestion around Endpoint is equivalent to renaming, right? Service -> Router/Endpoint, Archetype -> Service/Blueprint/Prototype? I think it's not bad, but let's keep #65 for naming.

I think we have two options:

1. Explore if TrafficSplit option can be made comfortable despite URL conflict issue.
2. Go back to idea of validating 1:1 and address naming separately in #65.

In (2), we said that creating a Prototype would automatically create a Router. What if the Router ever has any additional settings that could be set during creation? Do we know how would that work with automatic creation?

[mikehelmick]

I misunderstood @evankanderson comment above. Let me restate the proposal

• Rename Archetpype to ElaService. An ElaService encapsulates how to build revisions and owns a revision stream. ElaService on it's own does not get a URL that can be routed to.
• Introduce a new object, let's call it Endpoint that defines ingress URL mappings. An Endpoint can aggregate multiple ElaServices via a traffic configuration.

Questions derived from this: Do we lose the ability to route to a newly created revision for testing purposes? It seems that mapping a specific revision of an ElaService in the Endpoint traffic configuration could be awkward.

[mattmoor]

K8s has both Service and Endpoint. They are also related to each other, and not in these same ways :(

[vaikas]

@mikehelmick Couple of clarifications to make it more concrete in my head.
Rename Archetype -> ElaService. The way I read this is that there are no functional changes there, just a name change. As in, are there any visible API changes in this?

Second point, introduce a new object, seems like we are effectively renaming the existing ElaService -> Endpoint? Again, it seems to be very similar to the existing ElaService?

[mikehelmick]

Yes, this is a straight rename (credit @evankanderson and @dewitt ). That seems to solve the concerns of #65

If we agree to the rename Archetype -> ElaService
and Service -> Route (due to k8s conflict with Endpoint)
and agree that, for now, There is a 1:1 relationship between Route:ElaService

Does that solve the UX review concerns for both #65 and #65 (@qelo @steren )?

[qelo]

Let's not mix the issues. It's probably easier to keep #65 for straight renames and reserve this issue for discussing 1:1 and other structural changes in the API.

For 1:1, I'd be fine with validation provided that we agree that everyone (incl. Web UI) has to agree and commit before we can relax it.

I still think it's suboptimal, although acceptable to me. Different angle to why I think it's suboptimal:

If we knew that we will never relax 1:1, would we keep Archetype separate from a Service? I believe the answer should be "No, we would merge the objects". My claim is that it's quite likely that we will never relax it. First because relaxing it is not so trivial and not so backward-compatible after all -- different clients, tools and UIs can depend on 1:1, the same way our Web UI will depend on it. Second because customers will start achieving more complex scenarios simply by putting additional routing layers on top of it (e.g. GCLB when in GCP, or other load balancing solutions). They anyway have to do this to have custom domains or multi-region or when migrating between products (e.g. how do I migrate traffic from VM to Elafros?). When this happens, we ended up with unnecessarily separate concepts forever bound in 1:1 although it could have been one concept from start.

(I'm also still unclear on this question re validation of 1:1:
We said that creating a Prototype would automatically create a Router. What if the Router ever has any additional settings that could be set during creation? Do we know how would that work with automatic creation?)

[mikehelmick]

There are other APIs in existence that have mutable settings that can't be set on creation. For example, in the Google App Engine API, a service object is implicitly created when the first version object in that service is created. Later, a user can patch the service object to change state. The service object can't be created directly.

[cooperneil]

I agree that the 1:N use cases (A/B testing / experiments, and gradually replacing the implementation behind a service) are more compelling than the more esoteric N:N use cases (Strangler application). FWIW, in the near term I'd vote we have 1:1 by convention and potentially allow 1:N for advanced scenarios, restricting N:N through validation, if we find an acceptable representation in the UI for 1:N.

Beyond facilitating the 1:N cardinality, there are other benefits to the separate Archetype and Service, such as a separation of concerns. Consider the manual rollout case below:

apiVersion:  elafros.dev/v1alpha
kind: ElaService
metadata:
  name: thumbnailer
  ...
spec:
  traffic:
   # manually rolled out
   - revisionName: 123
     name: stable
     percent: 90
   - revisionName: 456
     name: canary
     percent: 10
  # automatically rolled out
   - archetypeName: thumbnailer
     name: head
     percent: 0

In this scenario, a manual rollout is incrementally shifting traffic from named revision 123 to 456. In addition, the archetype is referenced to have a floating addressable 'head' revision (that serves 0% traffic).

The act of creating a new Revision is done through a PATCH to the Archetype. The act of manually rolling it out is done through a PATCH to the Service. From an authorization perspective, these may be different user roles, which are reflected by the different resource types that embody these different concerns.

In the above advanced Service configuration, a user with a role to update the Archetype can still create new revisions and test them through the 'head' subdomain, but to manually rollout that new revision to serve customer traffic, a user with a more restrictive role to update the Service is necessary.

[dewitt]

@evankanderson's suggestion above that we could implicitly create a ElaService really got the ball rolling on a few related ideas that helped me better understand the object model itself. That in turn led to a whiteboard session, and led me to the following picture with Evan:

In particular, it helped me appreciate that in the happy 90% case, the user is really dealing only with one principle resource, shown here under a proposed new name of Configuration (previously known as the RevisionTemplate), and that this resource is used both to stamp out immutable Revisions and parent an implicitly created Route (previously known as ElaService).

It would be common for there only to be a single Route, and that this one Route will be implicitly created from default rules ("100% traffic to the latest Revision"). TODO: Figure out how the domain could be set implicitly without pushing that field up into the parent Configuration.

It also isn't unreasonable for the user to want 1:N Route:Configuration to support the experiment traffic splitting use-case that @cooperneil outlines above.

I don't have a strong opinion about N:N or N:1, other than to say we shouldn't do anything that would forever prohibit either, but I don't expect a critical use case for them in the near future.

[Edited for clarity.]

[mattmoor]

@dewitt I liked this model (and surprisingly the name, despite its generality; b/c: "config" vs. "code"). A couple thoughts:

• I think we should just always create a Route: this is consistent with (and cleaner than) our "by convention RevisionTemplate has the same name as the ElaService".

• I think that we can still leave the door open to manually creating another Route, should the use case for multi-Configuration splits arise. We shouldn't do this without a complete story, including UX.

I realized there is a dependency cycle here, which makes me a tiny bit uncomfortable, but I think I can get over it. :)

[cooperneil]

(going to use the old Service & Archetype names here to not confuse with issue #65)

Agree we could always have the BOGO behavior of creating a Service when an Archetype is created, even if the end intention is 1:N. Though this may result in an unused service. Example:

Starting state: User has Archetype and Service "thumbnailer", which is automatically rolled out to 100%
Goal: Creating an experiment
Steps:

1. client creates Archetype ("thumbnailer-experiment")
2. system creates new Service ("thumbnailer-experiment"), and the first Revision, which is served by that service
3. client PATCHes "thumbnailer" Service to add the 2nd "thumbnailer-experiment" Archetype, serving a % of traffic:

apiVersion:  elafros.dev/v1alpha
kind: ElaService
metadata:
  name: thumbnailer
  ...
spec:
  traffic:
   - archetypeName: thumbnailer
     name: thumbnailer
     percent: 90
   - archetypeName: thumbnailer-experiment
     name: exp
     percent: 10
...

End state: 2 archetypes, 2 services. "thumbnailer" service routes traffic to a split of the latest revisions from both "thumbnailer" and "thumbnailer-experiment" archteypes. "thumbnailer-experiment" service routes traffic to just "thumbnailer-experiment" archetype.

Note also that an advanced scenario like this, or doing manual rollouts, would be the only time the client would need to update a Service

[cooperneil]

... though it seems the side effect of always creating a Service when creating an Archetype is that 1:N becomes N:N, per the above example.

But we can separate the composition-like relationship (between Archetype “thumbnailer-experiment” and Service “thumbnailer-experiment”) from the aggregation-like relationship (between Archetype “thumbnailer-experiment” and Service “thumbnailer”)

[steren]

This discussion assumes that we cannot do without an "Archetype" / "Configuration" / "RevisionTemplate" (a resource that contains a base specification for new Revisions but that is not a Revision by itself), I opened #102 to understand why we do, and to challenge it.

[dewitt]

@steren -- I added a comment in #102. Good question to ask.

To test my understanding, I also made a picture that illustrates the model for @cooperneil's useful experiment traffic split exercise above. Again, I'm using the names 'Configuration' and 'Route' just because I find them intuitive, but ignore the names and focus how the 1:N Route:Configuration pattern looks in practice.

A fairly common use-case for 1:N, and fortunately, it's also fairly easy to model and reason about.

Note, I'm not as convinced I could draw a straightforward example of a real-world M:N, which probably says something...

[cooperneil]

@dewitt, yep exactly - nice illustration. Borrowed this to help explain the rationale for a separate Archetype resource in issue #102

[mchmarny]

Capturing decision from today's sync on this:

We will restrict Archetype (aka Configuration see #65) & Revisions such that an Archetype & created Revisions can only be pointed to by one Service (aka Route, see #65) at a time, using a mechanism like "claims" on the Archetype by the Service

Will close this issue when the related PRs are committed to master.

[evankanderson]

Details on proposed implementation:

(via validation prior to etcd commit)
For each Revision directly or indirectly referenced in the Route configuration, look up the Configuration from Revision.metadata.labels[elafros.dev/configuration]. The Route should then ensure that (for each such Configuration), the Configuration.metadata.labels[elafros.dev/route] label is either non-existent, or points to this Route. If the label is non-existent, update the Configuration to point the elafros.dev/route label to this Route.

[grantr]

If the label is non-existent, update the Configuration to point the elafros.dev/route label to this Route.

I'm just catching up to the latest on this, so forgive me if this isn't relevant, but was there consideration of whether this automatic ownership field should be part of the Route Status instead of Labels? I don't have an argument either way, but I'd like to see the reasoning documented.

Relevant k8s docs:
https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md#spec-and-status
https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md#references-to-related-objects

[evankanderson]

Two reasons why I'd suggested a label (I'd originally thought annotation):

1. It may be convenient for the UI to be able to list all the Configurations that a Route points to using a label query rather than needing to perform the scatter/gather across referenced Revisions.

2. We're not 100% sure whether this restriction rules out important use cases. Since it's easier to expand an API than restrict it*, I'd thought that implementing this restriction in a label might avoid making schema changes which would need to be backed out later.

I'm not sure either of these is strongly compelling, so we could do this as a Status field as well.

• This is only somewhat true, since users might take a dependency on item 1.

[jurekPapiorek]

I'd like to understand argument 1) better, as I've seen it used in other contexts too.

API clients will want to filter by many different attributes of Elafros resource objects - some of them will belong to the spec, some to the status, some could be labels. Is converting all filterable attributes into labels the only solution that k8s provides for filtering?

[evankanderson]

See kubernetes/kubernetes#14129 (comment) for a reason why label selectors may be more efficient than general field selection (which is also supported, though it may be tricky for jsonpath references to fields which are part of a list).

…

On Tue, Feb 13, 2018 at 1:03 AM, Jurek Papiorek ***@***.***> wrote: I'd like to understand argument 1) better, as I've seen it used in other contexts too. API clients will want to filter by many different attributes of Elafros resource objects - some of them will belong to the spec, some to the status, some could be labels. Is converting all filterable attributes into labels the only solution that k8s provides for filtering? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#63 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHlyN0keDSsP1ch1REqgjzNrpNuboHPTks5tUU_fgaJpZM4R39f2> .

-- Evan Anderson <[email protected]>

[mattmoor]

I believe this is now done. We set labels on Configurations that are used in a Route: elafros.dev/route: <route name>