googlecloudplatform / metacontroller

Lightweight Kubernetes controllers as a service

⚡ Metacontroller Has Moved

Active development of metacontroller continues at metacontroller/metacontroller. This repository is no longer maintained.

metacontroller's Issues

Removing labels/annotations

My DecoratorController is able to apply labels & annotations to resources (in my case, Namespaces) by returning corresponding maps. Is there a mechanism for removing them? Returning empty maps doesn't seem to do the job.

Support for cluster-scoped CRDs and namespace-scoped resources in LambdaController

I saw this demo'ed at KubeCon and it gave me some inspiration to try writing a custom LambdaController sync hook.

I'm attempting to watch cluster-scoped custom resources to manage both cluster-scoped and namespace-scoped child resources (for now, namespaces and services). However, for all of the namespace-scoped resources in the returned children array, I get the following error:

an empty namespace may not be set during creation

I've confirmed that the response coming from my controller contains a namespace for these services, so I'm a bit confused.

Looking through the lambda_controller.go code, it appears that there's a limitation that child resources must be in the same namespace as the parent resource? Just want to confirm if what I'm trying to do is possible.

Thanks!

Following build instrctions fails

Below are the build instructions to build metacontroller:
go get -u k8s.io/code-generator/cmd/{lister,client,informer,deepcopy}-gen
dep ensure
make

Following the steps as mentioned in build instructions fails at "dep ensure" step:
Solving failure: remote repository at https://github.com/kubernetes/metacontroller does not exist, or is inaccessible: fatal: could not read Username for 'https://github.com': terminal prompts disabled
: exit status 128

Hot loop when using Recreate update strategy with Go hook

As reported by @cohix, a lambda hook written in Go that generates a JSON response by serializing the Go structs from the Kubernetes API package can trigger a hot loop of child recreation because Metacontroller thinks the hook wants to set metadata.creationTimestamp to null.

This shows up in the Metacontroller logs as something like:

manage_children.go:143] reflect diff: a=observed, b=desired:

object[metadata][creationTimestamp]:
  a: "2018-08-20T21:21:50Z"
  b: <nil>

This happens because Metacontroller's apply semantics currently assume any field present in the hook response is a field the hook author explicitly wants to set. However, metadata.creationTimestamp is always serialized into the hook response regardless of whether the author wants it, because CreationTimestamp in the ObjectMetastruct is a non-pointer struct field, so the omitempty option has no effect:

https://github.com/kubernetes/apimachinery/blob/dcde72c465a0edef321bedc44fe1c16990970efe/pkg/apis/meta/v1/types.go#L175

Incidentally, metav1.Time has a custom JSON marshaler that serializes the zero value of the struct to null instead of "0001-01-01T00:00:00Z" (as you'd expect given it's a non-pointer field), but this doesn't change anything as far as Metacontroller is concerned. The problem is that the field gets serialized at all; we would prefer that it gets omitted.

---

At a high level, marshaling to JSON from a statically-typed language like Go makes it hard for hook authors to follow the guidance from Metacontroller that they should omit fields they don't care about. You would have to diligently ensure that all fields have proper omitempty semantics for uninitialized struct fields. In Go, that essentially requires using pointers for all struct fields, but if you import the structs from upstream Kubernetes, you don't have control over that.

One option could be to change Metacontroller so it doesn't try to update the field if the desired value matches the last-applied value. I actually thought it already does that because the intention was to match kubectl apply semantics, but then I remembered that Metacontroller is more aggressive about pushing towards desired state on purpose -- controllers generally ought to be more persistent than kubectl. Most controller authors would probably expect that their controller should "reset" the field back to the desired value if someone else changes it. With that said, I don't think we should rule out this option.

Another option is to somehow work around the problem by figuring out that, for some fields, null really means "I don't care" and not "please set this to null". However, that would require specific knowledge of the schema of objects being processed, which Metacontroller so far manages to avoid.

lost fields with decorator controller and jsonnet

hi,

I have a decorator controller that takes services in inputs and create networkpolicies based on a jsonnet template. Some of the networkpolicy fields get lost and I can't find a way to debug the issue.
Here is my sample service:

kind: Service
metadata:
  name: microsegmentation-controller-test
  annotations:
    io.raffa.microsegmentation: 'true'
    io.raffa.microsegmentation.additional-ports: 9999/TCP,8888/UDP
spec:
  selector:
    app: microsegmentation-controller-test
  ports:
  - port: 8080

here is my jsonnet template:

  local service = request.object,
  local additionalPorts = std.split(service.metadata.annotations["io.raffa.microsegmentation.additional-ports"],","), 

  // Create a netowrkpolicy for each service.
  attachments: [
    {
      apiVersion: "networking.k8s.io/v1",
      kind: "NetworkPolicy",
      metadata: {
        name: service.metadata.name + "-np"
      },
      spec: {
        podSelector: {
          matchLabels: service.spec.selector
        },  
        ingress: [
          {
            ports: [
              {
                protocol: port.protocol,
                port: port.targetPort,
              }
              for port in service.spec.ports
            ]
          },
          {  
            ports: [ 
              {
                protocol: std.split(port, "/")[1],
                port: std.parseInt(std.split(port, "/")[0]),
              }
              for port in additionalPorts
            ]
          }  
        ]
      }
    }
  ]    
}    

applying this template to the json of the deployed service gives the following result (tested at this link: http://jsonnet.org/docs/demo.html):

   "attachments": [
      {
         "apiVersion": "networking.k8s.io/v1",
         "kind": "NetworkPolicy",
         "metadata": {
            "name": "microsegmentation-controller-test-np"
         },
         "spec": {
            "ingress": [
               {
                  "ports": [
                     {
                        "port": 8080,
                        "protocol": "TCP"
                     }
                  ]
               },
               {
                  "ports": [
                     {
                        "port": 9999,
                        "protocol": "TCP"
                     },
                     {
                        "port": 8888,
                        "protocol": "UDP"
                     }
                  ]
               }
            ],
            "podSelector": {
               "matchLabels": {
                  "app": "microsegmentation-controller-test"
               }
            }
         }
      }
   ]
}

but when it's processed by the jsonnet server and metacontroller (not sure where the problem is) I get this result:

kind: NetworkPolicy
metadata:
  annotations:
    metacontroller.k8s.io/decorator-controller: microsegmentation
    metacontroller.k8s.io/last-applied-configuration: '{"apiVersion":"networking.k8s.io/v1","kind":"NetworkPolicy","metadata":{"annotations":{"metacontroller.k8s.io/decorator-controller":"microsegmentation"},"na
me":"microsegmentation-controller-test-np"},"spec":{"ingress":[],"podSelector":{"matchLabels":{"app":"microsegmentation-controller-test"}}}}'
  creationTimestamp: 2018-03-21T23:58:25Z
  generation: 1
  name: microsegmentation-controller-test-np
  namespace: metacontroller
  ownerReferences:
  - apiVersion: v1
    blockOwnerDeletion: true
    controller: true
    kind: Service
    name: microsegmentation-controller-test
    uid: 4770104d-2d30-11e8-8cca-fa163e7d0b02
  resourceVersion: "24418290"
  selfLink: /apis/extensions/v1beta1/namespaces/metacontroller/networkpolicies/microsegmentation-controller-test-np
  uid: be565181-2d63-11e8-808b-fa163eb8ad68
spec:
  podSelector:
    matchLabels:
      app: microsegmentation-controller-test

As you can see the ingress field is missing. Also in the last applied field we can see that ingress was applied as an empty array.
What can be happening?
Also is there a way to increase the logs?

Support configurable resync period.

Once we move away from polling (#8), we should add a resync period that can be configured for each lambda controller (e.g. in the CompositeController spec).

By default, there should be a non-zero resync priod for all controllers, to put an upper bound on converging to the desired state after missing changes. Making this period configurable will help enable controllers that do work periodically, even when nothing has changed (e.g. CronJob-type controllers).

pin to release version of k8s deps

Pin to release version of k8s dependencies to avoid flux in api chanages.
#36

CRD status subresource is not recognised by meta-controller

Hello, it looks like the CRD status subresource is not recognised by meta-controller if we struct it. For example, we created following CRD for "websites"

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  # name must match the spec fields below, and be in the form: <plural>.<group>
  name: websites.example.com
spec:
  scope: Namespaced
  group: example.com
  versions:
    - name: v1
      served: true
      storage: true
  names:
    kind: Website
    singular: website
    plural: websites
    shortNames:
    - ws
  subresources:
    status:
      current: 0
      updated: 0
      available: 0
      generation: 0
      route:        
    scale:
      specReplicasPath: .spec.replicas
      statusReplicasPath: .status.replicas
...

and this composite controller

apiVersion: metacontroller.k8s.io/v1alpha1
kind: CompositeController
metadata:
  name: website-controller
spec:
  generateSelector: true
  #resyncPeriodSeconds: 10
  parentResource:
    apiVersion: noverit.com/v1
    resource: websites
  childResources:
  - apiVersion: apps/v1
    resource: deployments
    updateStrategy:
      method: InPlace
  - apiVersion: v1
    resource: services
    updateStrategy:
      method: Recreate
  - apiVersion: extensions/v1beta1
    resource: ingresses
    updateStrategy:
      method: Recreate
  hooks:
    sync:
      webhook:
        url: http://192.168.1.103:8060/website

From the meta-controller log, we cannot see the status of parent in the webhook request.

On the other end, it works if we simply leave the status as following in the CRD

...
  subresources:
    status:      
    scale:
      specReplicasPath: .spec.replicas
      statusReplicasPath: .status.replicas
...

Could you please have a look in the code?

Thanks,

Adriano

Use-case for reconciling external state

(As requested in the FAQ)

One idea I've been toying with is implementing a PagerDuty operator for kubernetes

Things like a PagerDutyService CRD. And then automatically hooking up the generated service endpoint with the AlertManager resource from prometheus-operator.

The main reason to not create the services manually is for us replicating the same set of services across multiple environments, and we're trying to make all the ways of declaring our infrastructure kubernetes manifest shaped, so deploying an app is just applying a bunch of manifests.

Go client library for example controllers

Is there a way for a user to use go client to create a crd (e.g indexedJob) instead of using kubectl and yaml file?

panic when testing the decorator controller

hi, I am getting this error when using the decorator controller:

I0321 03:40:45.050706       1 main.go:55] Discovery cache flush interval: 10s
--
  | I0321 03:40:45.056929       1 main.go:56] API server object cache flush interval: 30m0s
  | I0321 03:40:45.058576       1 reflector.go:202] Starting reflector *v1alpha1.CompositeController (30m0s) from k8s.io/metacontroller/client/generated/informer/externalversions/factory.go:74
  | I0321 03:40:45.058604       1 reflector.go:240] Listing and watching *v1alpha1.CompositeController from k8s.io/metacontroller/client/generated/informer/externalversions/factory.go:74
  | I0321 03:40:45.058753       1 reflector.go:202] Starting reflector *v1alpha1.ControllerRevision (30m0s) from k8s.io/metacontroller/client/generated/informer/externalversions/factory.go:74
  | I0321 03:40:45.058763       1 reflector.go:240] Listing and watching *v1alpha1.ControllerRevision from k8s.io/metacontroller/client/generated/informer/externalversions/factory.go:74
  | I0321 03:40:45.060036       1 metacontroller.go:92] Starting CompositeController metacontroller
  | I0321 03:40:45.060052       1 controller.go:32] Waiting for caches to sync for CompositeController controller
  | I0321 03:40:45.060111       1 metacontroller.go:85] Starting DecoratorController metacontroller
  | I0321 03:40:45.060130       1 reflector.go:202] Starting reflector *v1alpha1.DecoratorController (30m0s) from k8s.io/metacontroller/client/generated/informer/externalversions/factory.go:74
  | I0321 03:40:45.060141       1 reflector.go:240] Listing and watching *v1alpha1.DecoratorController from k8s.io/metacontroller/client/generated/informer/externalversions/factory.go:74
  | I0321 03:40:45.060141       1 controller.go:32] Waiting for caches to sync for DecoratorController controller
  | I0321 03:40:45.160202       1 shared_informer.go:123] caches populated
  | I0321 03:40:45.160225       1 controller.go:39] Caches are synced for CompositeController controller
  | I0321 03:40:45.160328       1 shared_informer.go:123] caches populated
  | I0321 03:40:45.160337       1 controller.go:39] Caches are synced for DecoratorController controller
  | I0321 03:40:45.160371       1 metacontroller.go:141] sync DecoratorController microsegmentation
  | I0321 03:40:45.160582       1 factory.go:104] Starting shared informer for services in v1
  | I0321 03:40:45.160661       1 asm_amd64.s:573] Shutting down DecoratorController metacontroller
  | E0321 03:40:45.160733       1 runtime.go:66] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
  | /usr/local/google/home/enisoc/k8s/src/k8s.io/metacontroller/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
  | /usr/local/google/home/enisoc/k8s/src/k8s.io/metacontroller/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
  | /usr/local/google/home/enisoc/k8s/src/k8s.io/metacontroller/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
  | /usr/local/google/home/enisoc/.gvm/gos/go1.10/src/runtime/asm_amd64.s:573
  | /usr/local/google/home/enisoc/.gvm/gos/go1.10/src/runtime/panic.go:505
  | /usr/local/google/home/enisoc/.gvm/gos/go1.10/src/runtime/panic.go:63
  | /usr/local/google/home/enisoc/.gvm/gos/go1.10/src/runtime/signal_unix.go:388
  | /usr/local/google/home/enisoc/k8s/src/k8s.io/metacontroller/controller/decorator/controller.go:106
  | /usr/local/google/home/enisoc/k8s/src/k8s.io/metacontroller/controller/decorator/controller.go:126
  | /usr/local/google/home/enisoc/k8s/src/k8s.io/metacontroller/controller/decorator/metacontroller.go:171
  | /usr/local/google/home/enisoc/k8s/src/k8s.io/metacontroller/controller/decorator/metacontroller.go:156
  | /usr/local/google/home/enisoc/k8s/src/k8s.io/metacontroller/controller/decorator/metacontroller.go:124
  | /usr/local/google/home/enisoc/k8s/src/k8s.io/metacontroller/controller/decorator/metacontroller.go:94
  | /usr/local/google/home/enisoc/.gvm/gos/go1.10/src/runtime/asm_amd64.s:2361

my controller definition is the following:

kind: DecoratorController
metadata:
  name: microsegmentation
spec:
  resources:
  - apiVersion: v1
    resource: services
    annotationSelector:
      matchExpressions:
      - {key: io.raffa.microsegmentation, operator: In, values: ['true']}
  attachments:
  - apiVersion: v1beta1
    resource: networkpolicies
  hooks:
    sync:
      webhook:
        url: http://microsegmentation.metacontroller:8080/microsegmentation

CompositeController shouldn't allow empty parent selector.

A key feature of CompositeController is that it tracks which objects of a given type are yours, and hides from you any objects of that type that you don't own. This mechanism is defeated if the parent selector is empty, since there's no way to distinguish things you want from things you don't.

There might be legitimate use cases for a controller that selects everything of a given type, but I don't think it should be allowed by default in CompositeController, since more often than not an empty selector is unintentional. For example, this can happen if the parent object does not have the expected, duck-typed spec.selector (either because it's missing, or it's the old-style selector like in RC or Service).

compositecontroller and decoratorcontroller cannot be namespace scoped

If a namespace is provided it is ignored. Should controllers check to see if what they're watching is namespaced and limit their watches accordingly?

Metacontroller Roadmap (2018)

This is a tentative outline of future plans for Metacontroller.

See the project board for the status of individual tasks. The backlog there is split into three columns:

• Proposed: Not fleshed out enough to implement yet. Needs discussion.
• Blocked: We know what to do, but it has to wait for something external to happen.
• Ready: We're ready to do this, but no one has picked it up yet.

Q1 2018

• Add example to demonstrate building an Operator with Metacontroller.
  • Vitess Operator example.
• Work towards Alpha release.
  • Remove warning about running in real clusters.
    • Cache discovery info.
    • Use watches (informers) for Metacontroller API objects.
    • Use watches (informers) for arbitrary objects.
    • Parallelize controllers hosted by Metacontroller.
  • Internal refactors.
    • Break up main package, restructure to support multiple controller patterns.
    • Generate a client and informer set for Metacontroller APIs.

Q2 2018

• Alpha release.
  • Document alpha API for k8s objects (e.g. CompositeController) and corresponding webhook objects (Request/Response).
  • Document how to debug controllers when developing with Metacontroller.
  • Document best practices for developing controllers with Metacontroller.
  • Smoke tests.
• Work towards Beta release.
  • Emit events on controller parent objects.
  • Use leader election to prevent fighting among multiple Metacontroller instances (optionally enable HA).
  • Full regression test suite (unit, integration, e2e).
• Support more controller patterns (in addition to CompositeController).

Q3 2018

• Beta release.
  • API is a candidate for GA, although there could still be breaking changes if necessary.
  • Scalability benchmarks.
    • Consider sharding if significant bottlenecks are found.
  • Support mutual authentication of webhook requests.
  • Support least privilege (e.g. ServiceAccount per controller).

Q4 2018

• 1.0 release.
  • API will be backward compatible in future 1.x.y releases.
  • Clean up dependencies (share code instead of copying from k8s/k8s).
  • Support installing/updating through k8s add-on manager.

Generate a client and informer for Metacontroller APIs

We need to use the dynamic client for the objects that Lambda Controllers manage, since we allow them to manage arbitrary objects.

However, there are advantages to using a generated client to access objects that represent the Metacontroller APIs themselves (e.g. CompositeController).

This blog may be useful.

Design pattern to reconcile external resources

the initial controller patterns we support are focused on use cases where both the inputs and outputs of your controller can be expressed as Kubernetes API objects (either built-in or custom). If you have a concrete use case that involves reconciling external state, we’d appreciate if you file an issue describing it so we can work on defining additional patterns.

Hello, we're trying to use Metacontroller for a custom use case. We want to aggregate multiple Kubernetes clusters (two for production and multiple for development), without going into complexity of the federation feature. The basic idea is to use a tiny Kubernetes control plane as aggregation layer in front of the other clusters. We set an API Server with its own local etcd and a custom CRD resource modelling the cluster we want to manage. Here an example:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  # name must match the spec fields below, and be in the form: <plural>.<group>
  name: clusters.example.com
spec:
  scope: Namespaced
  group: example.com
  versions:
    - name: v1alpha2
      served: true
      storage: true      
  names:
    kind: Cluster
    singular: cluster
    plural: clusters
    shortNames:
    - cl
    - cls
  subresources:
    status:
  additionalPrinterColumns:
    - name: address
      type: string
      description: the cluster address, e.g. "35.128.9.208"
      JSONPath: .spec.address
    - name: port
      type: string
      description: the cluster listening port, e.g. 8443
      JSONPath: .spec.port
    - name: secure
      type: boolean
      description: secure communication with the remote cluster, e.g. false, true
      JSONPath: .spec.secure
    - name: admin
      type: string
      description: the administrative state of the cluster, e.g. true, false
      JSONPath: .spec.admin
    - name: state
      type: string
      description: the operational state of the cluster, e.g. "working", "maintenance", "unreachable"
      JSONPath: .status.state
    - name: Age
      type: date
      description: Creation timestamp
      JSONPath: .metadata.creationTimestamp
...

Children resources of the cluster CRD are: a Pod running a small agent to pull summary data from the managed cluster and checking the liveness of it, an headless Service specifying where to reach the managed cluster (address and port) and an Ingress required to instruct the ingress controller to proxy requests towards the managed cluster.

We set the Metacontroller and wrote a webhook to reconcile the status of the cluster when it is added/removed/put-in-maintenance by the sys admin. However, we still need how to reconcile the status when it becomes unreachable, because e.g., of the network issues or become unmanageable, because e.g., of token expiring, etc..

These type of events are not under the control of the managing API server and also using Metacontroller polling we cannot intercept them. A possible solution we're exploring is to pass these events from the agent Pod (it's able to intercept them, it's its own job) to the managing API server but we're not sure if possible. Any idea?

Thanks

Leader election

By @enisoc :

FYI there is an alpha API in client-go for leader election:
https://godoc.org/k8s.io/client-go/tools/leaderelection
We could consider using this if/when it makes sense to support HA.

Parallelize Lambda Controllers

Currently the PoC goes through each Lambda Controller one at a time. We should use some amount of concurrency.

Feature request to include some metadata on sync hook requests

It would be interesting to know which sync requests were made because of a refresh sync interval. Perhaps this could be a flag on the request.

Thinking more generally, would it be interesting in expanding this to be an enum indicating if the request is due to parent resource changing, or a child resource changing, or a refresh interval?

document the webhook contract between the metacontroller and the controller

Hi,

I'd like to experiment with metacontroller. I understand the general archietcture, I think, but I can't figure out the contract between the metactroller and the controller when the webhook is triggered.
I have looked at the petset controller and jobindexer controler, but not being familiar with those languages, they weren't very useful.

Metacontroller logo

Hi I think this project needs a beautiful logo. I can contribute project by designing a logo...

support updating status of resources managed by other controllers

Thanks @enisoc for your efforts. This makes writing controller much easier :)

I am trying to write a controller to create a load balancer service in external load balancer when a service is created in the cluster with type: LoadBalancer. I also need to create a corresponding NetworkPolicyto allow traffic from LB nodes to worker nodes for the given service. Looking from the document DecoratorController seems like a close fit (with an issue) as CompositeController assumes ownership of parentresource. In my current use case parentresource is Service.v1.

DecoratorController creates the desired NetworkPolicy object but does not allow me to update the Statusfield of the service which is target resource as defined below.

apiVersion: metacontroller.k8s.io/v1alpha1
kind: DecoratorController
metadata:
  name: lb
spec:
  resources:
  - apiVersion: v1
    resource: services
  attachments:
  - apiVersion: networking.k8s.io/v1
    resource: networkpolicies
  hooks:
    sync:
      webhook:
        url: http://service-per-pod.metacontroller/sync-lb

I need to update the Status field with info with newly created Loadbalancer IP and hostname, so users can see the IP address they can use to reach their service. I have created an example using jsonnet which in gist show I would like to use.

So wondering is there any plan to add support for this feature in DecoratorController or create any other controller which can let me do what I need in this use case.

Document RBAC permissions for Metacontroller

Hello, thanks for sharing this awesome tool!
Is there any reason for running the metacontroller in a dedicated namespace?
I see these RBAC permissions

kind: ClusterRole
metadata:
  name: metacontroller
rules:
- apiGroups:
  - "*"
  resources:
  - "*"
  verbs:
  - "*"

Is it possible to run the metacontroller in kube-system namespace? Which minimum permissions it needs to work?

Thanks.

[Feature Request] - User configurable webhook timeout.

I recently discovered this project and it looks really great!

Context

I was considering an application for which metacontroler would fit perfectly. A concern I have is that as part of my controller, I would need to make a couple REST calls to an external API to resolve certain conditions at run time. I learned from the source code that the hook calls have a hard-coded deadline of 10 seconds, which is quite limiting.

Is there a reason for this particular value (10s)? I assume it shouldn't be a big issue to rise this limit or make it user-defined if the hook calls are being called asynchronously under the hood (and assuming the hook handler is replicated).

Additional question: How would metacontroler handle error conditions on the hook (e.g. return status !200 or timeouts)? Will the call be skipped, retried, etc?

Request:

Add an option for the webhook configuration similar to:

spec:
  hooks:
    sync:
      webhook:
        url: http://hello-controller.hello/sync
        timeoutSeconds: 60               # Timeout in seconds for the webhook 

Notification of related objects

One of the things I've been bumping up against when trying to design a controller is incorporating the state of related objects.

For context - we have many environments, and are trying to have a strict separation between the application bundle of manifests, which is common to all (and can be released/rolled out etc), and the site specific manifests, which are mostly ConfigMaps / Secrets.

One of the things I'm thinking about how to do is to write a controller that can use ConfigMaps.
A concrete example would be designing a new object incorporating something akin to an EnvVar array. The desired outcome would be that a CompositeController can indicate in some fashion that it's interested in the state of some related objects, and be called whenever those change as well.

x509: failed to load system roots and no roots provided

I've created a simple decorator with an external (not a k8s service) webhook. I'm getting the x509 error.

Potential solution: mount /etc/ssl/certs in metacontroller container.

...
        volumeMounts:
        - mountPath: /etc/ssl/certs
          name: ca-certificates
          readOnly: true
    volumes:
    - name: ca-certificates
       hostPath:
          path: /etc/ssl/certs

Emit events from Metacontroller

As Metacontroller attempts to act on behalf of a CompositeController, for example, it should emit events on the parent object as appropriate. We should look at what events built-in controllers emit as a starting point.

Only allow one Metacontroller instance at a time during updates

We should probably only let one Metacontroller instance be active at a time for now. In the PoC, a single instance hosts Lambda Controllers for the whole cluster.

Elaborate Contributing Section of Doc

We can add explicit tabs to Contributing section for accommodating pages such as this.

Support status subresource and ObservedGeneration.

Now that we have support for /status in both CRD and the dynamic client, we should use /status whenever it's available.

As of k8s 1.10, /status for CRD is behind an alpha feature gate, and is also optional on a per-CRD basis. If we neglect to use /status when it's available, our status update will be silently ignored. If we try to use /status when it's not available, our status update will fail. We should be able to tell whether /status is available for a given resource by looking at discovery, without requiring the user to configure anything on the Metacontroller side (they'll just enable /status in the CRD itself).

We should also set status.observedGeneration now that CRD is capable of incrementing metadata.generation (if the /status subresource is enabled for that CRD).

https://github.com/kstmp/metacontroller/blob/8c2b6730d8db89c5eba31bf180ebd9fbe93b3a5c/controller/composite/controller.go#L256-L271

Lastly, for controllers that support ObservedGeneration, we can consider an optimization to avoid syncing if the controller object is updated, but the spec hasn't changed (ObservedGeneration = Generation). For example, we currently might trigger a sync as a result of updating our own status.

Feature Request: add support for adding/removing finalizer

The decorator controller can only add/remove labels/annotations. I have a usecase where I need to add/remove finalizers. The decorator controller seems the right place to do this.

Installation instructions end up using an old build of metacontroller

The installation documentation references the manifest file which references an older docker image.

https://github.com/GoogleCloudPlatform/metacontroller/blob/master/manifests/metacontroller.yaml#L66

References metacontroller/metacontroller:0.2. Should it reference latest?

How to design an operator that creates a kubernetes resource using other kubernetes resources

What is the possible design I should follow for my use case detailed here:

Scenario:

When a PVC is applied, I need to check for the existence of a Custom Resource
Then I need to create a Kubernetes Service
If above Service & Custom Resource are available
Then I need to extract the cluster IP from Service & a property from Custom Resource
Then I need to create a Deployment using above cluster IP & property values
Finally, I want to create a PV using the above cluster IP

Scenario:

When the PVC is deleted, I want to delete Service, Deployment & PV objects

Cache API discovery info

We should probably use a cached discovery interface and arrange for periodic invalidation. Currently the PoC hits discovery on every iteration.

Switch to new dynamic client

The old dynamic client is deprecated: kubernetes/kubernetes#63446

This should probably wait until after 1.11 is released, so we know the new dynamic client is finalized.

Feature request for supporting side-effects on add/delete

I don't believe there is a clean way right now to do one-off/side-effecting (and often off cluster work) in response to resource objects being created or deleted.

It would be good to be able to model those explicitly and don't know if this has already been considered in the past - but having some optional hooks for adds/deletes alongside sync could be interesting.

Set the GitHub website url to https://metacontroller.app/

Its really convenient to have a website on the github title line for a repository. I personally look for a link there first, I'm sure some others do the same thing. It's more of a NIT than anything else and I would do it myself but I'm not a maintainer.

Unable to run `make deepcopy`

I'm unable to run make because the deepcopy target is failing.

rssguar@cloudshell:~/gopath/src/k8s.io/metacontroller$ dep ensure --vendor-only
rssguar@cloudshell:~/gopath/src/k8s.io/metacontroller$ git s
rssguar@cloudshell:~/gopath/src/k8s.io/metacontroller$ make
+ Generating deepcopy funcs for metacontroller/v1alpha1
ERROR: logging before flag.Parse: F0822 17:04:10.405059     337 deepcopy.go:126] Failed loading boilerplate: open /home/rssguar/gopath/src/k8s.io/kubernetes/hack/boilerplate/boilerplate.go.txt: no such file or directory
goroutine 1 [running]:
k8s.io/metacontroller/vendor/github.com/golang/glog.stacks(0xc42000e000, 0xc42963dd40, 0xba, 0x10a)
        /home/rssguar/gopath/src/k8s.io/metacontroller/vendor/github.com/golang/glog/glog.go:769 +0xcf
k8s.io/metacontroller/vendor/github.com/golang/glog.(*loggingT).output(0x8d5fa0, 0xc400000003, 0xc429726a50, 0x8b5122, 0xb, 0x7e, 0x0)
        /home/rssguar/gopath/src/k8s.io/metacontroller/vendor/github.com/golang/glog/glog.go:720 +0x32d
k8s.io/metacontroller/vendor/github.com/golang/glog.(*loggingT).printf(0x8d5fa0, 0x3, 0x712bad, 0x1e, 0xc4270cfcf8, 0x1, 0x1)
        /home/rssguar/gopath/src/k8s.io/metacontroller/vendor/github.com/golang/glog/glog.go:655 +0x14b
k8s.io/metacontroller/vendor/github.com/golang/glog.Fatalf(0x712bad, 0x1e, 0xc4270cfcf8, 0x1, 0x1)
        /home/rssguar/gopath/src/k8s.io/metacontroller/vendor/github.com/golang/glog/glog.go:1148 +0x67
k8s.io/metacontroller/vendor/k8s.io/gengo/examples/deepcopy-gen/generators.Packages(0xc4296e8d80, 0xc4200a6960, 0x6e1f14, 0x6, 0xc4296e8d80)
        /home/rssguar/gopath/src/k8s.io/metacontroller/vendor/k8s.io/gengo/examples/deepcopy-gen/generators/deepcopy.go:126 +0xc5
k8s.io/metacontroller/vendor/k8s.io/gengo/args.(*GeneratorArgs).Execute(0xc4200a6960, 0xc420074db0, 0x6e1f14, 0x6, 0x71e060, 0x0, 0x0)
        /home/rssguar/gopath/src/k8s.io/metacontroller/vendor/k8s.io/gengo/args/args.go:193 +0x1b8
main.main()
        /home/rssguar/gopath/src/k8s.io/metacontroller/vendor/k8s.io/code-generator/cmd/deepcopy-gen/main.go:69 +0x1c4
Makefile:28: recipe for target 'deepcopy' failed
make: *** [deepcopy] Error 255
rssguar@cloudshell:~/gopath/src/k8s.io/metacontroller$ git show
commit 2a992e2bb526749a6f10cad9b570fbe0510599f5
Merge: 98b9014 8922193
Author: Anthony Yeh <[email protected]>
Date:   Mon Jul 23 15:05:26 2018 -0700

    Merge pull request #66 from crimsonfaith91/one-mc

    only allow one Metacontroller instance to be active at a time

rssguar@cloudshell:~/gopath/src/k8s.io/metacontroller$ git status

Is this just me doing something strange? I was going to take a look at #47 but I can't confirm I'm in a working state to being with. I noticed we never actually run make in CI so it might not work anymore.

Just like statefulSet, is there a way to ping each other using hostname in the pods launched by indexedjob?

I tried headless service, but it seems not working well.

$ cat my-indexedjob.yaml 
apiVersion: v1
kind: Service
metadata:
  name: default-subdomain
spec:
  selector:
    app: print-index
  clusterIP: None
  ports:
  - name: foo
    port: 1234
    targetPort: 1234
---
apiVersion: ctl.enisoc.com/v1
kind: IndexedJob
metadata:
  name: print-index
spec:
  completions: 3
  parallelism: 3
  selector:
     matchLabels:
       app: print-index
  template:
    metadata:
      labels:
        app: print-index
    spec:
      restartPolicy: Never
      subdomain: default-subdomain
      containers:
      - name: print-index
        image: busybox
        command: ["sh", "-c", "sleep 10000"]

$ kubectl exec print-index-0 -- cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local us-west-2.compute.internal
options ndots:5

$ kubectl exec print-index-0 -- nslookup print-index-0
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      print-index-0
Address 1: 10.244.2.54 print-index-0.default-subdomain.default.svc.cluster.local


$ kubectl exec print-index-0 -- nslookup print-index-0.default-subdomain
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

nslookup: can't resolve 'print-index-0.default-subdomain'
command terminated with exit code 1



$ kubectl exec print-index-0 -- nslookup print-index-1
nslookup: can't resolve 'print-index-1'
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

command terminated with exit code 1


$ kubectl exec print-index-0 -- nslookup print-index-1.default-subdomain.default.svc.cluster.local
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

nslookup: can't resolve 'print-index-1.default-subdomain.default.svc.cluster.local'
command terminated with exit code 1

More in-depth testing

Existing smoke tests only cover limited test paths for meta-controllers. For example, the tests do not cover whether a resource is namespaced or not. As the project becomes larger, perhaps it is time for us to write more in-depth tests like e2e and integration tests.

@enisoc

Upgrade kubernetes-incubator/service-catalog to 0.1.10

Since they are moving out of alpha quick, it's best to keep in sync with them

Double webhook calls with same state, succeeded objects disappearing (CompositeController)

I'm experimenting with writing a job pipeline controller which delays launching jobs until their specified dependencies have succeeded. I started from examples/indexedjob and created examples/jobtree based on it.

I see a strange behavior where my webhook is sometimes called twice with the same controller status but different observed child pod phases. In some of those cases, a pod which was previously Running is first shown as Succeeded in the first call, and then completely missing in the second call. I've added an incrementing counter in the controller status, and this reveals the double calls in the log:

• counter: 5
time: 2018-09-22T19:18:18
phases from previous call: {a: Running, b: Pending, c: Pending}
current pod phases: {a: Running, b: Running, c: Running}
phases combined: {a: Running, b: Running, c: Running}
desired jobs: [a, b, c]
• counter: 6
time: 2018-09-22T19:18:25
phases from previous call: {a: Running, b: Running, c: Running}
current pod phases: {a: Succeeded, b: Running, c: Running}
phases combined: {a: Succeeded, b: Running, c: Running}
desired jobs: [b, c]
• counter: 6
time: 2018-09-22T19:18:25
phases from previous call: {a: Running, b: Running, c: Running}
current pod phases: {b: Running, c: Running} (pod 'a' disappeared)
  phases combined: {a: Running, b: Running, c: Running}
desired jobs: [a, b, c]

To make the controller work correctly, I'd have to interpret a dissapeared pod as Succeeded. Otherwise it is re-inserted into desired jobs and started again.

I believe I've understood and followed the advice in Best Practices about "Side Effects" and "Status" by not including the desired pods in the controller status returned by the webhook. To me it seems that the second call is not a "hypothetical" proble from Metacontroller – it is acting on the desired state because I'm seeing some jobs run twice.

(Note that in my code base linked to above, I've worked around this issue by recording the phase of a disappearing running pod as Disappeared and interpreting is as Succeeded when calculating the desired jobs. So running the my-jobtree.yaml example will work around this problem by not re-inserting the disappeared job into the list of desired pods.)

I wonder if this behavior of Metacontroller is intentional or if I've misunderstood something as a novice Kubernetes user.

CompositeController should validate that child objects match the parent selector.

We should fail creation/update of child objects that don't match, because otherwise they'll be immediately orphaned.

There might be legitimate use cases for a controller that intentionally orphans objects, but I don't think it should be allowed by default when using CompositeController, since that's contrary to the intended pattern: the parent should represent a "composite" object that's made up of child objects.

Can’t label on node with DecoratorController

When i use DecoratorController for labeling node, nothing happend.

$ kubectl logs metacontroller-57dfc8cffc-jcx2t
round_trippers.go] GET https://172.30.0.1:443/apis/v1 200 OK in 1 milliseconds
controller.go:391] Node /10.0.0.1 has been deleted
controller.go:391] Node /10.0.0.2 has been deleted
controller.go:391] Node /10.0.0.3 has been deleted

DecoratorContoller can’t find parent object such as node
because the node doesn’t have a namespace.

Use shared informers

We need to maintain a map of SharedInformers, one for each type of resource. Unlike most controllers, we'll want to make each informer use the dynamic client and work on objects of type Unstructured. This way, we can generically handle any arbitrary objects.

DecoratorController can't decorate a Namespace or add resources to that Namespace

I add a DecoratorController that watches a Namespace and wants to create a RoleBinding in that new Namespace

From the log

I0831 18:44:37.219130       1 metacontroller.go:141] sync DecoratorController namespace-controller
I0831 18:44:37.219258       1 factory.go:69] Subscribed to shared informer for namespaces in v1 (total subscribers now 2)
I0831 18:44:37.219454       1 factory.go:104] Starting shared informer for rolebindings in rbac.authorization.k8s.io/v1
I0831 18:44:37.219572       1 controller.go:174] Starting DecoratorController namespace-controller
I0831 18:44:37.219582       1 controller.go:178] Waiting for DecoratorController namespace-controller caches to sync
I0831 18:44:37.219589       1 controller.go:32] Waiting for caches to sync for namespace-controller controller
I0831 18:44:37.219852       1 reflector.go:202] Starting reflector *unstructured.Unstructured (30m0s) from k8s.io/metacontroller/dynamic/informer/factory.go:111
I0831 18:44:37.219887       1 reflector.go:240] Listing and watching *unstructured.Unstructured from k8s.io/metacontroller/dynamic/informer/factory.go:111
I0831 18:44:37.319860       1 shared_informer.go:122] caches populated
I0831 18:44:37.319910       1 controller.go:39] Caches are synced for namespace-controller controller
I0831 18:44:37.320048       1 controller.go:406] DecoratorController namespace-controller: sync Namespace /herald
E0831 18:44:47.321005       1 controller.go:236] failed to sync namespace-controller "v1:Namespace::herald": sync hook failed: http error: Post http://isolation-decorator.isolation-admin/sync-namespaces: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I0831 18:44:47.326427       1 controller.go:406] DecoratorController namespace-controller: sync Namespace /herald
I0831 18:44:47.336815       1 controller.go:450] DecoratorController namespace-controller: updating Namespace /herald
I0831 18:44:47.348246       1 manage_children.go:183] Namespace /herald: creating RoleBinding herald
E0831 18:44:47.348465       1 controller.go:236] failed to sync namespace-controller "v1:Namespace::herald": can't reconcile children for Namespace /herald: an empty namespace may not be set during creation
I0831 18:44:47.348520       1 controller.go:406] DecoratorController namespace-controller: sync Namespace /herald

Here is the manifest of the decorator
kubectl get decoratorcontrollers.metacontroller.k8s.io namespace-controller -oyaml

apiVersion: metacontroller.k8s.io/v1alpha1
kind: DecoratorController
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"metacontroller.k8s.io/v1alpha1","kind":"DecoratorController","metadata":{"annotations":{},"name":"namespace-controller","namespace":""},"spec":{"attachments":[{"apiVersion":"rbac.authorization.k8s.io/v1","resource":"rolebindings"}],"hooks":{"sync":{"webhook":{"url":"http://isolation-decorator.isolation-admin/sync-namespaces"}}},"resources":[{"apiVersion":"v1","labelSelector":{"matchLabels":{"app":"herald"}},"resource":"namespaces"}]}}
  clusterName: ""
  creationTimestamp: 2018-08-31T18:44:37Z
  generation: 1
  name: namespace-controller
  namespace: ""
  resourceVersion: "827176"
  selfLink: /apis/metacontroller.k8s.io/v1alpha1/decoratorcontrollers/namespace-controller
  uid: e93f63d4-ad4d-11e8-9dbd-42010a8a014e
spec:
  attachments:
  - apiVersion: rbac.authorization.k8s.io/v1
    resource: rolebindings
  hooks:
    sync:
      webhook:
        url: http://isolation-decorator.isolation-admin/sync-namespaces
  resources:
  - apiVersion: v1
    labelSelector:
      matchLabels:
        app: herald
    resource: namespaces

Worth noting is that the Namespace is created by a CompositeController. So the owner of the namespace is the composite controller. But based on looking at the decorator controller source, it looks like there is an incorrect assumption that the Namespace has a namespace field. This results in weird errors messages like "v1:Namespace::herald": can't reconcile children for Namespace /herald: an empty namespace may not be set during creation. The latter part of the error comes from
vendor/k8s.io/client-go/rest/request.go, the odd looking name "/herald" which looks like I'm trying to create a Namespace with a leading slash comes from this line

can't find resource in apiVersion ...

hi I was able to run my controller based, on metacotroller but recently it broke with the below errors from the metacontroller logs:

E0221 03:36:18.827859       1 controller.go:133] can't sync Service prometheus/grafana: discovery: can't find resource  in apiVersion networking.k8s.io/v1
E0221 03:36:18.827867       1 controller.go:133] can't sync Service prometheus/kube-state-metrics: discovery: can't find resource  in apiVersion networking.k8s.io/v1
E0221 03:36:18.827877       1 controller.go:133] can't sync Service prometheus/prometheus: discovery: can't find resource  in apiVersion networking.k8s.io/v1

the metacontroller is looking for network policies. My controller CRD is as follows

apiVersion: metacontroller.k8s.io/v1alpha1
kind: CompositeController
metadata:
  name: microsegmentation-controller
spec:
  parentResource:
    apiVersion: v1
    resource: services  
  childResources:
    - apiVersion: extensions/v1beta1
      resources: ["networkpolicies"]
#    - apiVersion: networking.k8s.io/v1
#      resources: ["networkpolicies"]      
  clientConfig:
    service:
      name: microsegmentation-controller
      namespace: metacontroller
  hooks:
    sync:
      path: /sync

I am running in openshift 3.7 corresponding to kube 1.7 so I thought that networkpolicies might still be in beta (even though I saw it working) so I switched to extensions/v1beta1 but still the same error:

E0221 03:36:27.084535       1 controller.go:133] can't sync Service default/docker-registry: discovery: can't find resource  in apiVersion extensions/v1beta1
E0221 03:36:27.084571       1 controller.go:133] can't sync Service default/kubernetes: discovery: can't find resource  in apiVersion extensions/v1beta1
E0221 03:36:27.084616       1 controller.go:133] can't sync Service default/registry-console: discovery: can't find resource  in apiVersion extensions/v1beta1

not sure what is happening but I suspect some changes in a recently published image, can you confirm?

Execute custom cleanup code on deletion with finalizers

The use case is what I have discussed with you @enisoc in #58. We (@folago and me) are trying to implement a LB controller based on Service resource using DecoratorController. We create an external LB when a new service is created and pass the info using annotations on the Service resource. But when the Service resource gets deleted, we need to have this info passed to our controller, so we can delete the LB entry for this service as well. As the code stands now, it is only possible to delete the objects in the cluster using ownerreference, but objects created outside the cluster is not possible to clean up. It would be good to have this as a configurable option in the controller spec and default disabled, if preferred, to preserve the current behavior.

Use of EventRecorder?

Fantastic piece of software you guys have written! I’m actually considering using this to implement a custom operator instead of writing it from scratch in go for my masters thesis!

Have you considered implementing a way of using the EventRecorder from a webhook? That way, a user could log custom events to the crd. It could be as simple as adding a new field from the output and run the emit function on whatever you return back I suppose.

add the ability to specify the port in the client config

and potentially also the protocol as the communication may need to be over https.