Feature request to include some metadata on sync hook requests about metacontroller

[enisoc]

Similar to my comment on #69 (comment), my first impression is that knowing why the sync hook was called would be a Kubernetes controller anti-pattern.

Can you give a concrete example of how this information would be used?

[nikhilk]

Here is a concrete example -- I want to use resync to launch another job (and quite possibly outside of the k8s cluster, eg. a Dataflow job) on a timer/cron.

This is directly along the lines of what the doc at https://metacontroller.app/api/compositecontroller/ says:

Sometimes you may want to sync periodically even if nothing has changed in the Kubernetes API objects, either to simply observe the passage of time, or because your hook takes external state into account. For example, CronJob uses a periodic resync to check whether it’s time to start a new Job.

Now its possible my controller creates a cron job as one of its child resources, rather than itself doing the scheduled work, but ultimately this only defers the problem to another controller in the system - something needs to know whether its being invoked because some config/spec changed, because some child changed, or because time passed.

Or am I missing something?

[enisoc]

For context, the CronJob controller itself is implemented as a single sync function that doesn't know why it's being invoked. The idea of Metacontroller's periodic resync was that it enables hook authors to use the same strategy that CronJob does, which is upon each sync:

• Compute the set of "activations" you expect to have happened within some time window ending at "now" (desired state of the world).
• Look at the set of Jobs previously created (actual state of the world).
• Try to match up Jobs with the desired activation slots.
• If any desired slots are unfilled, create new Jobs to try to fill them.

CronJob is designed in this way to be resilient to things like controller downtime. Suppose you want to spawn a Job once a day at 4am. If the CronJob controller happens to go down at 3:59 and come back up at 4:01, you want it to trigger as soon as it comes back, not ~24hrs later.

You have a good point that it might be worth adding a feature that makes cron-like behavior easier to implement, but I don't think we want people to rely on resyncPeriod alone. The firing of periodic resyncs is not based on wall clock time -- the timer may reset at various times, for example if you edit the CompositeController spec or restart Metacontroller to update the server or underlying Node. The resyncPeriod is intended only to let you set an approximate lower bound on sampling frequency -- the sync operation is still conceptually polling (not event-based) as prescribed by the Kubernetes level-triggered controller model.

I would expect many controllers that need to do things periodically could be even simpler than CronJob. For example, you might want a policy like, "run this Job if it hasn't run in the last 2 hours." The sync logic for that would look like:

• Look at when the latest Job ran. Suppose that's job-45 (the 45th invocation).
• If it ran more than 2 hours ago, start a new Job called job-46 (n+1).

This is a simple level-triggered pattern for doing something periodically according to wall clock time. The numbered naming is important to ensure triggering is idempotent -- your hook might be called again before the result of the previous hook is observed, because Kubernetes controller work queues are only eventually consistent due to caching.

We could perhaps make this even easier by doing some bookkeeping for you, like remembering your previous invocations. But then there could be inconsistencies between our records and the actual source of truth -- either something like Job objects in the API server, or entries in an external DB. I would rather have the hook code check the source of truth directly, which is something we usually can't do for you generically.

[nikhilk]

Thanks for the writeup, and taking the time to explain in detail!
It suggests some different ways to think about the scenario, and will do so/try implementing it along those lines.