flant / shell-operator

Shell-operator is a tool for running event-driven scripts in a Kubernetes cluster

Shell-operator is a tool for running event-driven scripts in a Kubernetes cluster.

This operator is not an operator for a particular software product such as prometheus-operator or kafka-operator. Shell-operator provides an integration layer between Kubernetes cluster events and shell scripts by treating scripts as hooks triggered by events. Think of it as an operator-sdk but for scripts.

Shell-operator is used as a base for more advanced addon-operator that supports Helm charts and value storages.

Shell-operator provides:

• Ease of management of a Kubernetes cluster: use the tools that Ops are familiar with. It can be bash, python, kubectl, etc.
• Kubernetes object events: hook can be triggered by add, update or delete events. Learn more about hooks.
• Object selector and properties filter: shell-operator can monitor a particular set of objects and detect changes in their properties.
• Simple configuration: hook binding definition is a JSON or YAML document on script's stdout.
• Validating webhook machinery: hook can handle validating for Kubernetes resources.
• Conversion webhook machinery: hook can handle version conversion for Kubernetes resources.

Documentation

Please see the docs for more in-depth information and supported features.

Examples and notable users

More examples of how you can use shell-operator are available in the examples directory.

Prominent shell-operator use cases include:

• Deckhouse Kubernetes platform where both projects, shell-operator and addon-operator, are used as the core technology to configure & extend K8s features;
• KubeSphere Kubernetes platform's installer;
• Kafka DevOps solution from Confluent.

Please find out & share more examples in Show & tell discussions.

Articles & talks

Shell-operator has been presented during KubeCon + CloudNativeCon Europe 2020 Virtual (Aug'20). Here is the talk called "Go? Bash! Meet the shell-operator":

• YouTube video;
• text summary;
• slides.

Official publications on shell-operator:

• "shell-operator v1.0.0: the long-awaited release of our tool to create Kubernetes operators" (Apr'21);
• "shell-operator & addon-operator news: hooks as admission webhooks, Helm 3, OpenAPI, Go hooks, and more!" (Feb'21);
• "Kubernetes operators made easy with shell-operator: project status & news" (Jul'20);
• "Announcing shell-operator to simplify creating of Kubernetes operators" (May'19).

Other languages:

• Chinese: "介绍一个不太小的工具：Shell Operator"; "使用shell-operator实现Operator";
• Dutch: "Een operator om te automatiseren – Hoe pak je dat aan?";
• Russian: "shell-operator v1.0.0: долгожданный релиз нашего проекта для Kubernetes-операторов"; "Представляем shell-operator: создавать операторы для Kubernetes стало ещё проще".

Community

Please feel free to reach developers/maintainers and users via GitHub Discussions for any questions regarding shell-operator.

You're also welcome to follow @flant_com to stay informed about all our Open Source initiatives.

License

Apache License 2.0, see LICENSE.

shell-operator's Issues

Reload hooks list on folder changes

Consistency of reconciliation and onKubernetesEvents

The most natural way is to run initial reconciliation in onStartup and than follow all the changes using onKubernetesEvents hooks. But to do that there should be the guarantee, that:

1. No events will be missed between onStartup and creation of informers for onKubernetesEvents hooks.
2. No onKubernetesEvents hooks will be executed until onStartup is finished. We don't want to follow changes until initial reconciliation is completed. But if we receive the event for the change that already processed during reconciliation stage – that's not a problem, because all the actions in onKubernetesEvents hooks should be idempotent.

To do that, during hooks configuration stage we shall do the following:

1. Read all the hooks configuration.
2. Fill the queue with onStartup hooks (but not run them yet).
3. Create informers (they will add new tasks after onStartup).
4. Start running the queue.

P.S. We shall document described approach as recommended!

Add apiVersion field in onKubernetesEvent configuration

Propose:

• apiVersion is required field for v1 schema
• do not discover resources on informer Run
• discover remains for legacy schema

Create example with CRD

e2e tests

Implement e2e tests.

Dockerhub Releases

Could you release releases a little more often? The latest tag is updated frequently, but the last release was released 3 months ago.
Thanks

MAIN_LOOP error handling Schedule event

Hello. I added the schedule hook:

{
  "configVersion":"v1",
  "schedule": [
    { 
      "name":"Some name",
      "crontab":"0 */1 * * * *"
    }
  ]
}

Evety minutes gets error:

Running schedule manager entry '0 */1 * * * *' ...
EVENT Schedule event '0 */1 * * * *'
MAIN_LOOP error handling Schedule event '0 */1 * * * *': Unknown schedule event: no such crontab '0 */1 * * * *' is registered

My docker image based on flant/shell-operator:latest-alpine3.9

Temporary files should have unique names and be cleaned after hook run

Multi queues run hooks in parallel, so the hook can receive a wrong binding context.

watchEvent - Deleted

If you use only watchEvent - Deleted and kind deployment, then delete events are ignored, if add Added, then everything is ok

docs: find a more elegant form for binding syntax

#133

There should be a way to properly present available syntax options without resorting to EBNF notation or mocking a JSON document...

Adding common functions to the shell framework

Shell operator v1.0.0-beta.7 has introduced a framework to facilitate the creation of shell hooks.
To make it even greater I'd suggest include to the framework some of the common functions that you use throughout the examples, for instance: common/functions.sh.
Thank you!

Hooks configuration API refactoring

We have a few problems in API that should be fixed:

1. First of all – we mistakenly use operation instead of operator in matchExpressions clause. This is a real bug and should be fixed.
2. We use selector, containing matchLabels and matchExpressions fields. But the more Kubernetes-compatible name will be labelSelector. Rename selector to labelSelector.
3. In #32 we are going to add matchLabels and matchExpressions to namespace. Instead of just adding them to namespaceSelector, we better introduce new field namespace, and then add there labelSelector.
4. Fully abandon namespaceSelector, and instead of it namespace.labelSelector (as already suggested above), namespace.nameSelector (having matchNames).
5. #35 and #33 also implement in better way.
6. Introduce v1 (#36)

Full example:

{
  "configVersion": "v1",
  "onKubernetesEvent": [
    {
      "name": "Monitor labeled pods in cache tier",
      "kind": "Pod",
      "event": [ "add", "update", "delete" ],
      "nameSelector": {
        "matchNames": ["pod-xxx", "pod-yyy", "pod-zzz"],
      },
      "labelSelector": {
        "matchLabels": {
          "myLabel": "myLabelValue",
          "someKey": "someValue",
          ...
        },
        "matchExpressions": [
          {
            "key": "tier",
            "operator": "In",
            "values": ["cache"],
          },
          ...
        ],
      },
      "fieldSelector": { 
        "matchExpressions": [
          {
            "field": "status.phase",
            "operator": "Equal",
            "value": "Pending",
          },
          ...
        ],
      },
      "namespace": {
        "nameSelector": {
          "matchNames": ["somenamespace", "proj-production", "proj-stage"],
        },
        "labelSelector": {
          "matchLabels": {
            "myLabel": "myLabelValue",
            "someKey": "someValue",
            ...
          },
          "matchExpressions": [
            {
              "key": "tier",
              "operator": "In",
              "values": ["cache"],
            },
            ...
          ]
        }
      },
      "jqFilter": ".metadata.labels",
      "allowFailure": true|false,
    },
  ]
}

Combine binding contexts to speedup hook execution on high loads

1. If queue contains a sequence of HookRun tasks for one hook, then this hook can be executed with combined binding context.
2. Binding context is already an array, so hook should iterate over it, not just get first element.
3. A pack of binding contexts with similar skipKey can be compacted, so only the latest binding context will remain.

Support metrics publishing for hooks

The idea is that the hook can return a set of JSON objects and the Shell-operator will publish them as metrics.

For example:

Hook writes JSON objects to a temporary file $METRICS_PATH

{"name":"metric_gauge", "set":34.12, "labels":{"label1":"val1", ... }}
{"name":"metric_count", "add":1, "labels":{"label1":"val1", ... }}

Shell-operator parse these objects and publish them via /metrics endpoint:

metric_gauge{"hook":"hook.sh", "label1":"val1"} 34.12
metric_count{"hook":"hook.sh", "label1":"val1"} 1

Note 1: "add" operation is also available for the gauge metric.

Note 2: there are no checks for the uniqueness of the "name" value. Metric name is not prefixed with "shell_operator_".

Note 3: a developer should check, that similar metrics have a similar set of labels or an error will occur and hook will restart.

Note 4: hook label is added automatically.

support for hpa

It would be awesome to add hpa to the kind list

shell-operator/pkg/kube_events_manager/kube_events_manager.go

Line 165 in 45c6bbb

switch strings.ToLower(kind) {

Introduce versioning for hooks configuration API

We shall add version field to the API. And from now on if we need to rename or remove some field or change the format in any other incompatible way – we will trigger a new version. Suggested format (we aim for simplicity):

{
  "configVersion": "v1",      # Just one simple field
  "onStartup": ORDER,
  "schedule": [
    {SCHEDULE_PARAMETERS},
    {SCHEDULE_PARAMETERS}
  ],
  "onKubernetesEvent": [
    {ON_KUBERNETES_EVENT_PARAMETERS},
    {ON_KUBERNETES_EVENT_PARAMETERS}
  ]
}

Some other things to be considered:

1. We can think of the current version (having no number) as of legacy version, and add version v1 when we will need to introduce the first change.
2. Old versions shall be supported for at least one year (this promise should also be documented).
3. Probably we are going to use the simplest possible versioning schema: v1, v2, v3. No alfa and beta. No semver.

Delete hook, endless cycle

After resolving some issues (e.g. broken hook names) I have installed shell operator as Deployment app.

I used the configuration "secret-copier" from examples, but it goes into an endless loop.
Can you help mew with that?

The log is listed below.

2019-12-02T01:34:00Z INFO     : shell-operator master-a10d5213a3fea52c3f426502b79d8cd60adc28ae
2019-12-02T01:34:00Z INFO     : HTTP SERVER Listening on 0.0.0.0:9115
2019-12-02T01:34:00Z INFO     : Working dir: /hooks
2019-12-02T01:34:00Z INFO     : Use temporary dir: /tmp/shell-operator
2019-12-02T01:34:00Z INFO     : Initialize hooks manager. Search for and load all hooks.
2019-12-02T01:34:00Z INFO     : Load hook config from '/hooks/add_or_update_secret'
2019-12-02T01:34:00Z INFO     : Load hook config from '/hooks/create_namespace'
2019-12-02T01:34:00Z INFO     : Load hook config from '/hooks/delete_secret'
2019-12-02T01:34:00Z INFO     : Load hook config from '/hooks/schedule_sync_secret'
2019-12-02T01:34:00Z INFO     : Initializing schedule manager ...
2019-12-02T01:34:00Z INFO     : KUBE Init Kubernetes client
2019-12-02T01:34:00Z INFO     : KUBE-INIT Kubernetes client is configured successfully
2019-12-02T01:34:00Z INFO     : MAIN: run main loop
2019-12-02T01:34:00Z INFO     : QUEUE add all HookRun@OnStartup
2019-12-02T01:34:00Z INFO     : MSTOR Create new metric shell_operator_live_ticks
2019-12-02T01:34:00Z INFO     : MSTOR Create new metric shell_operator_tasks_queue_length
2019-12-02T01:34:00Z INFO     : Start monitors: 3
2019-12-02T01:34:00Z INFO     : Running schedule manager ...
2019-12-02T01:34:00Z INFO     : Resync Event
2019-12-02T01:34:00Z INFO     : Resync Event
2019-12-02T01:34:00Z INFO     : Resync Event
2019-12-02T01:34:00Z INFO     : EVENT Kube event '14adeee3-c4b6-4c4b-b32a-077c68a8c0ea'
2019-12-02T01:34:00Z INFO     : QUEUE add TASK_HOOK_RUN@ON_KUBERNETES_EVENT add_or_update_secret
2019-12-02T01:34:00Z INFO     : EVENT Kube event '26ce9088-d44f-4d9e-86d3-692e3192292a'
2019-12-02T01:34:00Z INFO     : QUEUE add TASK_HOOK_RUN@ON_KUBERNETES_EVENT delete_secret
2019-12-02T01:34:00Z INFO     : EVENT Kube event 'ece3742b-43b9-442b-ad10-b6fffd0ff97b'
2019-12-02T01:34:00Z INFO     : QUEUE add TASK_HOOK_RUN@ON_KUBERNETES_EVENT create_namespace
2019-12-02T01:34:03Z INFO     : TASK_RUN HookRun@ON_KUBERNETES_EVENT add_or_update_secret
2019-12-02T01:34:03Z INFO     : Running hook 'add_or_update_secret' binding 'ON_KUBERNETES_EVENT' ...
Got Synchronization event
2019-12-02T01:34:03Z INFO     : TASK_RUN HookRun@ON_KUBERNETES_EVENT delete_secret
2019-12-02T01:34:03Z INFO     : Running hook 'delete_secret' binding 'ON_KUBERNETES_EVENT' ...
Error from server (NotFound): secrets "null" not found
Error from server (NotFound): secrets "null" not found
Error from server (NotFound): secrets "null" not found
Error from server (NotFound): secrets "null" not found
Error from server (NotFound): secrets "null" not found
Error from server (NotFound): secrets "null" not found
2019-12-02T01:34:05Z INFO     : MSTOR Create new metric shell_operator_hook_errors
2019-12-02T01:34:05Z ERROR    : TASK_RUN TASK_HOOK_RUN 'delete_secret' on 'ON_KUBERNETES_EVENT' failed. Will retry after delay. Failed count is 1. Error: delete_secret FAILED: exit status 1
2019-12-02T01:34:05Z INFO     : TASK_RUN Delay for 5s
2019-12-02T01:34:10Z INFO     : TASK_RUN HookRun@ON_KUBERNETES_EVENT delete_secret
2019-12-02T01:34:10Z INFO     : Running hook 'delete_secret' binding 'ON_KUBERNETES_EVENT' ...
Error from server (NotFound): secrets "null" not found
Error from server (NotFound): secrets "null" not found
Error from server (NotFound): secrets "null" not found
Error from server (NotFound): secrets "null" not found
Error from server (NotFound): secrets "null" not found
Error from server (NotFound): secrets "null" not found

Add example of ConfigMap monitoring

monitored configMaps too.

{ "configVersion":"v1", "kubernetes":[ { "apiVersion": "events.k8s.io/v1beta1", "kind": "Event", "namespace": { "nameSelector": { "matchNames": ["example-monitor-events"] } }, "fieldSelector": { "matchExpressions": [ { "field": "metadata.namespace", "operator": "Equals", "value": "example-monitor-events" } ] } } ] }

monitored only activity of pods

Fix debug process

Part 1

• Remove --debug=yes flag
• Remove disableDebug parameter in onKubernetesEvent binding
• Migrate to logrus library
  • json logs
  • timestamps
  • custom log format

Part 2

• Remove files in /tmp after hook run
• debug over http: binding_context file, hook output for each hook run
• add settings for two endpoints: debug and metrics. DEBUG_LISTEN_ADDRESS, LISTEN_ADDRESS.
  • LISTEN_ADDRESS=:9115 — current setting — listen on 0.0.0.0 on 9115 port
  • LISTEN_ADDRESS=127.0.0.1:9115 — listen on localhost

Update 22.01.2020:

• Part 1 is done mostly in PR #62
• Remove tmp files is done in #100

Namespace wildcard matcher

Namespace can be specified only as exact names or as any. There is a case when you want to monitor pod in namespaces like 'app-name-preview-*'. This can be done now with "kind":"pod", "namespaceSelector": {"any": true} and filtering events in hooks, but this is not declarative. So shell-operator filtering can be improved to run hooks only for events in namespaces that matched a regexp or at least a simple wildcard.

how to get the value of jq.filter'key

hi
example : "jqFilter":".metadata.labels"
how could i get the value of .metadata.labels

thanks

Third party loggers bypass json logging

Sometimes shell-operator prints messages like this:
W1206 11:53:05.763798 1 reflector.go:303] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:99: watch of *v1.ConfigMap ended with: too old resource version: 1541237 (1543558)

These messages should be wrapped in json if LOG_TYPE=json is set.

JSON logs

Use logrus instead rlog, create meaningful set of labels.

No visible metrics when apiserver connection fails

There is not indication in metrics generated by shell-operator, when connection to kube-apiserver is unstable or not working at all.

The log is full of such messages:
E0705 10:45:58.761826 1 reflector.go:125] github.com/flant/shell-operator/pkg/kube_events_manager/kube_events_manager.go:354: Failed to list *unstructured.Unstructured: Get https://192.168.0.1:443/api/v1/namespaces/kube-system/endpoints?fieldSelector=metadata.name%3Dkube-dns&limit=500&resourceVersion=0: dial tcp 192.168.0.1:443: i/o timeout

Unfortunately, those are not as useful as a proper Prometheus metric.

build Dockerfile failed to solve with frontend dockerfile.v0: failed to build LLB: executor failed

=> ERROR [stage-0 6/6] RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w -X 'github.com/flant/shell-operator/pk 77.4s

[stage-0 6/6] RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w -X 'github.com/flant/shell-operator/pkg/app.Version=latest'" -o shell-operator ./cmd/shell-operator:
#13 24.74 # k8s.io/utils/trace
#13 24.74 /go/src/k8s.io/utils/trace/trace.go:100:57: invalid operation: stepThreshold == 0 || stepDuration > stepThreshold || klog.V(4) (mismatched types bool and klog.Verbose)
#13 24.74 /go/src/k8s.io/utils/trace/trace.go:112:56: invalid operation: stepThreshold == 0 || stepDuration > stepThreshold || klog.V(4) (mismatched types bool and klog.Verbose)
#13 37.42 # k8s.io/client-go/transport
#13 37.42 /go/src/k8s.io/client-go/transport/round_trippers.go:70:11: cannot convert klog.V(9) (type klog.Verbose) to type bool
#13 37.42 /go/src/k8s.io/client-go/transport/round_trippers.go:72:11: cannot convert klog.V(8) (type klog.Verbose) to type bool
#13 37.42 /go/src/k8s.io/client-go/transport/round_trippers.go:74:11: cannot convert klog.V(7) (type klog.Verbose) to type bool
#13 37.42 /go/src/k8s.io/client-go/transport/round_trippers.go:76:11: cannot convert klog.V(6) (type klog.Verbose) to type bool

---

failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to build LLB: executor failed running [/bin/sh -c CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w -X 'github.com/flant/shell-operator/pkg/app.Version=$appVersion'" -o shell-operator ./cmd/shell-operator]: buildkit-runc did not terminate sucessfully

monitor-pods example stopped working after merging #52

flant/shell-operator:latest failed to start up with monitor-pods example hook over the weekend

2019-09-30T09:25:35Z ERROR : MAIN Fatal: initialize hook manager: creating hook 'pods-hooks.sh': load hook 'pods-hooks.sh' config: 2 errors occurred:
 * kubernetes.watchEvent must be of type string: "array"
 * kubernetes.watchEvent should be one of [Added Modified Deleted]
hook --config output: {
  "configVersion":"v1",
  "kubernetes":[{
    "apiVersion": "v1",
    "kind": "Pod",
    "watchEvent":["Added"]
  }]
}

Reverting to flant/shell-operator:v1.0.0-beta.5 (3 month ago) works again for the same hooks

Various execution behaviors for hooks with Kubernetes binding

New field mode for kubernetes binding config:

mode: Incremental | Snapshot | v0

v0 — ignore existed objects, only run hooks for new events. The default behavior for legacy v0 configuration.

Incremental — run hook with Synchronization binding context with information about existed objects. Run hook with Event binding context with information about one object when an event is received. It is the default behavior for v1 configuration.

Snapshot — always run hook with Synchronization binding context.

Full support for CRDs

Brief overview:

1. Add new kind of hooks for CR conversion requests.
2. Add cmd-line options for conversion webhook parameters (listen port, SSL, service, and so on).
3. Add a better way of installing CRDs (special directory; fill webhookClientConfig).

Add new kind of hooks for CR conversion requests

1. Introduce new binding:

"onCustomResourceConversion": {
  "kind": "MyCustomResourceType",
  "conversions": [
    {from: "v1beta1", to: "v1beta2"}
  ]
}

2. TODO: Introduce format for passing ConversionRequest to the hooks

• File with objects to convert (path to the file in an environment variable)
• Several objects of the same kind can be passed at a time
• DesiredAPIVersion field as an environment variable

3. TODO: Introduce format for passing ConversionResponse from the hooks

• Result using return code
• Stdout as an error message
• File to save to converted objects (path to the file in an environment variable)

Add cmd-line options for conversion webhook parameters

• Port to listen
• Certificate to use
• CA to set to webhookClientConfig
• Service name, port, etc. to set to webhookClientConfig

Add a better way of installing CRDs

1. Add support for the directory with CRDs (exact path can be supplied using cmd-line args)
2. Parse all the CRDs in that directory and populate them with webhookClientConfig, if there are onCustomResourceConversion hooks, mentioning the same kind.
3. Install and update parsed and enriched CRDs to Kubernetes
4. Consider implementing preprocessing (using go-templates and sprig). Add built-in objects containing information about kubernetes version and available capabilities:
   • before 1.11 we don't have versions field, only version
   • before 1.13 we don't have webhook conversions (and until 1.15 they feature gate should be enabled)
   • before 1.15 we don't have defaulting

Implement extra logic

1. Support automatic chaining of several conversion hooks, if needed. For example, if one hook can convert from v1alfa1 to v1alfa2, and another can do the conversion from v1alfa1 to v1beta1, and the api-server asks us to do the conversion from v1alfa1 to v1beta1 – to hooks has to be chained.
2. Consider implementing logic, that automatically finds all objects stored in an old apiVersion and triggers a change to rewrite them (and trigger conversion).
3. Add cmd-line command to delete all installed CRDs. By default, only CRDs having no objects should be deleted. Add --force flag, to also delete CRDs having stored objects.

Add OpenAPI specification (and validations) for the hook configuration format

Can't build docker image.

1. git clone https://github.com/flant/shell-operator.git
2. docker build .

Sending build context to Docker daemon  519.2kB
Step 1/14 : FROM golang:1.12-alpine3.9
 ---> bcc226f44d8e
Step 2/14 : ARG appVersion=latest
 ---> Using cache
 ---> 1c999505e352
Step 3/14 : RUN apk --no-cache add git ca-certificates
 ---> Using cache
 ---> 12fa79d1bd54
Step 4/14 : ADD . /go/src/github.com/flant/shell-operator
 ---> fad6ecb2db1d
Step 5/14 : RUN go get -d github.com/flant/shell-operator/...
 ---> Running in f15e519054c5
Removing intermediate container f15e519054c5
 ---> 7e45c102ab06
Step 6/14 : WORKDIR /go/src/github.com/flant/shell-operator
 ---> Running in 57f0f8f79e0a
Removing intermediate container 57f0f8f79e0a
 ---> 8a0a147b24aa
Step 7/14 : RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w -X 'github.com/flant/shell-operator/pkg/app.Version=$appVersion'" -o shell-operator ./cmd/shell-operator
 ---> Running in 2e4b4481f957
# github.com/googleapis/gnostic/extensions
../../googleapis/gnostic/extensions/extensions.go:39:14: undefined: ExtensionHandlerRequest
../../googleapis/gnostic/extensions/extensions.go:50:15: undefined: ExtensionHandlerResponse
# k8s.io/client-go/transport
/go/src/k8s.io/client-go/transport/round_trippers.go:70:11: cannot convert klog.V(9) (type klog.Verbose) to type bool
/go/src/k8s.io/client-go/transport/round_trippers.go:72:11: cannot convert klog.V(8) (type klog.Verbose) to type bool
/go/src/k8s.io/client-go/transport/round_trippers.go:74:11: cannot convert klog.V(7) (type klog.Verbose) to type bool
/go/src/k8s.io/client-go/transport/round_trippers.go:76:11: cannot convert klog.V(6) (type klog.Verbose) to type bool
# k8s.io/utils/trace
/go/src/k8s.io/utils/trace/trace.go:100:57: invalid operation: stepThreshold == 0 || stepDuration > stepThreshold || klog.V(4) (mismatched types bool and klog.Verbose)
/go/src/k8s.io/utils/trace/trace.go:112:56: invalid operation: stepThreshold == 0 || stepDuration > stepThreshold || klog.V(4) (mismatched types bool and klog.Verbose)
The command '/bin/sh -c CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w -X 'github.com/flant/shell-operator/pkg/app.Version=$appVersion'" -o shell-operator ./cmd/shell-operator' returned a non-zero code: 2

I think this commit google/gnostic@51de230 had huge impact to this process.

Add support of matchLabels and matchExpressions for namespaceSelector

Instead of adding support for wildcards, as is described in #20, let we better add something like this:

  "onKubernetesEvent": [
    {
      "name": "Monitor labeled pods in cache tier",
      "kind": "Pod",

      "namespace": {
        "matchLabels": {                          # NEW
          "myLabel": "myLabelValue",              # NEW
          "someKey": "someValue",                 # NEW
          ...                                     # NEW
        },                                        # NEW
        "matchExpressions": [                     # NEW
          {                                       # NEW
            "key": "tier",                        # NEW
            "operation": "In",                    # NEW
            "values": ["cache"],                  # NEW
          },                                      # NEW
      },
    }
  ]

Concepts of matchLabels and matchExpressions are widely used throughout Kubernetes API, in contradiction to wildcards.

Add support for field selector for onKubernetesEvent hooks

Example, that seems to be self-explaining:

{
  "onKubernetesEvent": [
    {
      "fieldsSelector": {
        "matchExpressions":[
          {
            "field": "status.phase",
            "operator": "Equal",           # Equal or NotEqual
            "value": "Pending", 
          }, 
        ]
      }
    },
    ...
  ]
}

docs: enhance METRICS.md

1. shell_operator_tasks_queue_length is now have a label
2. May be add an example of alert?

Add new binding specifically to install CRDs

Definition of the problem:

1. We want to install CRD using a hook. Okay, we can use onStartup for that.
2. But we can't subscribe for that CRD's in another hook (using onKubernetesEvents), because shell-operator subscribes (create informers) during hooks configuration stage, and at this stage, we haven't yet install CRDs.

The proposed way of solving the problem:

1. Add special binding "installCRD" (with the same logic as onStartup)
2. During hooks configuration stage do the following:
   1. Read all the configuration.
   2. Run installCRD hooks.
   3. Fill the queue with onStartup hooks (but not run them yet).
   4. Create informers.
   5. Start running the queue and proceed with all the other logic.

Using this approach we'll solve both problems.

1. Provide the ability to run initial reconciliation in onStartup changes, and guarantee, that:
   1. No onKubernetesEventsHooks will be executed until onStartup is finished (we filling the queue before creating informers)
   2. No events will be missed between onStartup and subscription.
2. And at the same time – will enable the creation of CRD by the shell-operator hooks.

"Live" debug options

Debug endpoint introduced by #8 can be used to set debug options in runtime:

• endpoints:
  • /debug/settings[.json|.yaml] — list settings (DebugKeepTmpFiles, LogLevel, etc.)
  • /debug/settings/ GET | PUT (set value) | DELETE(reset to default)
• add cli command debug to addon-operator:
  • shell-operator debug settings list [-o yaml|json]
  • shell-operator debug settings get <name>|<group>
  • shell-operator debug settings set <name> <value>
  • shell-operator debug settings reset <name>
• possible options:
  • debug.keepTmpFiles
  • log.level
  • debug.log.helmOutput — log output from helm commands
  • debug.log.kubernetesEvents — log information about incoming kubernetes events
  • debug.log.kubernetesEvents.dumpObjects — also dump incoming objects
  • ...

Update kubectl in the Docker images

kubectl is now four major versions behind. Please update it to the latest stable version. I'd love to start experimenting with this.

Also, tagging and releasing the first stage of this build would make it easier for others to choose the kubectl version they want.

Can I use shell-operator to create a crud as operator-sdk does?

Hi folks,

I went through the document that says shell-operator can be reactive to any events from any api resources. For me, shell-operato looks like a generic event listener of k8s cluster.

My question is:
for example, I can use openshift's operator-sdk to create a mysql-operator with go language.
Can I use shell-operator to do the same thing ?

thanks.

Latest version of shell-operator

Hi! This is rather a question than an issue.

The current documentation that describes v1 API (with Synchronizationtype of hook binding context and other features) is inconsistent with the latest official release of the shell-operator which is v1.0.0-beta.5. Initially, our team implemented the solution using the shell-operator API described in the docs, but then we figured out it did not work if we used v1.0.0-beta.5 release. That's why we had to use :latest tag of the shell-operator image to make it work...
I'm wondering why there were no releases for more than 6 months already? Are you planning to release either v1.0.0-beta.6 or v1.0.0 version soon?

And thank you for everything you are doing - shell-operator is an awesome tool!

Binding to events from CRDs, better support different k8s versions

Use DynamicInformers to support subscription to events from CRDs and better support version/kind pairs.

Ability to return a set of actions from hook

Replace objectName with nameSelector.matchNames

Reasons:

1. We already have matchNames in namespaceSelector
2. matchNames will allow matching for more than one object at a time

To my eye, objectName looks extremely ugly and non-Kubernetes-API-common, so we better use matchNames from prometheus-operator – at least we will be not the only ones using this keyword. And at least we will use the same approach both for namespace matching and for the object matching.

Feature request - Get event message in resource object

It would be really useful if you could provide event message also in the resource object so we can act accordingly as well. This is probably mainly helpful for autoscaling events.

New metric for requests to a kubernetes api server

Increase a counter for each request. _total counter without labels. module has helm resource requests and synchronization requests — two counters for module.

How to measure Watch requests?

configmap update cannot be monitored

my hook：
{
"onKubernetesEvent": [{
"name": "Monitor configmap",
"kind": "ConfigMap",
"event": ["add", "update"],
"objectName": "test-config",
"namespaceSelector": {
"any": true
},
"jqFilter": ".metadata.labels",
"allowFailure": false
}]
}

It was monitored when I created the configmap ，but not when I modified the configmap
Is there anything wrong with my hook？

Include snapshots from other bindings into binding context

It's a change for #91

Add includeSnapshotsFrom field. This field is an array with binding names from kubernetes section. When the hook is triggered, all existing objects (snapshot) will be added as kubernetesSnapshots.<binding name>. It is allowed to do self-include.

schedule config:

"schedule": {
  "includeSnapshotsFrom": ["pods","secrets"]

kubernetes config:

"kubernetes": {
  "includeSnapshotsFrom": ["pods","secrets"]

Binding context content for schedule:

[{
  "binding":"schedule",
  "snapshots":{
    "pods":[
      {"object":{"metadata":{}, "spec":{}}, "filterResult":".."},
      {"object":{"metadata":{}, "spec":{}}, "filterResult":".."},
      {"object":{"metadata":{}, "spec":{}}, "filterResult":".."}
    ],
    "secrets":[
      {"object":{"metadata":{}, "spec":{}}, "filterResult":".."},
      {"object":{"metadata":{}, "spec":{}}, "filterResult":".."},
      {"object":{"metadata":{}, "spec":{}}, "filterResult":".."}
    ]
  }
}]

Binding context content for kubernetes:

[{
  "binding":"kubernetes",
  "type":"Event"
  "watchEvent": "",
  "object": {},
  "filterResult":"",
  "snapshots":{
    "pods":[
      {"object":{"metadata":{}, "spec":{}}, "filterResult":".."},
      {"object":{"metadata":{}, "spec":{}}, "filterResult":".."},
      {"object":{"metadata":{}, "spec":{}}, "filterResult":".."}
    ],
    "secrets":[
      {"object":{"metadata":{}, "spec":{}}, "filterResult":".."},
      {"object":{"metadata":{}, "spec":{}}, "filterResult":".."},
      {"object":{"metadata":{}, "spec":{}}, "filterResult":".."}
    ]
  }
}]

Resource name in config to monitor a particular resource only

There is no resource name field in onKubernetesEvent config section. Only kind can be specified for now. You can use labels to narrow events but resource name is way more simpler and cleaner.

Add documentation of environment variables

There is no description of SHELL_OPERATOR_* and LOG_* envs.

Run hooks in named Task queues

An adoption of addon-operator issue flant/addon-operator#35 :

There are several use cases when hooks should not block each other. For example, a secret copier and some cleaning tasks. Shell-operator can start several named queues and handlers to run hooks in parallel. schedule and kubernetes binding configuration should have a new field with the name of the desired queue. Flag name is something like queueName and an empty flag is adding the hook to the default or main queue.

The workaround for this problem is using several instances of shell-operator, but hooks with an additional field in the configuration is much simpler.

Migrate all examples to "v1" configuration schema

This should be done after #37, #36, #35, #33

fieldSelector should support not only metadata.name and metadata.namespace

DynamicInformer supports only these fields. Usual Informers support more fields.