fluxcd / notification-controller

The GitOps Toolkit event forwarded and notification dispatcher

notification-controller

Event forwarder and notification dispatcher for the GitOps Toolkit controllers. The notification-controller is an implementation of the notification.toolkit.fluxcd.io API based on the specifications described in the RFC.

notification-controller's Issues

Add Webex as an Alert Provider

I am happy to implement this myself based off the existing providers and open a PR if members of this community are okay with that.

Webhook for Bitbucket server causes error with unsupported Content-Type

I'm trying to trigger the notification-controller with a Bitbucket POST webhook.
My Bitbucket server sends a HTTP request with the header Content-type: application/json; charset=utf-8.
The flux notification-controller fails with this error:

{
	"level":"error",
	"ts":"2020-12-14T14:41:21.652Z",
	"logger":"receiver-server",
	"msg":"unable to validate payload",
	"receiver":"bitbucket-receiver",
	"error":"the Bitbucket server signature header is invalid, err: Webhook request has unsupported Content-Type \"application/json; charset=UTF-8\""
}

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Receiver
metadata:
  name: bitbucket-receiver
  namespace: flux-system
spec:
  type: bitbucket
  events:
  - "repo:refs_changed"
  secretRef:
    name: webhook-token
  resources:
  - kind: GitRepository
    name: flux-system

Allow signing of generic webhook provider requests

The HTTP endpoint should be able to ensure the provenance of incoming requests from the generic provider.

GitHub and GitLab use a shared-secret mechanism, where the outgoing requests have a header with an HMAC of the request body, the receiver can then sign the body it receives, and confirm the origin of the request.

This should be optional.

One proposal, is to add an additional signature field to the spec:

spec:
 signature:
   secretRef:
       name: hmac

Another option would be to use the presence of an hmac key in the Secret referenced by spec.secretRef.

Some other considerations, sha1 or sha256? What should the header name be?

Annotate with the controller-runtime unstructured client

We should drop the dependencies to the GitOps Toolkit API packages and use the controller-runtime unstructured client to annotate the GitOps Toolkit objects.

Specifying notification Provider's namespace in an alert

Running 0.6.1

Regarding this section:
https://github.com/fluxcd/notification-controller/blob/main/config/crd/bases/notification.toolkit.fluxcd.io_alerts.yaml#L92

It only allows you to specify the name of the alert Provider, in my case discord. But it does not allow to reference it form another namespace (monitoring in this case).

The intent I believe, in a single tenant cluster, is to have only one Provider, which sits nicely in the flux-system namespace.

But it is impossible now to create an alert in a namespace where there are eg. HelmReleases you want to watch (like monitoring).

This is what I must do right now:

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: discord-alert-monitoring
  namespace: flux-system
spec:
  providerRef:
    name: discord
  eventSeverity: info
  eventSources:
    - kind: HelmRelease
      name: '*'
      namespace: monitoring

This is what I would like to do:

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: discord-alert-monitoring
  namespace: monitoring  # <<--- not flux-system
spec:
  providerRef:
    name: discord
    namespace: flux-system   #  <<---- THIS IS NOT SUPPORTED
  eventSeverity: info
  eventSources:
    - kind: HelmRelease
      name: '*'

For me, the best solution would be:

Have issue #71 implemented, since I could subscribe to all (single tenant) cluster events at once, without having to worry about setting up separate alerts per namespace.

Notification controller does not start on clusters with active PSPs (runAsNonRoot)

There isn't much more to add.

Error: container has runAsNonRoot and image has non-numeric user (controller), cannot verify user is non-root

Error notifications contain too much not error data

Hi 👋 ,

I believe that error messages are too verbose and should focus more on the actual error, e.g. (taken from a slack notification):

apply failed: secret/flux-known-hosts-79bt28cg56 unchanged
secret/gitlab-infra-company-com unchanged
secret/slack-live-team-ops unchanged
service/notification-controller unchanged
deployment.apps/kustomize-controller unchanged
kustomization.kustomize.toolkit.fluxcd.io/aks07-euw-dev-link-default unchanged
kustomization.kustomize.toolkit.fluxcd.io/aks07-euw-dev-link-flux unchanged
kustomization.kustomize.toolkit.fluxcd.io/aks07-euw-dev-link-ingress unchanged
kustomization.kustomize.toolkit.fluxcd.io/aks07-euw-dev-link-kube-system unchanged
kustomization.kustomize.toolkit.fluxcd.io/aks07-euw-dev-link-logging unchanged
kustomization.kustomize.toolkit.fluxcd.io/aks07-euw-dev-link-maintenance unchanged
kustomization.kustomize.toolkit.fluxcd.io/aks07-euw-dev-link-monitoring unchanged
alert.notification.toolkit.fluxcd.io/slack-live-team-ops unchanged
provider.notification.toolkit.fluxcd.io/slack-live-team-ops unchanged
receiver.notification.toolkit.fluxcd.io/gitlab-infra-company-com unchanged
gitrepository.source.toolkit.fluxcd.io/flux-workspaces unchanged
ingressroute.traefik.containo.us/flux-receiver unchanged
The Service "webhook-receiver" is invalid: spec.clusterIP: Invalid value: "10.200.133.61": field is immutable

The only relevant information (according to me) is the last line, all the unchanged lines are noise that make the real error less visible (keep in mind that slack for example only display the first 2 lines by default).

Regards.

Add rate limiter for receiver

Exposing the receiver to the internet may result in an accidental dos of the api server, either through a bug in the other end making a request to the receiver or a bad actor figuring out the receiver endpoint.

The rate limiter is implemented in the events server in #167 but was not implemented for the receiver. The reason for this was because there is a high risk that too many requests would be rate limited as it is hard to differentiate requests for different commit references.

Adding cluster name (or something else) to Github commit status context

We run a few clusters with Flux and are moving towards Flux 2 now, and so far the experience is being awesome.

We have seen an issue in this notification controller, and it's that when having multiple clusters posting the commit status for the same resource names (such as kustomization/flux-system), the statuses get overwritten, as they use the <resource type>/<name> as context.

Therefore, it would be awesome if we could add some kind of prefix or suffix to this context in order to distinguish the status posted by the different clusters.

And once again, Flux 2 is great. Thanks for your hard work.

Add status support for Bitbucket

We should support Bitbucket in the same way that we support GitHub and GitLab.

ktrysmt/go-bitbucket is probably the best option for a client to use with Bitbucket.

Git commit status into dev deploy

Describe the bug

We want to upgrade our flux to flux v2 deployed into multiple clusters, all point same bitbucket repo. Seem like bootstrap using a git bitbucket repo is no ready yet.

Flux v1 tag the commit of the repo that are using. We are looking for similar function into flux v2, but only find the commit status solution to do similar function.

To deploy flux we are using dev mode, but when try to configure alerts, we get these errors:

2021-04-21T09:54:21.284Z info Kustomization/podinfo.flux-system - Dispatching event: Update completed
2021-04-21T09:54:21.744Z error Kustomization/podinfo.flux-system - failed to send notification 401 Unauthorized

To Reproduce

Steps to reproduce the behaviour:

1. Provide Flux install instructions

flux install --network-policy=true --watch-all-namespaces=true --namespace=flux-system

flux create source git name1 --url=https://bitbucketfactory.repo/route/to/repo.git --branch=master --interval=1m --username=user --password=pass

flux create kustomization name2 --source=name1 --path="./dev/services" --prune=true --interval=5m

podinfo (public kustomize copied into "./install/podinfo)

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 5m
  targetNamespace: podinfo
  path: ./install/podinfo
  prune: true
  sourceRef:
    kind: GitRepository
    name: name1
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: podinfo
      namespace: podinfo
  timeout: 1m

secret:

apiVersion: v1
kind: Secret
metadata:
  name: provider-secret
  namespace: flux-system
data:
  token: base64name:base64pass

alert and alert-provider

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Provider
metadata:
  name: flux-system
  namespace: flux-system
spec:
  type: bitbucket
  address: https://bitbucketfactory.repo/route/to/repo.git
  secretRef:
    name: provider-secret

---

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: podinfo
  namespace: flux-system
spec:
  providerRef:
    name: flux-system
  eventSeverity: info
  eventSources:
    - kind: Kustomization
      name: podinfo
      namespace: flux-system

Expected behavior

We want that the commit will marked when flux v2 are using it into every cluster.

Additional context

• Kubernetes version: 1.17.12
• Git provider: bitbucket

Below please provide the output of the following commands:

flux --version: 0.12.1
flux check

► checking prerequisites
✗ flux 0.12.1 <0.12.3 (new version is available, please upgrade)
✔ kubectl 1.19.0 >=1.18.0-0
✔ Kubernetes 1.17.12-eks-7684af >=1.16.0-0
► checking controllers
✔ all checks passed

GCR webhook receviers

Ref: https://github.com/fluxcd/flux-recv/blob/master/gcr.go

Enterprise github repo: not generating events and unable to write commit status to github

Team,
version using : flux2 v0.8.0
events are not generating and unable to write commit status to github about the deployment

Following is error noticing in notification controller log.. It was supposed to fetch from my github domain but it's trying to fetch from api.github.com please check following error
**```

{"level":"error","ts":"2021-03-12T18:40:29.577Z","logger":"event-server","msg":"failed to send notification","object":"platform/badlybehavingtestservice","kind":"Kustomizatio │
│ n","error":"could not list commit statuses: Get "https://api.github.com/repos/s-voggu/eks/commits/d1cc91e17c740a4d203e8eef04b8312bab47f46b/statuses?per_page=50\": net/http:

In gitrepository type resource I'm using ssh repo url and able to fetch latest commit status...In providers I'm using https repo url in address field.  In gitrespositry resource and kusomization resource events are getting generated but I don't see any events in alerts and providers . Observing above error in notification controller log please assist asap

Generic receiver shoudn't requires a SecretRef

The notification-controller looks for a secret even when the receiver type is "generic".

According to the documentation:

When the receiver type is set to generic, the controller will not perform token validation nor event filtering.

Is their actually a need for a token in this case ?

Shouldn't the SecretRef field be optional ?

bitbucket max key length

Hi,

Bitbucket api has a max key value length of 40 characters. Currently the value of the key for notification sent to bitbucket is the object kind and name separated by slash i.e kustomization/service-foo. If you happen to have long service names and/or combinations with namespace names because of multiple environments sharing the same config repo (kustomization/namespace-foo-and-service-bar) you hit the roof which results in 400 error responses from bitbucket.

Support deployment on GKE version 1.15.x

Do you plan to support deployment on GKE version 1.15.x?
How can I deploy to GKE version 1.15.x
Thank you

Quay webhook receiver

Should be similar to DockerHub: #112

Redact tokens from logs

It would be nice if we could redact tokens from logs.

GitHub notifier spams commit status

It seems like the GitHub api takes every commit status as a new update causing a huge amount of commit statuses per commit, if the reconciliation interval is short. Other providers either overwrites the previous commit status or does nothing if the status is not updated. We should add a check to the GitHub notifier so that it only sends a commit status if the state or message has changed from the previous one.

[
{
    "url": "",
    "avatar_url": "",
    "id": 0,
    "node_id": "",
    "state": "success",
    "description": "reconciliation succeeded",
    "target_url": null,
    "context": "kustomization/apps",
    "created_at": "2020-12-02T13:53:03Z",
    "updated_at": "2020-12-02T13:53:03Z",
    "creator": {}
  },
  {
    "url": "",
    "avatar_url": "",
    "id": 0,
    "node_id": "",
    "state": "success",
    "description": "reconciliation succeeded",
    "target_url": null,
    "context": "kustomization/apps",
    "created_at": "2020-12-02T13:52:03Z",
    "updated_at": "2020-12-02T13:52:03Z",
    "creator": {}
  }
]

Filter events sent by different types of notifiers

The kustomization controller limits the amount of events sent to the notification controller, to limit the amount of message sent by the chat notifiers (Slack, MS Teams, etc.). If this was not done the messages would be very verbose, as events would be sent every reconcile loop, even when nothing has changes.

On the other hand the git notifiers need to receive succeeded events if a deployment goes from unhealthy to healthy without the revision changing.

These two requirements are at odds with each other. To deal with this the chat notifiers need to filter out any succeeded events where the revision has not changed from the last sent event.

Add Sentry as a Provider

We'd like to push reconciliation events to sentry. I've already done some investigation and started implementing using the sentry-go library. Happy to make a PR and push my changes upstream if other members would find this integration useful.

HelmRelease errors spam slack

In setting up an alert to send to slack for HelmRelease objects at error level, it sends a ton of messages. Here is a simple HelmRelease I applied, with an alert defined for error:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: broken-test
  namespace: default
spec:
  chart:
    spec:
      chart: charts/mychart
      sourceRef:
        kind: GitRepository
        name: gitops
        namespace: flux-system
  interval: 10m0s
  values:
    nothing: noone

This chart has a required value, so I knew this would trigger an error. However, slack sent 22 messages within 3 seconds or so, all nearly identical. We got 11 identical notifications for Helm install failed and 11 identical notifications for reconciliation failed.

Would it be possible to either thread these messages, or have a setting to limit number of messages within a given time frame? I could image something like this on the AlertSpec:

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: myalert
  namespace: flux-sustem
spec:
  thread: true # For each alert, thread additional messages for the same object type/name
  notifyWindow: 
    numMessages: 1
    timeDelay: 10m0s # Allow only one notification per object per error within the given time frame

Receivers not covered by --watch-all-namespaces

While setting up a multi-tenant testbed, I fail to create a webhook to sync the tenant's GitRepository in the tenant's namespace. Receiver for the flux-system namespace works fine.

Looking at

notification-controller/internal/server/receiver_handlers.go

Line 46 in a7e3be0

err := s.kubeClient.List(ctx, &allReceivers, client.InNamespace(os.Getenv("RUNTIME_NAMESPACE")))

, it seems like notification-controller will only react on Receivers in its own namespace. Is this unintentional, or should we run a notficaiton-controller instance per namespace for Receivers?

Flux notification controller with github enterprise

Hi Team and @phillebaba
this is the continuation of previous request: #161
Notification controller image updated to :

image: ghcr.io/fluxcd/notification-controller:v0.13.0

When I'm trying to create github alert observing following issue in notification controller logs

kind":"Kustomization","name":"appname","namespace":"platform"}
{"level":"error","ts":"2021-05-13T08:35:41.567Z","logger":"event-server","msg":"failed to send notification","reconciler kind":"Kustomization","name":"appname","namespace":"platform","error":"Revision string format incorrect: environment/dv01aws/e0358f0de99e1a078105b05ec2e18a8f242aef5694a"}

the github authentication is through SSO that's why I set up the gitrepsotiory resource with SSH (URL and creds) however the Provider resource has HTTPS address

Will it be a reason for this kind of issue?

Also I didn't find in the doc https://fluxcd.io/docs/guides/notifications/ that github token should be base64 encoded or not?

ACR webhook receiver

Doc ref: https://docs.microsoft.com/en-us/azure/container-registry/container-registry-webhook

Self Signed Cert on Github Provider

Description

No option to specify TLS CA self signed cert in Provider Spec.

Details

We run a GitHub enterprise server that has a self signed CA cert attached to it. When trying to set up git commit status, the notification controller gets the below error.

failed to send notification could not list commit statuses: ... : x509: certificate signed by unknown authority

Maybe a solution similar to the ImageRepository CRD can be implemented in the provider as well. Specifically a value similar to certSecretRef?

Infinite loop when Receiver secret is invalid

If you add a Receiver that reference either:

• no secret
• a secret that doesn't exist
• a secret without the token field

Then the notification-controller enter in an infinite reconciliation loop.

manifest.yaml:

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Receiver
metadata:
  name: foo
spec:
  type: generic
  secretRef:
    name: foo-token
  resources:
    - kind: GitRepository
      name: bar

logs:

[...]
{"level":"info","ts":"2021-02-05T14:15:28.883Z","logger":"controller.receiver","msg":"Receiver initialised","reconciler group":"notification.toolkit.fluxcd.io","reconciler kind":"Receiver","name":"foo","namespace":"flux-system"}
{"level":"error","ts":"2021-02-05T14:15:28.890Z","logger":"controller.receiver","msg":"Reconciler error","reconciler group":"notification.toolkit.fluxcd.io","reconciler kind":"Receiver","name":"foo","namespace":"flux-system","error":"Operation cannot be fulfilled on receivers.notification.toolkit.fluxcd.io \"foo\": the object has been modified; please apply your changes to the latest version and try again"}
{"level":"info","ts":"2021-02-05T14:15:28.907Z","logger":"controller.receiver","msg":"Receiver initialised","reconciler group":"notification.toolkit.fluxcd.io","reconciler kind":"Receiver","name":"foo","namespace":"flux-system"}
{"level":"error","ts":"2021-02-05T14:15:28.913Z","logger":"controller.receiver","msg":"Reconciler error","reconciler group":"notification.toolkit.fluxcd.io","reconciler kind":"Receiver","name":"foo","namespace":"flux-system","error":"Operation cannot be fulfilled on receivers.notification.toolkit.fluxcd.io \"foo\": the object has been modified; please apply your changes to the latest version and try again"}
[...]
{"level":"error","ts":"2021-02-05T14:15:29.872Z","logger":"controller.receiver","msg":"Reconciler error","reconciler group":"notification.toolkit.fluxcd.io","reconciler kind":"Receiver","name":"foo","namespace":"flux-system","error":"Operation cannot be fulfilled on receivers.notification.toolkit.fluxcd.io \"foo\": StorageError: invalid object, Code: 4, Key: /registry/notification.toolkit.fluxcd.io/receivers/flux-system/foo, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: d99a981f-6d66-4403-8ff9-442354ab1de0, UID in object meta: "}
[...]

# kubectl get receivers -w
[...]
foo      False   unable to read token from secret 'flux-system/foo-token' error: Secret "foo-token" not found            1s
foo      True    Receiver initialised with URL: /hook/3066a8efff25c5e29792e8a96ee867f9f98533af0d8a6dc6ed9636ee1b5f3a5b   1s
foo      False   unable to read token from secret 'flux-system/foo-token' error: Secret "foo-token" not found            1s
foo      True    Receiver initialised with URL: /hook/3066a8efff25c5e29792e8a96ee867f9f98533af0d8a6dc6ed9636ee1b5f3a5b   1s
foo      False   unable to read token from secret 'flux-system/foo-token' error: Secret "foo-token" not found            1s
foo      True    Receiver initialised with URL: /hook/3066a8efff25c5e29792e8a96ee867f9f98533af0d8a6dc6ed9636ee1b5f3a5b   1s
foo      True    Receiver initialised with URL: /hook/3066a8efff25c5e29792e8a96ee867f9f98533af0d8a6dc6ed9636ee1b5f3a5b   1s
[...]

Exclude messages from alerting

In some cases it is desirable to not send alerts if the message body matches some string e.g. fluxcd/flux2#760

Azure EventHub integration

We are building an integration for the notification-controller to work with Azure EventHub: https://azure.microsoft.com/en-us/services/event-hubs/

Slack provider for a private channel doesn't work

I have an incoming webhook configured for a specific private channel and as far as I know slack has made a change at some point that it's forbidden to specify the channel name in this case, so it's impossible to use such webhook because "channel" is mandatory

notification-controller/internal/notifier/slack.go

Line 69 in 337b5ab

return nil, errors.New("empty Slack channel")

When I'm testing the webhook url using curl it works perfectly fine without specifying the channel, but if I specify it I always get "channel_not_found". And notification controller is completely silent about this.

Certain types of notification not sent

Flux version: 0.10.0
Provider: googlechat

Notifications for resources marked for deletion/deleted are sent to Google chat.
Notifications for resources created and health check notifications (as in https://toolkit.fluxcd.io/_files/slack-info-alert.png) are logged as dispatched by notification-controller but do not get sent to Google chat.

I think after a look at the code, this is because these events are incorrectly sent as "Progressing" events by kustomize-controller, and the notifier in notification-controller skips these events.

I think that events logged as dispatched by the notification-controller but which are black-holed by the notifier can be misleading, because they are not dispatched to the provider which it could be interpreted as.

GitHub Alert fails with "missing revision metadata" on GitRepository eventsource

I have configured a minimal notification Provider and Alert for GitHub, however the event is being dropped from the notification-controller with the following error missing revision metadata, the full details are below.

I would expect the event to be fired and handled correctly. Is there any more I can do to help debug here?

Flux Component versions

image: ghcr.io/fluxcd/helm-controller:v0.4.4
image: ghcr.io/fluxcd/kustomize-controller:v0.5.3
image: ghcr.io/fluxcd/notification-controller:v0.5.0
image: ghcr.io/fluxcd/source-controller:v0.5.6

Error Log:

{"level":"info","ts":"2021-01-06T15:13:59.632Z","logger":"event-server","msg":"Dispatching event","object":"flux-system/flux-system","kind":"GitRepository","message":"Fetched revision: main/029fc93e2718bed1b58c171e077467b269076f62"}
{"level":"error","ts":"2021-01-06T15:13:59.634Z","logger":"event-server","msg":"failed to send notification","object":"flux-system/flux-system","kind":"GitRepository","error":"missing revision metadata"}

Manifests:

---
apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: github-alert
  namespace: flux-system
spec:
  eventSeverity: info
  eventSources:
  - kind: GitRepository
    name: flux-system
    namespace: flux-system
  providerRef:
    name: github
---
apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Provider
metadata:
  name: github
  namespace: flux-system
spec:
  address: https://github.com/tobbbles/flux
  secretRef:
    name: github-api-token
  type: github

Github Notifier

The Github Notifier given an event containing revision metadata it updates the commit status in a Github repository.

The notifier would be configured in a similar manner as the other notifiers. With some small differences. The address given in the provider would have be the address of the Github repository. Aditionally the provider requires a secret ref to be given the personal access token to communicate with Github. The Alert would need to explicitly specify the resources to react upon as the source repository they are reconciled with have to match the repository of the provider. Another solution would be if we could get the source repository of the involved object and match it to a provider with the same repository, but that would go against the current behavior of the notifier.

apiVersion: notification.fluxcd.io/v1alpha1
kind: Provider
metadata:
  name: podinfo-gh-status
  namespace: gitops-system
spec:
  type: github-commit-status
  address: https://github.com/stefanprodan/podinfo # repo URL
  secretRef:
    name: github-token # GH token with 'repo:status' scope
---
apiVersion: v1
kind: Secret
metadata:
  name: github-token
  namespace: gitops-system
data:
  token: <token>
---
apiVersion: notification.fluxcd.io/v1alpha1
kind: Alert
metadata:
  name: podinfo-status-update
  namespace: gitops-system
spec:
  providerRef: 
    name: podinfo-gh-status 
  eventSeverity: info
  eventSources:
    - kind: Kustomization
      name: webapp-backend
---
apiVersion: kustomize.fluxcd.io/v1alpha1
kind: Kustomization
metadata:
  creationTimestamp: null
  name: webapp-backend
  namespace: gitops-system
spec:
  dependsOn:
  - webapp-common
  healthChecks:
  - kind: Deployment
    name: backend
    namespace: webapp
  interval: 10m0s
  path: ./deploy/webapp/backend
  prune: true
  sourceRef:
    kind: GitRepository
    name: webapp
  timeout: 2m0s
  validation: client
status: {}

An detail in this implementation is that if the Kustomization resource and the source repoisitory sepcified are not the same. The event revision relates to the source repository given in the Kustomization resource, and not the repository the Kustomization resource originates from.

provider address for gitlab needs to contain the project id

From the documentation here I tried setting up a provider for my gitlab-repo using the git-url used to clone the project. The notification-controller then reported a 404 when trying to post the commit-status.

https://***/api/v4/projects/xxx/xxx/xxx/xxx/statuses/b5d3df3bc3fd0d530b1ef897374c3af80e3560b3

As the url contains the full path of groups I tried settings up the provider using the projects id instead of the path in the form

https://***/123

This way the notification controller is using the correct api-url with project id.

From my understanding the gitlab-api does not support accessing the api via the project path instead the id has to be used. This should be mentioned in the documentation for setting up flux. Maybe even a sample for each of the supported providers would be nice as other providers.

Adding object kind to notifications

In some notification providers like Teams it is not always clear from which object type the notifications origin.

In Slack the originating controller is used as a username of the message, but in Teams the username is set by the webhook configuration.

Would it be possible to add the Kind of the causing object to the objName that is part of the message.
E.g. as pre-/ or postfix like
<Kind>/<name>.<namespace>
or
<name>.<namespace>.<kind> (to stay in hostname notation)

(from fluxcd/flux2#710 )

GitLab commit status notifier

Similar to GitHub, we could implement commit status updates for GitLab.

mTLS support for self-hosted gitlab behind cloudfare firewall

Unable to access Gitlab API behind cloudfare firewall.

We have a self-hosted Gitlab running behind a cloudfare firewall that requires mTLS to be accessed.
Currently there's no way to specifiy certificates to be used for mTLS.
We have the option in the Provider to specify a caFile, certFile and keyFile, but only the caFile seems to be implemented to support self-signed certificates.

Versions

• flux2 0.13.0
• Gitlab version 13.10.0

Relevant manifests:

1. create provider

---
apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Provider
metadata:
  name: self-hosted-gitlab
  namespace: demo
spec:
  type: gitlab
  channel: general
  address: <self-hosted-gitlab-URL>
  secretRef:
    name: self-hosted-gitlab

2. create alert

---
apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: sample
  namespace: demo
spec:
  providerRef:
    name: self-hosted-gitlab
  eventSeverity: info
  summary: "some summary"
  eventSources:
    - kind: GitRepository
      name: '*'
    - kind: Kustomization
      name: sample
      namespace: demo
---

3. create secret with gitlab token

---
apiVersion: v1
kind: Secret
metadata:
  name: self-hosted-gitlab
  namespace: cd
data:
  token: <redacted>
---

Relevant logs:

Error from notification-controller:

{
  "level": "info",
  "ts": "2021-04-26T07:35:08.088Z",
  "logger": "event-server",
  "msg": "Dispatching event: Update completed",
  "reconciler kind": "Kustomization",
  "name": "aws-sample",
  "namespace": "demo"
}
{
  "level": "error",
  "ts": "2021-04-26T07:35:08.414Z",
  "logger": "event-server",
  "msg": "failed to send notification",
  "reconciler kind": "Kustomization",
  "name": "aws-sample",
  "namespace": "demo",
  "error": "invalid character '<' looking for beginning of value"
}

Notes

Accessing the Gitlab API using curl and specifying the certificates with --cert and --key works fine.

Different alerts mix up summary

Issue description

I've created different alerts, listening to GitRepository and HelmRelease.
They have the same eventSources, the same exclusionList, but different summaries and different providers.
The notifications created by these alert-events will be pushed as expected to the different providers.

Not working as expected: But the summary seems to be mixed up.

Using Notification-Controller v0.9.0, v0.10.0

Steps to reproduce the issue

Creating three different providers and three alerts, like e.g.:

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: btest-alert-1
  namespace: flux-system
spec:
  summary: "Bugtest Helm Notification"
  providerRef: 
    name: webhook-provider
  eventSeverity: info
  eventSources:
    - kind: "GitRepository"
      name: "*"
      namespace: "flux-system"
    - kind: "HelmRelease"
      name: "*"
      namespace: "dev"
  exclusionList:
    - "HelmChart.*is.not.ready"
    - "i/o.*timeout"

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: btest-alert-2
  namespace: flux-system
spec:
  summary: "Another Helm Notification"
  providerRef: 
    name: webhook-provider-2
  eventSeverity: info
  eventSources:
    - kind: "GitRepository"
      name: "*"
      namespace: "flux-system"
    - kind: "HelmRelease"
      name: "*"
      namespace: "dev"
  exclusionList:
    - "HelmChart.*is.not.ready"
    - "i/o.*timeout"	

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: btest-alert-3
  namespace: flux-system
spec:
  summary: "Rocketchat Helm Notification"
  providerRef: 
    name: rocketchat-provider
  eventSeverity: info
  eventSources:
    - kind: "GitRepository"
      name: "*"
      namespace: "flux-system"
    - kind: "HelmRelease"
      name: "*"
      namespace: "dev"
  exclusionList:
    - "HelmChart.*is.not.ready"
    - "i/o.*timeout"		

What's the expected result?

Each notification has the correct summary of a defined alert.

What's the actual result?

The summary seems to be mixed up. Sometimes the rocketchat-provider gets the notification with the summary of the generic-webhook-provider and sometimes the correct summary and sometimes the summary from another generic-webhook-provider.

Allow a custom field to be set on Alert

We use Flux2 on 3 separate Kubernetes clusters. All of them monitor the same BitBucket repo for changes and deploy the appropriate apps based on GitRepository & Kustomization settings. There is also a Slack channel where we feed alerts into, coming from all three clusters.

Since all clusters are sending multiple alerts (we are currently at info alert level) there is no way to distinguish which alert is coming from which cluster.

Therefore we are asking for the option to add a custom field on Alert that will be included in the alert being sent

Nexus webhook receiver

Implement Nexus using X-Nexus-Webhook-Signature ref: https://github.com/fluxcd/flux-recv/blob/master/nexus.go

Add http(s) proxy support

To send notifications behind a proxy, we would need the ability to configure a http(s) proxy.

{"level":"error","ts":"2020-10-16T14:59:37.134Z","logger":"event-server","msg":"failed to send notification","object":"gotk-system/example-app","kind":"Kustomization","error":"postMessage failed: POST https://webhook-url giving up after 5 attempt(s): Post \"https://webhook-url\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}

receiver: support webhook

Implement webhook receivers according to specification.

Add Prometheus instrumentation to the HTTP servers

We could create a middleware that instruments the events and the webhook receiver servers, this way users can track how many requests have been processed, the number of failures, rate limits, etc.

ExclusionList not working

I was trying to use this new feature to filter out some events that I don't want to be notified. After some time trying to remove really simple messges I checked the code and I think I saw a bug in the implementation.

I would say the problem is here. Continue instructions could be tricky, they continue to the next iteration in the inner bug. In that case there are two nested loops, so I would say this continue instruction will continue evaluating the next ExclusionList entry instead of going to the next event. As a result it never skips an event.

leaving channel unset silently breaks Slack provider

The channel field is marked optional, and leaving it out passes validation.
I would expect this to implicitly use the default channel of the webhook.
Checking the status of the Provider and Alert resources reveals nothing, and the notification controller appears healthy.

edef@jaguar ~/s/m/p/e/p/gke ((f478031a…))> kubectl describe -n flux-system provider/slack alert/slack svc/notification-controller
Name:         slack
Namespace:    flux-system
Labels:       kustomize.toolkit.fluxcd.io/checksum=de849d6f08587510d1a21f9a7adf9de6ed889e28
              kustomize.toolkit.fluxcd.io/name=platform-prod-gke
              kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations:  API Version:  notification.toolkit.fluxcd.io/v1beta1
Kind:         Provider
Metadata:
  Creation Timestamp:  2020-11-06T15:38:36Z
  Generation:          1
  Resource Version:    5091913
  Self Link:           /apis/notification.toolkit.fluxcd.io/v1beta1/namespaces/flux-system/providers/slack
  UID:                 0db75803-2f28-4297-96e9-0d6178eab6fb
Spec:
  Secret Ref:
    Name:  slack-webhook
  Type:    slack
Status:
  Conditions:
    Last Transition Time:  2020-11-06T15:38:36Z
    Message:               Initialized
    Reason:                Initialized
    Status:                True
    Type:                  Ready
Events:                    <none>


Name:         slack
Namespace:    flux-system
Labels:       kustomize.toolkit.fluxcd.io/checksum=de849d6f08587510d1a21f9a7adf9de6ed889e28
              kustomize.toolkit.fluxcd.io/name=platform-prod-gke
              kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations:  API Version:  notification.toolkit.fluxcd.io/v1beta1
Kind:         Alert
Metadata:
  Creation Timestamp:  2020-11-06T15:38:36Z
  Generation:          1
  Resource Version:    5091911
  Self Link:           /apis/notification.toolkit.fluxcd.io/v1beta1/namespaces/flux-system/alerts/slack
  UID:                 fd502ac1-cdd6-44ff-ac7a-c4684f547086
Spec:
  Event Severity:  info
  Event Sources:
    Kind:  GitRepository
    Name:  *
    Kind:  Kustomization
    Name:  *
  Provider Ref:
    Name:  slack
Status:
  Conditions:
    Last Transition Time:  2020-11-06T15:38:36Z
    Message:               Initialized
    Reason:                Initialized
    Status:                True
    Type:                  Ready
Events:                    <none>


Name:              notification-controller
Namespace:         flux-system
Labels:            app.kubernetes.io/instance=flux-system
                   app.kubernetes.io/version=0.2.1
                   control-plane=controller
                   kustomize.toolkit.fluxcd.io/checksum=de849d6f08587510d1a21f9a7adf9de6ed889e28
                   kustomize.toolkit.fluxcd.io/name=platform-prod-gke
                   kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations:       Selector:  app=notification-controller
Type:              ClusterIP
IP:                10.185.148.249
Port:              http  80/TCP
TargetPort:        http/TCP
Endpoints:         10.80.7.6:9090
Session Affinity:  None
Events:            <none>

The error is only visible after checking the logs of the notification controller service itself:

edef@jaguar ~/s/m/p/e/p/gke ((f478031a…))> kubectl logs -n flux-system svc/notification-controller | tail -n1
{"level":"error","ts":"2020-11-06T15:52:36.607Z","logger":"event-server","msg":"failed to initialise provider","provider":{"namespace":"flux-system","name":"slack"},"type":"slack","error":"empty Slack channel"}

Alert EventSeverity has no default value

The documentation states that if EventSeverity is not set it will default to info. I cannot find any code that sets the default value.

// Filter events based on severity, defaults to ('info').
// +kubebuilder:validation:Enum=info;error
// +optional
EventSeverity string `json:"eventSeverity,omitempty"`

Additionally the check for EventSeverity has an undocumented behavior. As I understand it an Alert will be sent either if the EventSeverity of the event matches the alert. Or if the Alert severity is of type info. If this is the intended behavior it should be documented that setting the severity to info will also include alerts with severity error.

notification-controller/internal/server/event_handlers.go

Lines 80 to 81 in 20f07c8

	if event.Severity == alert.Spec.EventSeverity ||
	alert.Spec.EventSeverity == recorder.EventSeverityInfo {

[Security] Potential Secret Leak

It has been noticed that while using fossa-contrib/fossa-action@v1 your fossa-api-key 5ee8bf422db1471e0bcf2bcb289185de is present in plaintext. Please ensure that secrets are encrypted or not passed as plain text in github workflows.

Notifications of errors and recoveries

It would be nice if we could have a setting which would allow to be notified when:

• an error occurs after previous apply was successful.
• an apply is successful after previous one was in error.

This would limit notification traffic to what's really important.

Regards.

Add possibility to configure wildcard namespace value in Alert eventSources

It would be nice to pick up all error events within a cluster using a wildcard namespace value in the eventSources section of an Alert - similar to how it works for name:

notification-controller/internal/server/event_handlers.go

Line 77 in fbf1ea0

if (source.Name == "*" || event.InvolvedObject.Name == source.Name) &&

Currently you'll either need to duplicate Provider (including any secrets) and Alert resources to all relevant namespaces or create a "global" Alert which references other namespaces. It would be simpler and less error prone to allow namespace: "*".

Add support for notification templates

When working with a notification, an established format has been created, which doesn't provide much flexibility in the message notification.
This is especially relevant for notifiers that support markdown, yet it's set to a default of text:
https://github.com/fluxcd/notification-controller/blob/main/internal/notifier/discord.go#L96
https://github.com/fluxcd/notification-controller/blob/main/internal/notifier/rocket.go#L88
https://github.com/fluxcd/notification-controller/blob/main/internal/notifier/slack.go#L109

Can we create some common template struct to allow more flexibility in the message displayed?