New Sink: Telegram about robusta

[aantn]

Got it, so it would be OK do this in two parts, right?

Part 1: send messages to Telegram without two-way interactivity

Part 2: enable the fix buttons with two-way interactivity

[arikalon1]

By default, we omit the 'replicas' changes, because on clusters with HPA (Horizontal Pod Autoscaler), it might become spammy.
If you do want to see it, you need to override the default 'omitted_fields' parameter:
Just add it to the action parameters:
omitted_fields: ["status", "metadata.generation","metadata.resourceVersion","metadata.managedFields"]
These are the defaults, just without the 'replicas' field.

[arikalon1]

Sounds like a good solution for me.
Will fix it soon.
Thanks for the input

[arikalon1]

Thanks for bringing it up.
The resource name is taken from the deployment metadata name and namespace.
What are the metadata name and namespace for the changed deployment?

[arikalon1]

This might be happening, if you add the entire metadata object to the omitted fields.
I recommend to use this as omitter fields, as a workaround:
omitted_fields: ["status", "metadata.generation","metadata.resourceVersion","metadata.managedFields"]
Opening a separate issue to report the correct name, even when the whole 'metadata' section is omitted.

[arikalon1]

Thank you.
It might be a bug indeed.
Will check and update.

[arikalon1]

We already support multiple sinks, however while trying to test it I discovered some bug there.
Opening another issue and will fix it shortly

[aantn]

Yes, we can add it!

Would you be able to help beta test this?

[tuananh2508]

Absolutely

[aantn]

Cool, it'll be another few days before it's ready. I'll let you know here.

Which Robusta features are you most interested in? I'll make sure we test those specific features on our end specifically for the Telegram integration.

[tuananh2508]

@aantn Aside from the basic things, I think the "ChatOps" feature is pretty cool.

[aantn]

@tuananh2508 Cool, that helps. It means we'll need to add support to the Telegram sink for two-way-interactivity.

@arikalon1 FYI

[tuananh2508]

Yeah tks, but the most important things to us is still the core function of Robusta - Alerting :D. Hope that feature is going to be release soon.

[tuananh2508]

Yes, love it. I will try to be most supportive for this feature

[tuananh2508]

Thanks for all the hard work. We're on holiday so I will report the result or any bugs next week.

[arikalon1]

Gladly. We'd love to hear any feedback you have.

[tuananh2508]

Till now everything is working great for the first stage as mentioned above. But for the resources_babysitter actions, I have to specify omited and monitor field. If I don't specify it, robusta sometime doesn't notify me about some changes.

[tuananh2508]

Also the NodeEnricher is not working right now.

[arikalon1]

Can you please send the out of 'robusta logs', after triggering the node enricher?

[tuananh2508]

@arikalon1

Logs after I used the command in the picture: https://pikab.in/78788d9feb

[arikalon1]

Do you know if there are pods running on that specific node?

[tuananh2508]

There are 18 pods running on that node on different namespaces:

[arikalon1]

I looked into this and figured out the problem.

Currently we don't send tables to the telegram sink, because the tables are too wide to be rendered properly on a mobile device. The output of this playbook is in a table, and that's the reason you don't see it in telegram.

Any ideas for how to fix this? We could add a placeholder like "" or we could find alternative way to render the tables so it looks OK. Open to ideas.

[tuananh2508]

I think for Telegram, maybe this kind of table can be paste to a .txt file. Just like what you're doing right now with Crash Pod event.

[tuananh2508]

Hope to hear from you soon ! 👯

[tuananh2508]

@arikalon1 Another problem when I used resources_babysitter for a specific application. When there are some changes to the application, Robusta did notify me but It didn't tell me which Application changed.

It's okay now because we only use resources_babysitter for 1 application so we know what happened. But there will be problems when we use resources_babysitter for 2 to 3 applications.

[tuananh2508]

This might be happening, if you add the entire metadata object to the omitted fields. I recommend to use this as omitter fields, as a workaround: omitted_fields: ["status", "metadata.generation","metadata.resourceVersion","metadata.managedFields"] Opening a separate issue to report the correct name, even when the whole 'metadata' section is omitted.

Thanks, It's worked now. Also I don't know if this is a bug or not, but you can't use auto generated labels from K8s such as app.kubernetes.io/instance: wonder-application as a LabelSelector for resources_babysitter

[arikalon1]

Can you share the output of kubectl get deployment -o yaml .. for the relevant deployment?

[tuananh2508]

Can you share the output of kubectl get deployment -o yaml .. for the relevant deployment?

→ k get deployments.apps -n staging wonder-app -o yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: "2022-02-11T04:03:22Z"
  generation: 13
  labels:
    app: wonder-app <- this one a manually add it
    app.kubernetes.io/instance: hn-staging-wonder-app
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: wonder-app
    app.kubernetes.io/version: 1.16.0
    helm.sh/chart: wonder-app-0.1.0
....

At first I was trying to use the app.kubernetes.io/instance: hn-staging-wonder-app label. Cause we're using ArgoCD and K8s would create some default labels for us. But after numerous attemps, I have to add a label myself. Doesn't know if this happened to anyone else.

[arikalon1]

We couldn't replicate it locally.
Setting the following did trigger the change notification:

- triggers:
  - on_deployment_all_changes:
      labels_selector: app.kubernetes.io/instance=hn-staging-wonder-app
  actions:
  - resource_babysitter:
      omitted_fields:
        - metadata

Are you sure the label appears in the Deployment metadata, and not in the Pod template?

[tuananh2508]

Yeah pretty sure It was deployment labels. I got it through this command:

kubectl get deploy -n staging --show-labels | grep wonder
cloud-dns-backend                              3/3     3            3           3d5h   app.kubernetes.io/instance=hn-staging-wonder-app, ....

Anyway, If no one else have this maybe this is just a misconfiguration by me. Cheer guys.

Another question, Do you guys plan to support multiple telegram sink ? Till now if I defined 2 telegram group to send the notification to then 1 group is going to be deleted by Robusta. It would be great to have this feature. Cause in some case we would want to send general notify to Group1 and detailed notify to Group2. Right now all the notify can only goes in the same group.

[arikalon1]

Added tables as files on telegram sinks