sylr / alertmanager-splunkbot

alertmanager alerts to splunk

alertmanager-splunkbot

Forwarding alerts sent by prometheus alertmanager to splunk in order to understand the logic ...

Alertmanager splunkbot k8s container:

      - name: prometheus-alertmanager-splunkbot
        image: quay.io/sylr/alertmanager-splunkbot:v0.0.9
        args:
        - -v
        - --insecure
        env:
        - name: SPLUNKBOT_LISTENING_ADDRESS
          value: 127.0.0.1
        - name: SPLUNKBOT_LISTENING_PORT
          value: "44553"
        - name: SPLUNKBOT_SPLUNK_URL
          value: https://10.101.0.46/services/collector/event/1.0
        - name: SPLUNKBOT_SPLUNK_TOKEN
          valueFrom:
            secretKeyRef:
              key: splunk-token
              name: splunkbot-secrets
        imagePullPolicy: IfNotPresent
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File

Alertmanager config:

receivers:
  webhook_configs:
  - send_resolved: true
    url: http://127.0.0.1:44553

alertmanager-splunkbot's Issues

Bad Request 400 from splunkbot service in K8s

I recently upgraded to the latest prometheus-community https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack and the alertmanager webhook is unable to post to the splunkbot service. Following is the message reported in the alertmanger pod:
level=error ts=2020-12-01T19:04:34.242Z caller=dispatch.go:309 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="splunk_webhook/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 400: http://alertmanager-splunkbot-service:44553"

Following is the config defined in the values.yaml for the splunk_webhook which works in the deprecated prometheus-operator

config:
global:
resolve_timeout: 5m
route:
group_by: [alertname]
group_wait: 30s
group_interval: 5m
repeat_interval: 1h
receiver: 'splunk_webhook'
routes:
- receiver: 'splunk_webhook'
continue: true

receivers:
- name: 'splunk_webhook'
  webhook_configs:
  - url: 'http://alertmanager-splunkbot-service:44553'
    send_resolved: true
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['instance']