knative-extensions / monitoring

Monitoring support for Knative

Knative Monitoring

This repository contains Grafana Dashboards and Prometheus Scraping configurations to monitor Knative.

It has been tested with the following configuration:

• k8s 1.19
• Istio 1.9
• kube-prometheus-stack v16.9 https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
• knative 0.23

Also, kube-state-metrics < v2.1 doesn't export all labels by default which means you must pass the following argument to kube-state-metrics:

- --metric-labels-allowlist=pods=[*],deployments=[app.kubernetes.io/name,app.kubernetes.io/component,app.kubernetes.io/instance]

Once kube-state-metrics has been adjusted, you will need to deploy the following servicemonitors and import the dashboards in the grafana folder.

kubectl apply -f https://raw.githubusercontent.com/knative-sandbox/monitoring/main/servicemonitor.yaml

monitoring's Issues

Label value shows None

This needs to be label_values(controller_reconcile_count{reconciler=~".+"}, reconciler) otherwise template variable shows None. Here is after the fix:

Notice however that there is panel row named Row title which has zero panels in, needs to be removed.

Originally posted by @skonto in #1 (comment)

Metric has been removed

This metric has been removed knative/serving@888e492. For an updated list check here: https://knative.dev/docs/serving/metrics/

Originally posted by @skonto in #1 (comment)

Cannot collect the request Information

I deploy the grafana and the prometheus following https://knative.dev/docs/serving/observability/metrics/collecting-metrics/ on my local minikube.

I hit the service using temporary DNS

curl -H "Host:greeter-10.default.example.com" http://10.104.139.27

But on the web concole, I cannot collect the date of request information.

Am I missing someting?

kube-prometheus-stack and Grafana dashboards

Hi,

I think I followed the docs correctly, however Knative dashboads do not show up in Grafana after applying https://raw.githubusercontent.com/knative-sandbox/monitoring/main/grafana/dashboards.yaml (cf. https://github.com/knative-sandbox/monitoring/tree/main/grafana).

Note that I can import the JSON files by hand and get working dashboards. I'd like to be able to automate that part.

Here is my procedure:

# Install the Prometheus Operator through Helm
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack -n default -f values.yaml

# Apply the ServiceMonitors/PodMonitors to collect metrics from Knative
kubectl apply -f https://raw.githubusercontent.com/knative-sandbox/monitoring/main/servicemonitor.yaml

# Import Knative Grafana dashboards
kubectl apply -f https://raw.githubusercontent.com/knative-sandbox/monitoring/main/grafana/dashboards.yaml

Here are the contents of my values.yaml file:

kube-state-metrics:
  metricLabelsAllowlist:
   - pods=[*]
   - deployments=[app.kubernetes.io/name,app.kubernetes.io/component,app.kubernetes.io/instance]
prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
grafana:
  sidecar:
    dashboards:
      enabled: true
      searchNamespace: ALL

Any help would be appreciated.

Thanks!

Wrong DataSource shown when editing

This is my control plane efficiency dashboard, when I import

when I click on edit in one of the panels, I don't see the correct data source selected. Same for other dashboards as well.

Originally posted by @itsmurugappan in #1 (comment)

Include a helm values.yaml to use with the kube-prometheus-stack

I got tripped up on this and I wasn't configuring the stack correctly:

see: #1 (comment)

Recent changes that caused metrics collection to fail

Hi, I believe a recent PR: #21 have caused the prometheus metrics collection to fail.

I was following the instruction on https://knative.dev/docs/eventing/observability/metrics/collecting-metrics/ but then I realized that the knative metrics do not appear on the dashboard. After hours of debugging, I noticed that the service monitors have configured their port to "metrics", but the actual services(i.e. activator), exposes their metrics on a port with "http-metrics".

I was able to get the metrics collected by manually patching the monitor with the correct port.

@zchenyu

When there are no requests it prints no namespace for the template variable:

When there are no requests it prints no namespace for the template variable:

However I had a service running without any requests hitting it. Namespace could cover this case too if i was based on (not a big issue), probably we should list the service and show no data as before:

"query": "label_values(kube_pod_labels{label_serving_knative_dev_configuration=~\".+\"}, namespace)",
          

Originally posted by @skonto in #1 (comment)

`Key` does not exist in currently emitted metrics.

Key does not exist in currently emitted metrics.
Here is a sample of what controller emits:

controller_reconcile_count{namespace_name="default",reconciler="knative.dev.serving.pkg.reconciler.gc.reconciler",success="true"} 11

Originally posted by @skonto in #1 (comment)