openebs-archive / monitor-pv

custom stats collector for OpenEBS persistent volumes

monitor-pv

custom stats collector for OpenEBS persistent volumes (jiva, localpv)

Pre-requisite

• openebs (https://docs.openebs.io/)

How to use

• Run kubectl apply -f node-exporter-pv-metrics.yaml
• Verify if the pods are up and running.

$ kubectl get pods -n openebs 
NAME                                                              READY   STATUS    RESTARTS   AGE
cspc-operator-6c4cc7c64d-698ps                                    1/1     Running   0          6d23h
cvc-operator-77d749c559-9phff                                     1/1     Running   0          6d23h
maya-apiserver-5fb947d74d-r9skp                                   1/1     Running   0          6d23h
monitor-pv-bvzgv                                                  2/2     Running   0          8s
monitor-pv-nk76b                                                  2/2     Running   0          11s
monitor-pv-wnvp5                                                  2/2     Running   0          8s
openebs-admission-server-6c4b4998f8-zcg9n                         1/1     Running   0          6d23h
openebs-localpv-provisioner-5b744fc789-5wr8d                      1/1     Running   0          6d23h
openebs-ndm-g474w                                                 1/1     Running   0          6d23h
openebs-ndm-k2nnp                                                 1/1     Running   0          6d23h
openebs-ndm-operator-b58c79cc5-z8zw6                              1/1     Running   1          6d23h
openebs-ndm-rwzrb                                                 1/1     Running   0          6d23h
openebs-provisioner-54d45b55db-rt5rv                              1/1     Running   0          6d23h
openebs-snapshot-operator-6d4f5d7688-6g7zw                        2/2     Running   0          6d23h
pvc-dd03f0ae-731c-4f78-bdbf-86485f32ab3d-ctrl-89b44f6cb-pbnmk     2/2     Running   0          40h
pvc-dd03f0ae-731c-4f78-bdbf-86485f32ab3d-rep-1-857b65c68d-qrdx9   1/1     Running   0          40h
pvc-dd03f0ae-731c-4f78-bdbf-86485f32ab3d-rep-2-58c4f54f7-m6n45    1/1     Running   0          40h

How it works

The monitor PV daemonset pods consist of two containers i.e node-exporter and monitor-pv. The monitor-pv collects the PV size and PV utilization information & places it in a file on the shared mount. The node exporter uses its text-file collector to expose this data as metrics.

It exposes two metrics pv_capacity_bytes and pv_utilization_bytes.

Prometheus Configuration

To scrape the metrics in prometheus add this configuration in prometheus configuration file:

    - job_name: 'monitor-pv'
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: monitor-pv
        action: keep

If you want to scrape only monitor-pv metrics and drop all other node-exporter metrics use the below configuration:

    - job_name: 'monitor-pv'
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: monitor-pv
        action: keep
      metric_relabel_configs:
      - source_labels: [__name__]
        regex: '(pv_capacity_bytes|pv_utilization_bytes)'
        action: keep

Example:

monitor-pv's Issues

Hacktoberfest?

Would you like to opt-in to Hacktoberfest (2022)?

Would be fun to try tackle the #9 issue or something like that in the name of Hacktoberfest.

Or is https://github.com/openebs/m-exporter the sequel to this repo? Is this monitor-pv repo obsolete?

Fix "period of no metrics" in monitor-pv exporter

Both the functions collect_pv_capacity_metrics() & collect_pv_utilization_metrics() first clear the respective textfiles holding the metrics info before recreating them with latest updated info. However, the procedure to derive these metrics involves kubectl commands and/or system calls with some output parsing. Under cases where these take more than a few seconds & if the prometheus scrape interval is such that it coincides with this state of the textfiles (empty), then the dashboards are prone to show a few seconds worth of "no-metrics". This manifests itself like shown below:

This should be fixed. The changes may occur in two phases:

• The existing bash script will be updated to replace metrics over clearing files
• The collection approach maybe rewritten in golang

monitor-pv behaviour across different OpenEBS storage engines

Test Params:

• Sample size is 2 GB.
• Sample active data size is 4 MB.
• Source command is dd.

Results:

	Monitor PV	maya exporter	kubelet metrics
	(pv_utilization_bytes)	(openebs_actual_used)	(kubelet_volume_capacity_bytes - kubelet_volume_available_bytes)
Jiva	3.9258 MB	69.0703 MB	25.9141 MB
Local PV Hostpath	1.2142 GB
	(Sample size is 1GB + 220MB)
Local PV device	2.4079 GB
	(Sample size is 2GB + 418MB)

Notes:

• Jiva shows more utilization as can be seen: two possible reasons:

  • Filesystem metadata upon format (however, this additional usage seems more for a jiva vol compared to a higer sized raw disk attached to the VM. Needs confirmation from Sumit)
  • Smaller blocks treated as 4k blocks/packing of blocks (however, there is some fs optimization activity here which minimizes this impact for contiguous sequential workload like dd, v/s staggered/random writes)
  • Might be worth trying out comparing different workloads across local-pv / jiva from same relative start-point (i.e., 20K local PV = 65 MB from jiva and see if same increase in util is observed)
  • All above questions are important if monitor-pv is used as metrics source for both jiva as well as local PV.

• In case of Local PV hostpath & device, monitor-pv mirrors expected app usage via du.

Thoughts based on above observations:

• It is better to rely on different metrics sources for different PVs (jiva, cstor, local pv, zfs local pv) -- i.e. grafana panels uses different queries against diff metrics sources.

• For jiva/cstor where there is possibility of much divergence b/w user/app usage and actual usage, we can show different graphs or lines to highlight the difference and set the right expectation.

Rewrite the monitor-pv in golang

• Better error handling
• Granular control to enable per volume collection
• Scalable, if we feed custom info into exporter

Project dead?

Have OpenEBS team forgotten about this, or do not deem it worth further development?

We are making good use of this, but the project could use some updating. For instance, the Docker image has 128 high severity vulnerabilities (at the time of writing)

Add ability to monitor the utilization parent/backing store for Jiva

Right now the collector is purely PV based, however can be tuned to derive other utilization info too.

Fix Trivy download steps in buildscripts to retry/increase timeout

Travis builds sometimes are seen to fail with inability to pull the Trivy release artifact. This needs to be handled to retry & succeed. Adoption of alternate sources to pull Trivy can also be looked at.

Make path filters for mount locations configurable via ENV

Ref: #2 (comment)