kubelet does not export metric `kubelet_volume_stats_capacity_bytes` about kind

[BenTheElder]

This is probably due to #3360

[irizzant]

Thanks for the updates, can you please detail how?

[BenTheElder]

A featuregate is enabled currently to disable local storage isolation, because enabling it broke kind on some hosts.

There's more discussion in the linked PR and transitively linked issues.

You can try this config to see if that PR would solve it:
#3360 (comment)

[irizzant]

Thank you again for the updates.
I see localStorageCapacityIsolation option graduated to GA in k8s v1.25 so it's enabled by default.

As far as I understand that option was disabled in kind because it broke kubernetes in some scenarios, but I still don't understand the connection with the missing metric because as far as I can see the option is about limiting the ephemeral storage in LimitRange and Pod specs.

Am I missing anything?

[irizzant]

I've just tried with this config

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: KubeletConfiguration
    localStorageCapacityIsolation: true
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  - |
    kind: ClusterConfiguration
    controllerManager:
      extraArgs:
        bind-address: 0.0.0.0
    etcd:
      local:
        extraArgs:
          listen-metrics-urls: http://0.0.0.0:2381
    scheduler:
      extraArgs:
        bind-address: 0.0.0.0
  - |
    kind: KubeProxyConfiguration
    metricsBindAddress: 0.0.0.0
  extraMounts:
  - containerPath: /var/lib/kubelet/config.json
    hostPath: "$HOME/.docker/config.json"
  extraPortMappings:
    - containerPort: 443
      hostPort: 443
    - containerPort: 80
      hostPort: 80
- role: worker
- role: worker
- role: worker

still no metric available!

[BenTheElder]

kind is shipping kubelet from upstream sources, I was guessing that this metric is gated behind the same code in kubelet because it's related to tracking filesystem stats. We're not modifying kubelet.

That config should be equivilant enough because kubeadm currently cluster-scopes kubelet config, but preferably it should be explicitly cluster scoped in case kind fixes this in the future (so kubeadmConfigPatches at the top level not under one of the nodes)

It would also be helpful to test this with a minimal configuration for reproducing purposes.

Note: I just looked up this metric and it's considered alpha, and related to PVs.

https://kubernetes.io/docs/reference/instrumentation/metrics/

The other thought is this may not work with PVs from https://github.com/rancher/local-path-provisioner

What's your use case?

[irizzant]

Here you have the reproducer, just run the script and port-forward Grafana in monitoring namespace to be able to see that the metric is missing.
reproducer.zip

Using this reproducer I launched a kube-proxy and then issued this curl on a worker node:
curl -X GET http://127.0.0.1:8001/api/v1/nodes/<node-name>/proxy/configz | jq .

to verify that the localStorageCapacityIsolation: true was there and the metric still missing