GithubHelp home page GithubHelp logo

"kubectl top nodes" reports "unknown" when executing multiple concurrent “kubectl exec” requests against a pod running on a Windows node about kubernetes HOT 6 OPEN

junjiezhang1997 avatar junjiezhang1997 commented on June 16, 2024
"kubectl top nodes" reports "unknown" when executing multiple concurrent “kubectl exec” requests against a pod running on a Windows node

from kubernetes.

Comments (6)

k8s-ci-robot avatar k8s-ci-robot commented on June 16, 2024

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

from kubernetes.

brianpursley avatar brianpursley commented on June 16, 2024

I tried this w/ Linux nodes and did not experience the problem, so I think this problem is specific to Windows nodes.

/retitle "kubectl top nodes" reports "unknown" when executing multiple concurrent “kubectl exec” requests against a pod running on a Windows node

/sig windows

@junjiezhang1997 Can you run kubectl top nodes -v9 to get verbose log output? There might be something in there that could help identify what is causing this problem.

from kubernetes.

junjiezhang1997 avatar junjiezhang1997 commented on June 16, 2024

@brianpursley

I repro this issue and run kubectl top nodes -v9. The logs are as follows:

I0513 10:23:15.423644 3175933 loader.go:373] Config loaded from file:  /home/junjiezhang/.kube/config
I0513 10:23:15.425219 3175933 round_trippers.go:466] curl -v -XGET  -H "User-Agent: kubectl/v1.27.1 (linux/amd64) kubernetes/abfec7d" -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "Authorization: Bearer <masked>" 'https://test-clust-junjiezhang-test-8ecadf-42z3zi6z.hcp.eastus.azmk8s.io:443/api'
I0513 10:23:15.427724 3175933 round_trippers.go:495] HTTP Trace: DNS Lookup for test-clust-junjiezhang-test-8ecadf-42z3zi6z.hcp.eastus.azmk8s.io resolved to [{4.255.115.91 }]
I0513 10:23:15.653081 3175933 round_trippers.go:510] HTTP Trace: Dial to tcp:4.255.115.91:443 succeed
I0513 10:23:16.134266 3175933 round_trippers.go:553] GET https://test-clust-junjiezhang-test-8ecadf-42z3zi6z.hcp.eastus.azmk8s.io:443/api 200 OK in 709 milliseconds
I0513 10:23:16.134306 3175933 round_trippers.go:570] HTTP Statistics: DNSLookup 2 ms Dial 225 ms TLSHandshake 252 ms ServerProcessing 228 ms Duration 709 ms
I0513 10:23:16.134319 3175933 round_trippers.go:577] Response Headers:
I0513 10:23:16.134334 3175933 round_trippers.go:580]     Cache-Control: public
I0513 10:23:16.134346 3175933 round_trippers.go:580]     Content-Type: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList
I0513 10:23:16.134357 3175933 round_trippers.go:580]     Etag: "7E2E4FBD8CDC884130EBFEF64757F6BC507A8EA6A2EA8F94E05F1B6F207B4E183C63D6D6EA5513FDDC10FC16233248F933DC787C288493D52745A3FAFD68CCD8"
I0513 10:23:16.134369 3175933 round_trippers.go:580]     Vary: Accept
I0513 10:23:16.134381 3175933 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: c63bbe1a-37b9-4027-a25b-c0af27c789f9
I0513 10:23:16.134392 3175933 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 52025469-7ffe-46a6-8b56-59ca942f158b
I0513 10:23:16.134400 3175933 round_trippers.go:580]     Date: Mon, 13 May 2024 10:23:16 GMT
I0513 10:23:16.134413 3175933 round_trippers.go:580]     Audit-Id: a6c10bfc-8ad3-4c62-af55-a3bf6bba25ba
I0513 10:23:16.134531 3175933 request.go:1188] Response Body: {"kind":"APIGroupDiscoveryList","apiVersion":"apidiscovery.k8s.io/v2beta1","metadata":{},"items":[{"metadata":{"creationTimestamp":null},"versions":[{"version":"v1","resources":[{"resource":"bindings","responseKind":{"group":"","version":"","kind":"Binding"},"scope":"Namespaced","singularResource":"binding","verbs":["create"]},{"resource":"componentstatuses","responseKind":{"group":"","version":"","kind":"ComponentStatus"},"scope":"Cluster","singularResource":"componentstatus","verbs":["get","list"],"shortNames":["cs"]},{"resource":"configmaps","responseKind":{"group":"","version":"","kind":"ConfigMap"},"scope":"Namespaced","singularResource":"configmap","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["cm"]},{"resource":"endpoints","responseKind":{"group":"","version":"","kind":"Endpoints"},"scope":"Namespaced","singularResource":"endpoints","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["ep"]},{"resource":"events","responseKind":{"group":"","version":"","kind":"Event"},"scope":"Namespaced","singularResource":"event","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["ev"]},{"resource":"limitranges","responseKind":{"group":"","version":"","kind":"LimitRange"},"scope":"Namespaced","singularResource":"limitrange","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["limits"]},{"resource":"namespaces","responseKind":{"group":"","version":"","kind":"Namespace"},"scope":"Cluster","singularResource":"namespace","verbs":["create","delete","get","list","patch","update","watch"],"shortNames":["ns"],"subresources":[{"subresource":"finalize","responseKind":{"group":"","version":"","kind":"Namespace"},"verbs":["update"]},{"subresource":"status","responseKind":{"group":"","version":"","kind":"Namespace"},"verbs":["get","patch","update"]}]},{"resource":"nodes","responseKind":{"group":"","version":"","kind":"Node"},"scope":"Cluster","singularResource":"node","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["no"],"subresources":[{"subresource":"proxy","responseKind":{"group":"","version":"","kind":"NodeProxyOptions"},"verbs":["create","delete","get","patch","update"]},{"subresource":"status","responseKind":{"group":"","version":"","kind":"Node"},"verbs":["get","patch","update"]}]},{"resource":"persistentvolumeclaims","responseKind":{"group":"","version":"","kind":"PersistentVolumeClaim"},"scope":"Namespaced","singularResource":"persistentvolumeclaim","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["pvc"],"subresources":[{"subresource":"status","responseKind":{"group":"","version":"","kind":"PersistentVolumeClaim"},"verbs":["get","patch","update"]}]},{"resource":"persistentvolumes","responseKind":{"group":"","version":"","kind":"PersistentVolume"},"scope":"Cluster","singularResource":"persistentvolume","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["pv"],"subresources":[{"subresource":"status","responseKind":{"group":"","version":"","kind":"PersistentVolume"},"verbs":["get","patch","update"]}]},{"resource":"pods","responseKind":{"group":"","version":"","kind":"Pod"},"scope":"Namespaced","singularResource":"pod","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["po"],"categories":["all"],"subresources":[{"subresource":"attach","responseKind":{"group":"","version":"","kind":"PodAttachOptions"},"verbs":["create","get"]},{"subresource":"binding","responseKind":{"group":"","version":"","kind":"Binding"},"verbs":["create"]},{"subresource":"ephemeralcontainers","responseKind":{"group":"","version":"","kind":"Pod"},"verbs":["get","patch","update"]},{"subresource":"eviction","responseKind":{"group":"policy","version":"v1","kind":"Eviction"},"verbs":["create"]},{"subresource":"exec","responseKind":{"group":"","version":"","kind":"PodExecOptions"},"verbs":["create","get"]},{"subresource":"log","responseKind":{"group":"","version":"","kind":"Pod"},"verbs":["get"]},{"subresource":"portforward","responseKind":{"group":"","version":"","kind":"PodPortForwardOptions"},"verbs":["create","get"]},{"subresource":"proxy","responseKind":{"group":"","version":"","kind":"PodProxyOptions"},"verbs":["create","delete","get","patch","update"]},{"subresource":"status","responseKind":{"group":"","version":"","kind":"Pod"},"verbs":["get","patch","update"]}]},{"resource":"podtemplates","responseKind":{"group":"","version":"","kind":"PodTemplate"},"scope":"Namespaced","singularResource":"podtemplate","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"resource":"replicationcontrollers","responseKind":{"group":"","version":"","kind":"ReplicationController"},"scope":"Namespaced","singularResource":"replicationcontroller","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["rc"],"categories":["all"],"subresources":[{"subresource":"scale","responseKind":{"group":"autoscaling","version":"v1","kind":"Scale"},"verbs":["get","patch","update"]},{"subresource":"status","responseKind":{"group":"","version":"","kind":"ReplicationController"},"verbs":["get","patch","update"]}]},{"resource":"resourcequotas","responseKind":{"group":"","version":"","kind":"ResourceQuota"},"scope":"Namespaced","singularResource":"resourcequota","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["quota"],"subresources":[{"subresource":"status","responseKind":{"group":"","version":"","kind":"ResourceQuota"},"verbs":["get","patch","update"]}]},{"resource":"secrets","responseKind":{"group":"","version":"","kind":"Secret"},"scope":"Namespaced","singularResource":"secret","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"resource":"serviceaccounts","responseKind":{"group":"","version":"","kind":"ServiceAccount"},"scope":"Namespaced","singularResource":"serviceaccount","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["sa"],"subresources":[{"subresource":"token","responseKind":{"group":"authentication.k8s.io","version":"v1","kind":"TokenRequest"},"verbs":["create"]}]},{"resource":"services","responseKind":{"group":"","version":"","kind":"Service"},"scope":"Namespaced","singularResource":"service","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["svc"],"categories":["all"],"subresources":[{"subresource":"proxy","responseKind":{"group":"","version":"","kind":"ServiceProxyOptions"},"verbs":["create","delete","get","patch","update"]},{"subresource":"status","responseKind":{"group":"","version":"","kind":"Service"},"verbs":["get","patch","update"]}]}],"freshness":"Current"}]}]}
I0513 10:23:16.135217 3175933 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.27.1 (linux/amd64) kubernetes/abfec7d" -H "Authorization: Bearer <masked>" 'https://test-clust-junjiezhang-test-8ecadf-42z3zi6z.hcp.eastus.azmk8s.io:443/apis'
I0513 10:23:16.363471 3175933 round_trippers.go:553] GET https://test-clust-junjiezhang-test-8ecadf-42z3zi6z.hcp.eastus.azmk8s.io:443/apis 200 OK in 228 milliseconds
I0513 10:23:16.363505 3175933 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 228 ms Duration 228 ms
I0513 10:23:16.363524 3175933 round_trippers.go:577] Response Headers:
I0513 10:23:16.363543 3175933 round_trippers.go:580]     Cache-Control: public
I0513 10:23:16.363563 3175933 round_trippers.go:580]     Content-Type: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList
I0513 10:23:16.363581 3175933 round_trippers.go:580]     Etag: "F3CA980379F4458FDC5E712C487718213D325C01671AC580CFDF3CE5483D8E326404020681338D0251257E9D296F0F876D6CA648937AD3785D79557DE34A9BD2"
I0513 10:23:16.363603 3175933 round_trippers.go:580]     Vary: Accept
I0513 10:23:16.363621 3175933 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: c63bbe1a-37b9-4027-a25b-c0af27c789f9
I0513 10:23:16.363639 3175933 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 52025469-7ffe-46a6-8b56-59ca942f158b
I0513 10:23:16.363658 3175933 round_trippers.go:580]     Date: Mon, 13 May 2024 10:23:16 GMT
I0513 10:23:16.363676 3175933 round_trippers.go:580]     Audit-Id: e023d35b-d6c9-4e3d-b75b-65d4c3a6b421
I0513 10:23:16.588368 3175933 request.go:1188] Response Body: {"kind":"APIGroupDiscoveryList","apiVersion":"apidiscovery.k8s.io/v2beta1","metadata":{},"items":[{"metadata":{"name":"apiregistration.k8s.io","creationTimestamp":null},"versions":[{"version":"v1","resources":[{"resource":"apiservices","responseKind":{"group":"","version":"","kind":"APIService"},"scope":"Cluster","singularResource":"apiservice","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"categories":["api-extensions"],"subresources":[{"subresource":"status","responseKind":{"group":"","version":"","kind":"APIService"},"verbs":["get","patch","update"]}]}],"freshness":"Current"}]},{"metadata":{"name":"apps","creationTimestamp":null},"versions":[{"version":"v1","resources":[{"resource":"controllerrevisions","responseKind":{"group":"","version":"","kind":"ControllerRevision"},"scope":"Namespaced","singularResource":"controllerrevision","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"resource":"daemonsets","responseKind":{"group":"","version":"","kind":"DaemonSet"},"scope":"Namespaced","singularResource":"daemonset","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["ds"],"categories":["all"],"subresources":[{"subresource":"status","responseKind":{"group":"","version":"","kind":"DaemonSet"},"verbs":["get","patch","update"]}]},{"resource":"deployments","responseKind":{"group":"","version":"","kind":"Deployment"},"scope":"Namespaced","singularResource":"deployment","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["deploy"],"categories":["all"],"subresources":[{"subresource":"scale","responseKind":{"group":"autoscaling","version":"v1","kind":"Scale"},"verbs":["get","patch","update"]},{"subresource":"status","responseKind":{"group":"","version":"","kind":"Deployment"},"verbs":["get","patch","update"]}]},{"resource":"replicasets","responseKind":{"group":"","version":"","kind":"ReplicaSet"},"scope":"Namespaced","singularResource":"replicaset","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["rs"],"categories":["all"],"subresources":[{"subresource":"scale","responseKind":{"group":"autoscaling","version":"v1","kind":"Scale"},"verbs":["get","patch","update"]},{"subresource":"status","responseKind":{"group":"","version":"","kind":"ReplicaSet"},"verbs":["get","patch","update"]}]},{"resource":"statefulsets","responseKind":{"group":"","version":"","kind":"StatefulSet"},"scope":"Namespaced","singularResource":"statefulset","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["sts"],"categories":["all"],"subresources":[{"subresource":"scale","responseKind":{"group":"autoscaling","version":"v1","kind":"Scale"},"verbs":["get","patch","update"]},{"subresource":"status","responseKind":{"group":"","version":"","kind":"StatefulSet"},"verbs":["get","patch","update"]}]}],"freshness":"Current"}]},{"metadata":{"name":"events.k8s.io","creationTimestamp":null},"versions":[{"version":"v1","resources":[{"resource":"events","responseKind":{"group":"","version":"","kind":"Event"},"scope":"Namespaced","singularResource":"event","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["ev"]}],"freshness":"Current"}]},{"metadata":{"name":"authentication.k8s.io","creationTimestamp":null},"versions":[{"version":"v1","resources":[{"resource":"selfsubjectreviews","responseKind":{"group":"","version":"","kind":"SelfSubjectReview"},"scope":"Cluster","singularResource":"selfsubjectreview","verbs":["create"]},{"resource":"tokenreviews","responseKind":{"group":"","version":"","kind":"TokenReview"},"scope":"Cluster","singularResource":"tokenreview","verbs":["create"]}],"freshness":"Current"},{"version":"v1beta1","resources":[{"resource":"selfsubjectreviews","responseKind":{"group":"","version":"","kind":"SelfSubjectReview"},"scope":"Cluster","singularResource":"selfsubjectreview","verbs":["create"]}],"freshness":"Current"}]},{"metadata":{"name":"authorization.k8s.io","creationTimestamp":null},"versions":[{"version":"v1","resources":[{"resource":"localsubjectaccessreviews","responseKind":{"group":"","version":"","kind":"LocalSubjectAccessReview"},"scope":"Namespaced","singularResource":"localsubjectaccessreview","verbs":["create"]},{"resource":"selfsubjectaccessreviews","responseKind":{"group":"","version":"","kind":"SelfSubjectAccessReview"},"scope":"Cluster","singularResource":"selfsubjectaccessreview","verbs":["create"]},{"resource":"selfsubjectrulesreviews","responseKind":{"group":"","version":"","kind":"SelfSubjectRulesReview"},"scope":"Cluster","singularResource":"selfsubjectrulesreview","verbs":["create"]},{"resource":"subjectaccessreviews","responseKind":{"group":"","version":"","kind":"SubjectAccessReview"},"scope":"Cluster","singularResource":"subjectaccessreview","verbs":["create"]}],"freshness":"Current"}]},{"metadata":{"name":"autoscaling","creationTimestamp":null},"versions":[{"version":"v2","resources":[{"resource":"horizontalpodautoscalers","responseKind":{"group":"","version":"","kind":"HorizontalPodAutoscaler"},"scope":"Namespaced","singularResource":"horizontalpodautoscaler","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["hpa"],"categories":["all"],"subresources":[{"subresource":"status","responseKind":{"group":"","version":"","kind":"HorizontalPodAutoscaler"},"verbs":["get","patch","update"]}]}],"freshness":"Current"},{"version":"v1","resources":[{"resource":"horizontalpodautoscalers","responseKind":{"group":"","version":"","kind":"HorizontalPodAutoscaler"},"scope":"Namespaced","singularResource":"horizontalpodautoscaler","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["hpa"],"categories":["all"],"subresources":[{"subresource":"status","responseKind":{"group":"","version":"","kind":"HorizontalPodAutoscaler"},"verbs":["get","patch","update"]}]}],"freshness":"Current"}]},{"metadata":{"name":"batch","creationTimestamp":null},"versions":[{"version":"v1","resources":[{"resource":"cronjobs","responseKind":{"group":"","version":"","kind":"CronJob"},"scope":"Namespaced","singularResource":"cronjob","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["cj"],"categories":["all"],"subresources":[{"subresource":"status","responseKind":{"group":"","version":"","kind":"CronJob"},"verbs":["get","patch","update"]}]},{"resource":"jobs","responseKind":{"group":"","version":"","kind":"Job"},"scope":"Namespaced","singularResource":"job","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"categories":["all"],"subresources":[{"subresource":"status","responseKind":{"group":"","version":"","kind":"Job"},"verbs":["get","patch","update"]}]}],"freshness":"Current"}]},{"metadata":{"name":"certificates.k8s.io","creationTimestamp":null},"versions":[{"version":"v1","resources":[{"resource":"certificatesigningrequests","responseKind":{"group":"","version":"","kind":"CertificateSigningRequest"},"scope":"Cluster","singularResource":"certificatesigningrequest","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["csr"],"subresources":[{"subresource":"approval","responseKind":{"group":"","version":"","kind":"CertificateSigningRequest"},"verbs":["get","patch","update"]},{"subresource":"status","responseKind":{"group":"","version":"","kind":"CertificateSigningRequest"},"verbs":["get","patch","update"]}]}],"freshness":"Current"}]},{"metadata":{"name":"networking.k8s.io","creationTimestamp":null},"versions":[{"version":"v1","resources":[{"resource":"ingressclasses","responseKind":{"group":"","version":"","kind":"IngressClass"},"scope":"Cluster","singularResource":"ingressclass","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"resource":"ingresses","responseKind":{"group":"","version":"","kind":"Ingress"},"scope":"Namespaced","singularResource":"ingress","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["ing"],"subresources":[{"subresource":"status","responseKind":{"group":"","version":"","kind":"Ingress"},"verbs":["get","patch","update"]}]},{"resource":"networkpolicies","responseKind":{"group":"","version":"","kind":"NetworkPolicy"},"scope":"Namespaced","singularResource":"networkpolicy","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["netpol"]}],"freshness":"Current"}]},{"metadata":{"name":"policy","creationTimestamp":null},"versions":[{"version":"v1","resources":[{"resource":"poddisruptionbudgets","responseKind":{"group":"","version":"","kind":"PodDisruptionBudget"},"scope":"Namespaced","singularResource":"poddisruptionbudget","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["pdb"],"subresources":[{"subresource":"status","responseKind":{"group":"","version":"","kind":"PodDisruptionBudget"},"verbs":["get","patch","update"]}]}],"freshness":"Current"}]},{"metadata":{"name":"rbac.authorization.k8s.io","creationTimestamp":null},"versions":[{"version":"v1","resources":[{"resource":"clusterrolebindings","responseKind":{"group":"","version":"","kind":"ClusterRoleBinding"},"scope":"Cluster","singularResource":"clusterrolebinding","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"resource":"clusterroles","responseKind":{"group":"","version":"","kind":"ClusterRole"},"scope":"Cluster","singularResource":"clusterrole","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"resource":"rolebindings","responseKind":{"group":"","version":"","kind":"RoleBinding"},"scope":"Namespaced","singularResource":"rolebinding","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"resource":"roles","responseKind":{"group":"","version":"","kind":"Role"},"scope":"Namespaced","singularResource":"role","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]}],"freshness":"Current"}]},{"metadat [truncated 18294 chars]
I0513 10:23:16.589355 3175933 round_trippers.go:466] curl -v -XGET  -H "Accept: application/vnd.kubernetes.protobuf, */*" -H "User-Agent: kubectl/v1.27.1 (linux/amd64) kubernetes/abfec7d" -H "Authorization: Bearer <masked>" 'https://test-clust-junjiezhang-test-8ecadf-42z3zi6z.hcp.eastus.azmk8s.io:443/apis/metrics.k8s.io/v1beta1/nodes'
I0513 10:23:16.818811 3175933 round_trippers.go:553] GET https://test-clust-junjiezhang-test-8ecadf-42z3zi6z.hcp.eastus.azmk8s.io:443/apis/metrics.k8s.io/v1beta1/nodes 200 OK in 229 milliseconds
I0513 10:23:16.818843 3175933 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 229 ms Duration 229 ms
I0513 10:23:16.818862 3175933 round_trippers.go:577] Response Headers:
I0513 10:23:16.818878 3175933 round_trippers.go:580]     Audit-Id: 0ad280de-994e-4617-be16-771d5d1f6a98
I0513 10:23:16.818899 3175933 round_trippers.go:580]     Audit-Id: 0ad280de-994e-4617-be16-771d5d1f6a98
I0513 10:23:16.818918 3175933 round_trippers.go:580]     Date: Mon, 13 May 2024 10:23:16 GMT
I0513 10:23:16.818935 3175933 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: c63bbe1a-37b9-4027-a25b-c0af27c789f9
I0513 10:23:16.818953 3175933 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 52025469-7ffe-46a6-8b56-59ca942f158b
I0513 10:23:16.818972 3175933 round_trippers.go:580]     Cache-Control: no-cache, private
I0513 10:23:16.818991 3175933 round_trippers.go:580]     Cache-Control: no-cache, private
I0513 10:23:16.819005 3175933 round_trippers.go:580]     Content-Type: application/vnd.kubernetes.protobuf
I0513 10:23:16.819035 3175933 round_trippers.go:580]     Content-Length: 1546
I0513 10:23:16.819120 3175933 request.go:1186] Response Body:
00000000  6b 38 73 00 0a 29 0a 16  6d 65 74 72 69 63 73 2e  |k8s..)..metrics.|
00000010  6b 38 73 2e 69 6f 2f 76  31 62 65 74 61 31 12 0f  |k8s.io/v1beta1..|
00000020  4e 6f 64 65 4d 65 74 72  69 63 73 4c 69 73 74 12  |NodeMetricsList.|
00000030  d4 0b 0a 06 0a 00 12 00  1a 00 12 c9 0b 0a 87 0b  |................|
00000040  0a 21 61 6b 73 2d 6e 6f  64 65 70 6f 6f 6c 31 2d  |.!aks-nodepool1-|
00000050  38 31 38 37 36 34 33 34  2d 76 6d 73 73 30 30 30  |81876434-vmss000|
00000060  30 30 30 12 00 1a 00 22  00 2a 00 32 00 38 00 42  |000....".*.2.8.B|
00000070  08 08 94 d4 87 b2 06 10  00 5a 16 0a 09 61 67 65  |.........Z...age|
00000080  6e 74 70 6f 6f 6c 12 09  6e 6f 64 65 70 6f 6f 6c  |ntpool..nodepool|
00000090  31 5a 20 0a 17 62 65 74  61 2e 6b 75 62 65 72 6e  |1Z ..beta.kubern|
000000a0  65 74 65 73 2e 69 6f 2f  61 72 63 68 12 05 61 6d  |etes.io/arch..am|
000000b0  64 36 34 5a 33 0a 20 62  65 74 61 2e 6b 75 62 65  |d64Z3. beta.kube|
000000c0  72 6e 65 74 65 73 2e 69  6f 2f 69 6e 73 74 61 6e  |rnetes.io/instan|
000000d0  63 65 2d 74 79 70 65 12  0f 53 74 61 6e 64 61 72  |ce-type..Standar|
000000e0  64 5f 44 53 32 5f 76 32  5a 1e 0a 15 62 65 74 61  |d_DS2_v2Z...beta|
000000f0  2e 6b 75 62 65 72 6e 65  74 65 73 2e 69 6f 2f 6f  |.kubernetes.io/o|
00000100  73 12 05 6c 69 6e 75 78  5a 32 0a 28 66 61 69 6c  |s..linuxZ2.(fail|
00000110  75 72 65 2d 64 6f 6d 61  69 6e 2e 62 65 74 61 2e  |ure-domain.beta.|
00000120  6b 75 62 65 72 6e 65 74  65 73 2e 69 6f 2f 72 65  |kubernetes.io/re|
00000130  67 69 6f 6e 12 06 65 61  73 74 75 73 5a 2b 0a 26  |gion..eastusZ+.&|
00000140  66 61 69 6c 75 72 65 2d  64 6f 6d 61 69 6e 2e 62  |failure-domain.b|
00000150  65 74 61 2e 6b 75 62 65  72 6e 65 74 65 73 2e 69  |eta.kubernetes.i|
00000160  6f 2f 7a 6f 6e 65 12 01  30 5a 2b 0a 1e 6b 75 62  |o/zone..0Z+..kub|
00000170  65 72 6e 65 74 65 73 2e  61 7a 75 72 65 2e 63 6f  |ernetes.azure.co|
00000180  6d 2f 61 67 65 6e 74 70  6f 6f 6c 12 09 6e 6f 64  |m/agentpool..nod|
00000190  65 70 6f 6f 6c 31 5a 47  0a 1c 6b 75 62 65 72 6e  |epool1ZG..kubern|
000001a0  65 74 65 73 2e 61 7a 75  72 65 2e 63 6f 6d 2f 63  |etes.azure.com/c|
000001b0  6c 75 73 74 65 72 12 27  4d 43 5f 6a 75 6e 6a 69  |luster.'MC_junji|
000001c0  65 7a 68 61 6e 67 2d 74  65 73 74 5f 74 65 73 74  |ezhang-test_test|
000001d0  2d 63 6c 75 73 74 65 72  5f 65 61 73 74 75 73 5a  |-cluster_eastusZ|
000001e0  5f 0a 37 6b 75 62 65 72  6e 65 74 65 73 2e 61 7a  |_.7kubernetes.az|
000001f0  75 72 65 2e 63 6f 6d 2f  63 6f 6e 73 6f 6c 69 64  |ure.com/consolid|
00000200  61 74 65 64 2d 61 64 64  69 74 69 6f 6e 61 6c 2d  |ated-additional-|
00000210  70 72 6f 70 65 72 74 69  65 73 12 24 35 61 61 65  |properties.$5aae|
00000220  61 35 35 36 2d 31 30 65  64 2d 31 31 65 66 2d 39  |a556-10ed-11ef-9|
00000230  34 66 31 2d 66 36 32 65  36 63 31 37 30 39 32 65  |4f1-f62e6c17092e|
00000240  5a 57 0a 2f 6b 75 62 65  72 6e 65 74 65 73 2e 61  |ZW./kubernetes.a|
00000250  7a 75 72 65 2e 63 6f 6d  2f 6b 75 62 65 6c 65 74  |zure.com/kubelet|
00000260  2d 69 64 65 6e 74 69 74  79 2d 63 6c 69 65 6e 74  |-identity-client|
00000270  2d 69 64 12 24 30 35 33  39 33 61 32 32 2d 65 34  |-id.$05393a22-e4|
00000280  63 65 2d 34 62 30 37 2d  62 66 65 30 2d 65 62 37  |ce-4b07-bfe0-eb7|
00000290  31 65 34 34 65 30 66 37  64 5a 23 0a 19 6b 75 62  |1e44e0f7dZ#..kub|
000002a0  65 72 6e 65 74 65 73 2e  61 7a 75 72 65 2e 63 6f  |ernetes.azure.co|
000002b0  6d 2f 6d 6f 64 65 12 06  73 79 73 74 65 6d 5a 53  |m/mode..systemZS|
000002c0  0a 27 6b 75 62 65 72 6e  65 74 65 73 2e 61 7a 75  |.'kubernetes.azu|
000002d0  72 65 2e 63 6f 6d 2f 6e  6f 64 65 2d 69 6d 61 67  |re.com/node-imag|
000002e0  65 2d 76 65 72 73 69 6f  6e 12 28 41 4b 53 55 62  |e-version.(AKSUb|
000002f0  75 6e 74 75 2d 32 32 30  34 67 65 6e 32 63 6f 6e  |untu-2204gen2con|
00000300  74 61 69 6e 65 72 64 2d  32 30 32 34 30 35 2e 30  |tainerd-202405.0|
00000310  33 2e 30 5a 3d 0a 22 6b  75 62 65 72 6e 65 74 65  |3.0Z=."kubernete|
00000320  73 2e 61 7a 75 72 65 2e  63 6f 6d 2f 6e 6f 64 65  |s.azure.com/node|
00000330  70 6f 6f 6c 2d 74 79 70  65 12 17 56 69 72 74 75  |pool-type..Virtu|
00000340  61 6c 4d 61 63 68 69 6e  65 53 63 61 6c 65 53 65  |alMachineScaleSe|
00000350  74 73 5a 25 0a 1b 6b 75  62 65 72 6e 65 74 65 73  |tsZ%..kubernetes|
00000360  2e 61 7a 75 72 65 2e 63  6f 6d 2f 6f 73 2d 73 6b  |.azure.com/os-sk|
00000370  75 12 06 55 62 75 6e 74  75 5a 22 0a 19 6b 75 62  |u..UbuntuZ"..kub|
00000380  65 72 6e 65 74 65 73 2e  61 7a 75 72 65 2e 63 6f  |ernetes.azure.co|
00000390  6d 2f 72 6f 6c 65 12 05  61 67 65 6e 74 5a 2e 0a  |m/role..agentZ..|
000003a0  23 6b 75 62 65 72 6e 65  74 65 73 2e 61 7a 75 72  |#kubernetes.azur|
000003b0  65 2e 63 6f 6d 2f 73 74  6f 72 61 67 65 70 72 6f  |e.com/storagepro|
000003c0  66 69 6c 65 12 07 6d 61  6e 61 67 65 64 5a 2f 0a  |file..managedZ/.|
000003d0  20 6b 75 62 65 72 6e 65  74 65 73 2e 61 7a 75 72  | kubernetes.azur|
000003e0  65 2e 63 6f 6d 2f 73 74  6f 72 61 67 65 74 69 65  |e.com/storagetie|
000003f0  72 12 0b 50 72 65 6d 69  75 6d 5f 4c 52 53 5a 1b  |r..Premium_LRSZ.|
00000400  0a 12 6b 75 62 65 72 6e  65 74 65 73 2e 69 6f 2f  |..kubernetes.io/|
00000410  61 72 63 68 12 05 61 6d  64 36 34 5a 3b 0a 16 6b  |arch..amd64Z;..k|
00000420  75 62 65 72 6e 65 74 65  73 2e 69 6f 2f 68 6f 73  |ubernetes.io/hos|
00000430  74 6e 61 6d 65 12 21 61  6b 73 2d 6e 6f 64 65 70  |tname.!aks-nodep|
00000440  6f 6f 6c 31 2d 38 31 38  37 36 34 33 34 2d 76 6d  |ool1-81876434-vm|
00000450  73 73 30 30 30 30 30 30  5a 19 0a 10 6b 75 62 65  |ss000000Z...kube|
00000460  72 6e 65 74 65 73 2e 69  6f 2f 6f 73 12 05 6c 69  |rnetes.io/os..li|
00000470  6e 75 78 5a 1b 0a 12 6b  75 62 65 72 6e 65 74 65  |nuxZ...kubernete|
00000480  73 2e 69 6f 2f 72 6f 6c  65 12 05 61 67 65 6e 74  |s.io/role..agent|
00000490  5a 21 0a 1d 6e 6f 64 65  2d 72 6f 6c 65 2e 6b 75  |Z!..node-role.ku|
000004a0  62 65 72 6e 65 74 65 73  2e 69 6f 2f 61 67 65 6e  |bernetes.io/agen|
000004b0  74 12 00 5a 33 0a 20 6e  6f 64 65 2e 6b 75 62 65  |t..Z3. node.kube|
000004c0  72 6e 65 74 65 73 2e 69  6f 2f 69 6e 73 74 61 6e  |rnetes.io/instan|
000004d0  63 65 2d 74 79 70 65 12  0f 53 74 61 6e 64 61 72  |ce-type..Standar|
000004e0  64 5f 44 53 32 5f 76 32  5a 19 0a 0e 73 74 6f 72  |d_DS2_v2Z...stor|
000004f0  61 67 65 70 72 6f 66 69  6c 65 12 07 6d 61 6e 61  |ageprofile..mana|
00000500  67 65 64 5a 1a 0a 0b 73  74 6f 72 61 67 65 74 69  |gedZ...storageti|
00000510  65 72 12 0b 50 72 65 6d  69 75 6d 5f 4c 52 53 5a  |er..Premium_LRSZ|
00000520  24 0a 20 74 6f 70 6f 6c  6f 67 79 2e 64 69 73 6b  |$. topology.disk|
00000530  2e 63 73 69 2e 61 7a 75  72 65 2e 63 6f 6d 2f 7a  |.csi.azure.com/z|
00000540  6f 6e 65 12 00 5a 33 0a  2e 74 6f 70 6f 6c 6f 67  |one..Z3..topolog|
00000550  79 2e 6b 75 62 65 72 6e  65 74 65 73 2e 61 7a 75  |y.kubernetes.azu|
00000560  72 65 2e 63 6f 6d 2f 73  75 62 2d 66 61 75 6c 74  |re.com/sub-fault|
00000570  2d 64 6f 6d 61 69 6e 12  01 30 5a 27 0a 1d 74 6f  |-domain..0Z'..to|
00000580  70 6f 6c 6f 67 79 2e 6b  75 62 65 72 6e 65 74 65  |pology.kubernete|
00000590  73 2e 69 6f 2f 72 65 67  69 6f 6e 12 06 65 61 73  |s.io/region..eas|
000005a0  74 75 73 5a 20 0a 1b 74  6f 70 6f 6c 6f 67 79 2e  |tusZ ..topology.|
000005b0  6b 75 62 65 72 6e 65 74  65 73 2e 69 6f 2f 7a 6f  |kubernetes.io/zo|
000005c0  6e 65 12 01 30 7a 00 12  08 08 d5 d3 87 b2 06 10  |ne..0z..........|
000005d0  00 1a 07 08 80 88 98 d3  e0 01 22 13 0a 03 63 70  |.........."...cp|
000005e0  75 12 0c 0a 0a 31 38 36  36 35 34 32 38 34 6e 22  |u....186654284n"|
000005f0  15 0a 06 6d 65 6d 6f 72  79 12 0b 0a 09 32 31 30  |...memory....210|
00000600  30 39 32 38 4b 69 1a 00  22 00                    |0928Ki..".|
I0513 10:23:16.819378 3175933 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.27.1 (linux/amd64) kubernetes/abfec7d" -H "Authorization: Bearer <masked>" 'https://test-clust-junjiezhang-test-8ecadf-42z3zi6z.hcp.eastus.azmk8s.io:443/api/v1/nodes'
I0513 10:23:17.048296 3175933 round_trippers.go:553] GET https://test-clust-junjiezhang-test-8ecadf-42z3zi6z.hcp.eastus.azmk8s.io:443/api/v1/nodes 200 OK in 228 milliseconds
I0513 10:23:17.048320 3175933 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 228 ms Duration 228 ms
I0513 10:23:17.048334 3175933 round_trippers.go:577] Response Headers:
I0513 10:23:17.048352 3175933 round_trippers.go:580]     Cache-Control: no-cache, private
I0513 10:23:17.048368 3175933 round_trippers.go:580]     Content-Type: application/json
I0513 10:23:17.048383 3175933 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: c63bbe1a-37b9-4027-a25b-c0af27c789f9
I0513 10:23:17.048400 3175933 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 52025469-7ffe-46a6-8b56-59ca942f158b
I0513 10:23:17.048416 3175933 round_trippers.go:580]     Date: Mon, 13 May 2024 10:23:16 GMT
I0513 10:23:17.048432 3175933 round_trippers.go:580]     Audit-Id: a78221cf-e726-4c11-aa62-66c21d97d1da
I0513 10:23:17.048642 3175933 request.go:1188] Response Body: {"kind":"NodeList","apiVersion":"v1","metadata":{"resourceVersion":"58839"},"items":[{"metadata":{"name":"aks-nodepool1-81876434-vmss000000","uid":"4c0e41c9-e79c-4f36-8765-25131f0f941f","resourceVersion":"57760","creationTimestamp":"2024-05-13T05:58:54Z","labels":{"agentpool":"nodepool1","beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/instance-type":"Standard_DS2_v2","beta.kubernetes.io/os":"linux","failure-domain.beta.kubernetes.io/region":"eastus","failure-domain.beta.kubernetes.io/zone":"0","kubernetes.azure.com/agentpool":"nodepool1","kubernetes.azure.com/cluster":"MC_junjiezhang-test_test-cluster_eastus","kubernetes.azure.com/consolidated-additional-properties":"5aaea556-10ed-11ef-94f1-f62e6c17092e","kubernetes.azure.com/kubelet-identity-client-id":"05393a22-e4ce-4b07-bfe0-eb71e44e0f7d","kubernetes.azure.com/mode":"system","kubernetes.azure.com/node-image-version":"AKSUbuntu-2204gen2containerd-202405.03.0","kubernetes.azure.com/nodepool-type":"VirtualMachineScaleSets","kubernetes.azure.com/os-sku":"Ubuntu","kubernetes.azure.com/role":"agent","kubernetes.azure.com/storageprofile":"managed","kubernetes.azure.com/storagetier":"Premium_LRS","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"aks-nodepool1-81876434-vmss000000","kubernetes.io/os":"linux","kubernetes.io/role":"agent","node-role.kubernetes.io/agent":"","node.kubernetes.io/instance-type":"Standard_DS2_v2","storageprofile":"managed","storagetier":"Premium_LRS","topology.disk.csi.azure.com/zone":"","topology.kubernetes.azure.com/sub-fault-domain":"0","topology.kubernetes.io/region":"eastus","topology.kubernetes.io/zone":"0"},"annotations":{"csi.volume.kubernetes.io/nodeid":"{\"disk.csi.azure.com\":\"aks-nodepool1-81876434-vmss000000\",\"file.csi.azure.com\":\"aks-nodepool1-81876434-vmss000000\"}","node.alpha.kubernetes.io/ttl":"0","volumes.kubernetes.io/controller-managed-attach-detach":"true"},"managedFields":[{"manager":"ccp-initializer","operation":"Update","apiVersion":"v1","time":"2024-05-13T05:58:54Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2024-05-13T05:58:54Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.azure.com/agentpool":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/consolidated-additional-properties":{},"f:kubernetes.azure.com/kubelet-identity-client-id":{},"f:kubernetes.azure.com/mode":{},"f:kubernetes.azure.com/node-image-version":{},"f:kubernetes.azure.com/nodepool-type":{},"f:kubernetes.azure.com/os-sku":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.azure.com/storageprofile":{},"f:kubernetes.azure.com/storagetier":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:storageprofile":{},"f:storagetier":{}}}}},{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2024-05-13T05:58:55Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}}},{"manager":"cloud-node-manager","operation":"Update","apiVersion":"v1","time":"2024-05-13T05:59:10Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.azure.com/sub-fault-domain":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2024-05-13T10:18:11Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.disk.csi.azure.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}},"subresource":"status"},{"manager":"node-problem-detector","operation":"Update","apiVersion":"v1","time":"2024-05-13T10:18:26Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainerRuntimeProblem\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FilesystemCorruptionProblem\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentContainerdRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentDockerRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentKubeletRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentUnregisterNetDevice\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"KernelDeadlock\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"KubeletProblem\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"ReadonlyFilesystem\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"VMEventScheduled\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}},"subresource":"status"}]},"spec":{"providerID":"azure:///subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/mc_junjiezhang-test_test-cluster_eastus/providers/Microsoft.Compute/virtualMachineScaleSets/aks-nodepool1-81876434-vmss/virtualMachines/0"},"status":{"capacity":{"cpu":"2","ephemeral-storage":"129886128Ki","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"7097620Ki","pods":"30"},"allocatable":{"cpu":"1900m","ephemeral-storage":"119703055367","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"4652308Ki","pods":"30"},"conditions":[{"type":"FrequentDockerRestart","status":"False","lastHeartbeatTime":"2024-05-13T10:18:26Z","lastTransitionTime":"2024-05-13T06:01:47Z","reason":"NoFrequentDockerRestart","message":"docker is functioning properly"},{"type":"FrequentContainerdRestart","status":"False","lastHeartbeatTime":"2024-05-13T10:18:26Z","lastTransitionTime":"2024-05-13T06:01:47Z","reason":"NoFrequentContainerdRestart","message":"containerd is functioning properly"},{"type":"FrequentUnregisterNetDevice","status":"False","lastHeartbeatTime":"2024-05-13T10:18:26Z","lastTransitionTime":"2024-05-13T06:01:47Z","reason":"NoFrequentUnregisterNetDevice","message":"node is functioning properly"},{"type":"KernelDeadlock","status":"False","lastHeartbeatTime":"2024-05-13T10:18:26Z","lastTransitionTime":"2024-05-13T06:01:47Z","reason":"KernelHasNoDeadlock","message":"kernel has no deadlock"},{"type":"ReadonlyFilesystem","status":"False","lastHeartbeatTime":"2024-05-13T10:18:26Z","lastTransitionTime":"2024-05-13T06:01:47Z","reason":"FilesystemIsNotReadOnly","message":"Filesystem is not read-only"},{"type":"FilesystemCorruptionProblem","status":"False","lastHeartbeatTime":"2024-05-13T10:18:26Z","lastTransitionTime":"2024-05-13T06:01:47Z","reason":"FilesystemIsOK","message":"Filesystem is healthy"},{"type":"ContainerRuntimeProblem","status":"False","lastHeartbeatTime":"2024-05-13T10:18:26Z","lastTransitionTime":"2024-05-13T06:01:47Z","reason":"ContainerRuntimeIsUp","message":"container runtime service is up"},{"type":"KubeletProblem","status":"False","lastHeartbeatTime":"2024-05-13T10:18:26Z","lastTransitionTime":"2024-05-13T06:01:47Z","reason":"KubeletIsUp","message":"kubelet service is up"},{"type":"FrequentKubeletRestart","status":"False","lastHeartbeatTime":"2024-05-13T10:18:26Z","lastTransitionTime":"2024-05-13T06:01:47Z","reason":"NoFrequentKubeletRestart","message":"kubelet is functioning properly"},{"type":"VMEventScheduled","status":"False","lastHeartbeatTime":"2024-05-13T10:18:26Z","lastTransitionTime":"2024-05-13T06:02:52Z","reason":"NoVMEventScheduled","message":"VM has no scheduled event"},{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2024-05-13T10:18:11Z","lastTransitionTime":"2024-05-13T05:58:54Z","reason":"KubeletHasSufficientMemory","message":"kubelet has sufficient memory available"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2024-05-13T10:18:11Z","lastTransitionTime":"2024-05-13T05:58:54Z","reason":"KubeletHasNoDiskPressure","message":"kubelet has no disk pressure"},{"type":"PIDPressure","status":"False","lastHeartbeatTime":"2024-05-13T10:18:11Z","lastTransitionTime":"2024-05-13T05:58:54Z","reason":"KubeletHasSufficientPID","message":"kubelet has sufficient PID available"},{"type":"Ready","status":"True","lastHeartbeatTime":"2024-05-13T10:18:11Z","lastTransitionTime":"2024-05-13T05:58:54Z","reason":"KubeletReady","message":"kubelet is posting ready status. AppArmor enabled"}],"addresses":[{"type":"InternalIP","address":"10.224.0.4"},{"type":"Hostname","address":"aks-nodepool1-81876434-vmss000000"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"5a1df07f9c934f2c93edd6633c426d95","systemUUID":"36346b6d-bcdd-4540-91b3-26c5afadfdee","bootID":"fc0e2e97-3d64-40ad-bc24-a7bd9a208d18","kernelVersion":"5.15.0-1061-azure","osImage":"Ubuntu 22.04.4 LTS","containerRuntimeVersion":"containerd://1.7.15-1","kubeletVersion":"v1.28.9","kubeProxyVersion":"v1.28.9","operatingSystem":"linux","architecture":"amd64"},"images":[{"names":["mcr.microsoft.com/azuredefender/stable/ [truncated 19893 chars]
NAME                                CPU(cores)   CPU%        MEMORY(bytes)   MEMORY%     
aks-nodepool1-81876434-vmss000000   187m         9%          2051Mi          45%         
akswin22000000                      <unknown>    <unknown>   <unknown>       <unknown>

from kubernetes.

brianpursley avatar brianpursley commented on June 16, 2024

Here's the code where <unknown> is printed.

kubernetes/staging/src/k8s.io/kubectl/pkg/metricsutil/metrics_printer.go

Line 185 in 71acf56

unknownMetricsStatus := "<unknown>"

I can't tell from the log output (because it gets truncated), but I'm guessing that /apis/metrics.k8s.io/v1beta1/nodes doesn't return metrics for akswin22000000.

You should be able to check for sure by running:

kubectl get --raw https://test-clust-junjiezhang-test-8ecadf-42z3zi6z.hcp.eastus.azmk8s.io:443/apis/metrics.k8s.io/v1beta1/nodes

Typical output (formatted via jq) should look something like this:

{
  "kind": "NodeMetricsList",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "metadata": {
        "name": "k8s-master",
        "creationTimestamp": "2024-05-13T14:46:48Z",
        "labels": {
          "beta.kubernetes.io/arch": "amd64",
          "beta.kubernetes.io/os": "linux",
          "kubernetes.io/arch": "amd64",
          "kubernetes.io/hostname": "k8s-master",
          "kubernetes.io/os": "linux",
          "node-role.kubernetes.io/control-plane": "",
          "node.kubernetes.io/exclude-from-external-load-balancers": ""
        }
      },
      "timestamp": "2024-05-13T14:46:37Z",
      "window": "10.025s",
      "usage": {
        "cpu": "199102244n",
        "memory": "2754236Ki"
      }
    },
    {
      "metadata": {
        "name": "k8s-worker-1",
        "creationTimestamp": "2024-05-13T14:46:48Z",
        "labels": {
          "beta.kubernetes.io/arch": "amd64",
          "beta.kubernetes.io/os": "linux",
          "kubernetes.io/arch": "amd64",
          "kubernetes.io/hostname": "k8s-worker-1",
          "kubernetes.io/os": "linux"
        }
      },
      "timestamp": "2024-05-13T14:46:38Z",
      "window": "10.027s",
      "usage": {
        "cpu": "39892290n",
        "memory": "1849496Ki"
      }
    },
    {
      "metadata": {
        "name": "k8s-worker-2",
        "creationTimestamp": "2024-05-13T14:46:48Z",
        "labels": {
          "beta.kubernetes.io/arch": "amd64",
          "beta.kubernetes.io/os": "linux",
          "kubernetes.io/arch": "amd64",
          "kubernetes.io/hostname": "k8s-worker-2",
          "kubernetes.io/os": "linux"
        }
      },
      "timestamp": "2024-05-13T14:46:43Z",
      "window": "20.045s",
      "usage": {
        "cpu": "38712895n",
        "memory": "1770956Ki"
      }
    }
  ]
}

from kubernetes.

junjiezhang1997 avatar junjiezhang1997 commented on June 16, 2024

I tried kubectl get --raw https://test-clust-junjiezhang-test-8ecadf-42z3zi6z.hcp.eastus.azmk8s.io:443/apis/metrics.k8s.io/v1beta1/nodes | jq and got the following results:

{
  "kind": "NodeMetricsList",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "metadata": {
        "name": "aks-nodepool1-81876434-vmss000000",
        "creationTimestamp": "2024-05-14T02:34:04Z",
        "labels": {
          "agentpool": "nodepool1",
          "beta.kubernetes.io/arch": "amd64",
          "beta.kubernetes.io/instance-type": "Standard_DS2_v2",
          "beta.kubernetes.io/os": "linux",
          "failure-domain.beta.kubernetes.io/region": "eastus",
          "failure-domain.beta.kubernetes.io/zone": "0",
          "kubernetes.azure.com/agentpool": "nodepool1",
          "kubernetes.azure.com/cluster": "MC_junjiezhang-test_test-cluster_eastus",
          "kubernetes.azure.com/consolidated-additional-properties": "5aaea556-10ed-11ef-94f1-f62e6c17092e",
          "kubernetes.azure.com/kubelet-identity-client-id": "05393a22-e4ce-4b07-bfe0-eb71e44e0f7d",
          "kubernetes.azure.com/mode": "system",
          "kubernetes.azure.com/node-image-version": "AKSUbuntu-2204gen2containerd-202405.03.0",
          "kubernetes.azure.com/nodepool-type": "VirtualMachineScaleSets",
          "kubernetes.azure.com/os-sku": "Ubuntu",
          "kubernetes.azure.com/role": "agent",
          "kubernetes.azure.com/storageprofile": "managed",
          "kubernetes.azure.com/storagetier": "Premium_LRS",
          "kubernetes.io/arch": "amd64",
          "kubernetes.io/hostname": "aks-nodepool1-81876434-vmss000000",
          "kubernetes.io/os": "linux",
          "kubernetes.io/role": "agent",
          "node-role.kubernetes.io/agent": "",
          "node.kubernetes.io/instance-type": "Standard_DS2_v2",
          "storageprofile": "managed",
          "storagetier": "Premium_LRS",
          "topology.disk.csi.azure.com/zone": "",
          "topology.kubernetes.azure.com/sub-fault-domain": "0",
          "topology.kubernetes.io/region": "eastus",
          "topology.kubernetes.io/zone": "0"
        }
      },
      "timestamp": "2024-05-14T02:32:58Z",
      "window": "1m0.239s",
      "usage": {
        "cpu": "177028171n",
        "memory": "2140828Ki"
      }
    }
  ]
}

I think you are right. It doesn't return the result for windows node.

from kubernetes.

brianpursley avatar brianpursley commented on June 16, 2024

Possibly related:

Those issues are closed, but it could be a related problem, or they weren't entirely fixed.

from kubernetes.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.