stefanprodan / k8s-prom-hpa Goto Github PK
View Code? Open in Web Editor NEWKubernetes Horizontal Pod Autoscaler with Prometheus custom metrics
License: MIT License
Kubernetes Horizontal Pod Autoscaler with Prometheus custom metrics
License: MIT License
-bash-4.2# kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" | jq .
Error from server (NotFound): the server could not find the requested resource
-bash-4.2# kubectl get --raw "/apis/metrics.k8s.io/v1beta1/pods" | jq .
Error from server (NotFound): the server could not find the requested resource
The Metric Pod status is Terminated and the log has following error / panic
Error: Get https://10.96.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 10.96.0.1:443: i/o timeout
In your ReadMe, you have demoed a way to fetch any custom metrics related to Pod. How do i fetch the same for other services such as nodes,replications etc ?
Ex:
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .
{
"name": "namespaces/kube_deployment_status_replicas_updated",
"singularName": "",
"namespaced": false,
"kind": "MetricValueList",
"verbs": [
"get"
]
},
{
"name": "replicasets.extensions/kube_replicaset_status_observed_generation",
"singularName": "",
"namespaced": true,
"kind": "MetricValueList",
"verbs": [
"get"
]
},
{
"name": "pods/memory_failcnt",
"singularName": "",
"namespaced": true,
"kind": "MetricValueList",
"verbs": [
"get"
]
},
{
"name": "daemonsets.extensions/kube_daemonset_status_number_available",
"singularName": "",
"namespaced": true,
"kind": "MetricValueList",
"verbs": [
"get"
]
}
What is the self -link for "daemonsets.extensions/kube_daemonset_status_number_available" and replicasets.extensions/kube_replicaset_status_observed_generation ?
I1207 09:53:35.292482 1 request.go:836] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps "extension-apiserver-authentication" not found","reason":"NotFound","details":{"name":"extension-apiserver-authentication","kind":"configmaps"},"code":404}
W1207 09:53:35.292976 1 authentication.go:231] Unable to get configmap/extension-apiserver-authentication in kube-system. Usually fixed by 'kubectl create rolebinding -n kube-system ROLE_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
Error: configmaps "extension-apiserver-authentication" not found
@stefanprodan thanks for creating this project. I am starting to work on latency based autoscaler's soon. My first question is why is this very basic and very useful hpa not part of istio ?
Hi,
I followed the instructions and deployed podinfo app and i am able to scale the app on http_requests. However, whenever i run my own app or just nginx, i am getting error as "unable to get metric http_requests: no metrics returned from custom metrics API". Deployment files are exactly the same, just changed port number, health_checks and labels.
[ec2-user@ip-192-168-100-253 nginx]$ kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
nginx-custom-hpa Deployment/nginx <unknown>/10 2 10 2 10m
podinfo Deployment/podinfo 888m/10 2 10 2 23m
[ec2-user@ip-192-168-100-253 nginx]$ kubectl describe hpa nginx-custom-hpa
Name: nginx-custom-hpa
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Mon, 19 Aug 2019 08:32:58 +0000
Reference: Deployment/nginx
Metrics: ( current / target )
"http_requests" on pods: <unknown> / 10
Min replicas: 2
Max replicas: 10
Deployment pods: 2 current / 0 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetPodsMetric the HPA was unable to compute the replica count: unable to get metric http_requests: no metrics returned from custom metrics API
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedComputeMetricsReplicas 8m18s (x12 over 11m) horizontal-pod-autoscaler failed to get object metric value: unable to get metric http_requests: no metrics returned from custom metrics API
Warning FailedGetPodsMetric 63s (x41 over 11m) horizontal-pod-autoscaler unable to get metric http_requests: no metrics returned from custom metrics API
Is my app running??
[ec2-user@ip-192-168-100-253 nginx]$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 21d
nginx-svc NodePort 10.100.195.60 <none> 9899:31199/TCP 13m
podinfo NodePort 10.100.112.143 <none> 9898:31198/TCP 24m
[ec2-user@ip-192-168-101-39 ~]$ curl -I http://10.100.195.60:9899
HTTP/1.1 200 OK
Server: nginx/1.17.3
[ec2-user@ip-192-168-100-253 ~]$ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_requests" | jq .
{
"kind": "MetricValueList",
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/http_requests"
},
"items": [
{
"describedObject": {
"kind": "Pod",
"namespace": "default",
"name": "podinfo-7b8c9bc5c9-xgppt",
"apiVersion": "/v1"
},
"metricName": "http_requests",
"timestamp": "2019-08-19T08:48:38Z",
"value": "888m"
},
{
"describedObject": {
"kind": "Pod",
"namespace": "default",
"name": "podinfo-7b8c9bc5c9-zkkbd",
"apiVersion": "/v1"
},
"metricName": "http_requests",
"timestamp": "2019-08-19T08:48:38Z",
"value": "888m"
}
]
}
I would like to know why i am not able to see http_requests for nginx app?
master1 kube-apiserver: E0412 22:18:30.255424 2628 available_controller.go:295] v1beta1.metrics.k8s.io failed with: Get https://10.233.53.4:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
[root@master1 1.8+]# kubectl exec -it metrics-server-7bcc5bf8f-pk865 -n kube-system sh / # ps -ef PID USER TIME COMMAND 1 root 0:00 /metrics-server --source=kubernetes.summary_api:'' --requestheader-client-ca-file=/etc/kubernetes/ssl/front-proxy-ca.pem 38 root 0:00 sh 43 root 0:00 ps -ef / # netstat -anpt Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 10.233.53.4:49394 10.254.0.1:443 ESTABLISHED 1/metrics-server tcp 0 0 10.233.53.4:52990 192.168.200.52:10255 ESTABLISHED 1/metrics-server tcp 0 0 10.233.53.4:51762 192.168.200.53:10255 ESTABLISHED 1/metrics-server tcp 0 0 :::443 :::* LISTEN 1/metrics-server / #
[root@master1 1.8+]# kubectl logs -f metrics-server-7bcc5bf8f-pk865 -n kube-system I0412 14:15:02.347895 1 heapster.go:71] /metrics-server --source=kubernetes.summary_api:'' --requestheader-client-ca-file=/etc/kubernetes/ssl/front-proxy-ca.pem I0412 14:15:02.347957 1 heapster.go:72] Metrics Server version v0.2.1 I0412 14:15:02.348159 1 configs.go:61] Using Kubernetes client with master "https://10.254.0.1:443" and version I0412 14:15:02.348194 1 configs.go:62] Using kubelet port 10255 I0412 14:15:02.349072 1 heapster.go:128] Starting with Metric Sink I0412 14:15:02.555531 1 serving.go:308] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key) I0412 14:15:02.933037 1 heapster.go:101] Starting Heapster API server... [restful] 2018/04/12 14:15:02 log.go:33: [restful/swagger] listing is available at https:///swaggerapi [restful] 2018/04/12 14:15:02 log.go:33: [restful/swagger] https:///swaggerui/ is mapped to folder /swagger-ui/ I0412 14:15:02.934173 1 serve.go:85] Serving securely on 0.0.0.0:443
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
User=root
ExecStart=/usr/local/bin/kube-apiserver
--admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota
--advertise-address=192.168.200.51
--allow-privileged=true
--anonymous-auth=false
--apiserver-count=1
--audit-policy-file=/etc/kubernetes/audit-policy.yaml
--audit-log-maxage=30
--audit-log-maxbackup=3
--audit-log-maxsize=100
--audit-log-path=/var/log/kubernetes/audit.log
--authorization-mode=Node,RBAC
--bind-address=0.0.0.0
--secure-port=6443
--client-ca-file=/etc/kubernetes/ssl/ca.pem
--enable-swagger-ui=true
--etcd-cafile=/etc/kubernetes/ssl/ca.pem
--etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem
--etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem
--etcd-servers=https://192.168.200.51:2379,https://192.168.200.52:2379,https://192.168.200.53:2379
--event-ttl=1h
--kubelet-https=true
--insecure-bind-address=192.168.200.51
--insecure-port=8080
--service-account-key-file=/etc/kubernetes/ssl/ca-key.pem
--service-cluster-ip-range=10.254.0.0/16
--service-node-port-range=30000-32000
--tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem
--tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem
--enable-bootstrap-token-auth=true
--token-auth-file=/etc/kubernetes/token.csv
--requestheader-client-ca-file=/etc/kubernetes/ssl/front-proxy-ca.pem
--proxy-client-cert-file=/etc/kubernetes/ssl/front-proxy-client.pem
--proxy-client-key-file=/etc/kubernetes/ssl/front-proxy-client-key.pem
--requestheader-allowed-names=aggregator
--requestheader-group-headers=X-Remote-Group
--requestheader-extra-headers-prefix=X-Remote-Extra-
--requestheader-username-headers=X-Remote-User
--runtime-config=admissionregistration.k8s.io/v1alpha1
--runtime-config=api/all=true
--enable-aggregator-routing=true
--v=0
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
ExecStart=/usr/local/bin/kube-controller-manager
--address=0.0.0.0
--master=http://192.168.200.51:8080
--service-cluster-ip-range=10.254.0.0/16
--cluster-name=kubernetes
--cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem
--cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem
--service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem
--root-ca-file=/etc/kubernetes/ssl/ca.pem
--leader-elect=true
--v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
Hi Team,
I have followed the Readme and able to scale up/down the pods using http_request.
Typically, I want to scale up/down the HPA based on Istio Prometheus http_request value. Hence, I have tried to configure the Istio Prometheus in custom metric API server afterward unable to get the http_request metric values.
Just changed the Prometheus endpoint in the custom-metrics-apiserver-deployment.yaml.
- name: custom-metrics-apiserver
image: quay.io/coreos/k8s-prometheus-adapter-amd64:v0.2.0
args:
- /adapter
- --secure-port=6443
- --tls-cert-file=/var/run/serving-cert/serving.crt
- --tls-private-key-file=/var/run/serving-cert/serving.key
- --logtostderr=true
- --prometheus-url=http://prometheus.aruntest.tk:9090/ # its a public domain
- --metrics-relist-interval=30s
- --rate-int
Error:
root@ip-172-21-15-19:/k8s-prom-hpa/custom-metrics-api# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/monitoring/pods/*/http_requests" | j/k8s-prom-hpa/custom-metrics-api#
q .
Error from server (NotFound): the server could not find the metric http_requests for pods
root@ip-172-21-15-19:
About the how-to in Readme.
I thought that it would be better to include the procedure of deleting the old HPA.
https://github.com/stefanprodan/k8s-prom-hpa#auto-scaling-based-on-custom-metrics
Deploy the podinfo HPA in the default namespace:
kubectl create -f ./podinfo/podinfo-hpa-custom.yaml
Write like below, before above.
kubectl delete -f ./podinfo/podinfo-hpa.yaml
thanks, great products.
I am trying to access the values of node metrics (exposed by node exporter to Prometheus) in custom metrics API which appear e.g. jobs.batch/node_memory_MemTotal. But when I do kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/batch.jobs/node_memory_MemTotal" | jq . I get Error from server (NotFound): the server could not find the requested resource
Installed custom metric but I do not see all the metrics I need specially for JVM metrics. In addition, it appears the image does not like using custom config.yaml. Below I created adapter-config configmap specifically for JVM.
apiVersion: v1
kind: ConfigMap
metadata:
name: adapter-config
namespace: monitoring
data:
config.yaml: |
- seriesQuery: '{__name__=~"jvm_.*",kubernetes_name!="",kubernetes_namespace!=""}'
resources:
overrides:
kubernetes_namespace:
resource: namespace
kubernetes_name:
resource: pod
name:
matches: "^jvm_(.*)"
metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
However, when I use the config in deployment.yaml, the pod crashes pointing that it does not recognize the flag.
spec:
serviceAccountName: custom-metrics-apiserver
containers:
- name: custom-metrics-apiserver
image: quay.io/coreos/k8s-prometheus-adapter-amd64:v0.2.0
args:
- /adapter
- --secure-port=6443
- --tls-cert-file=/var/run/serving-cert/serving.crt
- --tls-private-key-file=/var/run/serving-cert/serving.key
- --logtostderr=true
- --prometheus-url=http://prometheus.monitoring.svc:9090/
- --metrics-relist-interval=30s
- --rate-interval=5m
- --v=10
- --config=/etc/adapter/config.yaml
panic: unknown flag: --config
goroutine 1 [running]:
main.main()
/go/src/github.com/directxman12/k8s-prometheus-adapter/cmd/adapter/adapter.go:41 +0x114
Can you assist on how I can expose jvm metrics to custom.metrics.k8s.io?
Thanks,
Christian M.
Hi Stefanprodan, I have some questions after flowing your guidance:
In the second part, you use the prometheus instead of metrics server to collect the metrics, why don't we use metrics server?Can it only collect the core metrics like cpu or memory? Could you please share your reasons to me? Thanks!
For the file "prometheus-cfg.yaml", when i installed the prometheus, there is no such file. So when i write my own customer metrics server, can i use it into my prometheus directly? If it needs to change some items, could you please point them out for me? Thanks!
[root@dev3 k8s-prom-hpa]# make certs
make: *** No rule to make target `certs'. Stop.
Hi,
I am building custom metrics server by following this github instructions. Thanks. It works. Now, I would like to deploy node_exporter from https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus/manifests/node-exporter to get node level information, but I don't see any metrics from node_exporter , such as node_memory_MemFree or node_memory_MemTotal, when I invoked kubectl get --raw /metrics
. Do I need any additional components?
Thanks.
The TLS certificate required by the Prometheus adapter cannot be built, the container engine uses containerd, and the Makefile writes docker
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)
I0404 05:21:09.136867 1 serving.go:273] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
[restful] 2019/04/04 05:21:10 log.go:33: [restful/swagger] listing is available at https://:443/swaggerapi
[restful] 2019/04/04 05:21:10 log.go:33: [restful/swagger] https://:443/swaggerui/ is mapped to folder /swagger-ui/
I0404 05:21:10.729295 1 serve.go:96] Serving securely on [::]:443
E0404 06:51:15.246723 1 reflector.go:322] github.com/kubernetes-incubator/metrics-server/vendor/k8s.io/client-go/informers/factory.go:130: Failed to watch *v1.Pod: Get https://10.254.0.1:443/api/v1/pods?resourceVersion=17196317&timeoutSeconds=434&watch=true: dial tcp 10.254.0.1:443: connect: connection refused
help me,thanks!
I'm using mongodb-exporter for store/query the metrics via prometheus. I have set up a custom metric server and storing values for that .
That is the evidence of prometheus-exporter and custom-metric-server works compatible .
Query : kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/monitoring/pods/*/mongodb_mongod_wiredtiger_cache_bytes"
Result : {"kind":"MetricValueList","apiVersion":"custom.metrics.k8s.io/v1beta1","metadata":{"selfLink":"/apis/custom.metrics.k8s.io/v1beta1/namespaces/monitoring/pods/%2A/mongodb_mongod_wiredtiger_cache_bytes"},"items":[{"describedObject":{"kind":"Pod","namespace":"monitoring","name":"mongo-exporter-2-prometheus-mongodb-exporter-68f95fd65d-dvptr","apiVersion":"/v1"},"metricName":"mongodb_mongod_wiredtiger_cache_bytes","timestamp":"TTTTT","value":"0"}]}
In my case when I create a hpa for this custom metrics from mongo exporter, hpa return this error to me :
failed to get mongodb_mongod_wiredtiger_cache_bytes utilization: unable to get metrics for resource mongodb_mongod_wiredtiger_cache_bytes: no metrics returned from resource metrics API
What is the main issue on my case ? I have checked all configs and flow is looking fine, but where is the my mistake .
Help
Thanks :)
Getting the following error Error: cluster doesn't provide client-ca-file
from the quay.io/coreos/k8s-prometheus-adapter-amd64:v0.2.0
.
I am not sure if it is related with this kubernetes/kubernetes#65724 or is a misconfiguration on my AWS EKS cluster.
Any ideas would be appreciated.
I understand that the volume "volume-serving-cert" has to be set in order to get these two cert works:
But where are these cert from? Is it for prometheus or api server or just a self cert need to be generate?
Many thanks
Hi Stefan,
First of thank you for providing custom metrics dependencies which is working fine for us. But we need to enrich small code into this to use.
currently this custom metrics image is fetching metric value from prometheus and responding back to HPA for autoscaling.
But we want to add irate function to prometheus metrics to fetch tps directly from prometheus like below.
sum(irate(<>[1m]))
So can you please guide us in this.
Thanks,
Surendra
@stefanprodan
Hi,
In the README.md, you say "The podinfo app exposes a custom metric named http_requests_total. The Prometheus adapter removes the _total suffix and marks the metric as a counter metric."
While I can't find where it is. So could you please help me to point them out:
Thanks!
Warning FailedGetResourceMetric 4s (x2 over 34s) horizontal-pod-autoscaler unable to get metrics for resource memory: no metrics returned from heapster
Warning FailedComputeMetricsReplicas 4s (x2 over 34s) horizontal-pod-autoscaler failed to get memory utilization: unable to get metrics for resource memory: no metrics returned from heapster
Maybe is something that i need to do, but followed your readme file and didn't get it working
`
cat /etc/kubernetes/manifests/kube-apiserver.manifest
apiVersion: v1
kind: Pod
metadata:
name: kube-apiserver
namespace: kube-system
labels:
k8s-app: kube-apiserver
kubespray: v2
annotations:
kubespray.etcd-cert/serial: "E0B8497F43AA5A66"
kubespray.apiserver-cert/serial: "BEE5A9B9D94E41B8"
spec:
hostNetwork: true
dnsPolicy: ClusterFirst
containers:
Hi,How do I add http_requests
indicators? thx.
[root@kube-master custom-metrics-apiserver]# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_requests" | jq .
Error from server (NotFound): the server could not find the metric http_requests for pods
[root@kube-master custom-metrics-apiserver]# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .|grep http_requests
[root@kube-master custom-metrics-apiserver]#
[root@kube-master custom-metrics-apiserver]# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .|grep fs_usage_bytes
"name": "pods/fs_usage_bytes"
[root@kube-master custom-metrics-apiserver]#
I am getting the below error when Irun the command make certs
under the project folder.
Generating TLS certs
error: cannot open .git/FETCH_HEAD: Permission denied
package github.com/cloudflare/cfssl/cmd/cfssl: exit status 1
make: *** [gencerts] Error 1
I don't know how to expand the bandwidth according to the cluster. Most of the examples are about CPU and memory.I can't write this yaml file. Please give me some help.
Steps to reproduce:
$ kubectl create -f metrics-server/
$ kubectl create -f ./podinfo/podinfo-svc.yaml,./podinfo/podinfo-dep.yaml
$ kubectl create -f ./podinfo/podinfo-hpa.yaml
Soon after, I start seeing erroneous values for the CPU (they're not 100%, I've not yet started sending any traffic); and the memory is showing as <unknown>
.
$ kubectl get hpa -w
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
podinfo Deployment/podinfo <unknown>/200Mi, 100%/80% 2 10 4 2m10s
If I look at the logs of the metrics server I see the following:
$ kubectl logs metrics-server-68d85f76bb-kgqlr -n kube-system
E0320 14:29:47.201497 1 reststorage.go:144] unable to fetch pod metrics for pod default/podinfo-7b8c9bc5c9-d5f42: no metrics known for pod
I'm running on CentoOS on clusters created with kubeadm init
with just the defaults.
$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:35:32Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
$ docker version
Client:
Version: 18.09.3
API version: 1.39
Go version: go1.10.6
Git commit: 142dfce
Built: Thu Feb 28 06:08:06 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine
Engine:
Version: 18.09.3
API version: 1.39 (minimum version 1.12)
Go version: go1.10.8
Git commit: 142dfce
Built: Thu Feb 28 06:03:18 2019
OS/Arch: linux/amd64
Experimental: false
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:37:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:30:26Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
when finished it , I curl kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq and get
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "custom.metrics.k8s.io/v1beta1",
"resources": []
}
the resources is nothing. From the pod ,I get the logs
Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
sorry to spam your github issues but I wanted to share my appreciation for your work
@GitCash send $5 BCH to @stefanprodan 🍻
Below error reported when doing custom metric:
Failed to load config file: {"code":5200,"message":"could not read configuration file"}Failed to parse input: unexpected end of JSON input
Makefile:14: recipe for target 'gencerts' failed
make: *** [gencerts] Error 1
Hi, Thank you very much for your documentation. I deployed custom-metrics-api in my cluster and I didn't report an error during the deployment process. But when I executed the 'kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq . ' command, I reported the following error:
error: You must be logged in to the server (Unauthorized)
logs:
I0613 03:59:15.327246 1 request.go:897] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
I0613 03:59:15.327513 1 authorization.go:73] Forbidden: "/", Reason: ""
I0613 03:59:15.327730 1 wrap.go:42] GET /: (12.371276ms) 403 [[Go-http-client/2.0] 177.245.72.64:21169]
W0613 03:59:24.780796 1 x509.go:172] x509: subject with cn=front-proxy-client is not in the allowed list: [aggregator]
E0613 03:59:24.780859 1 authentication.go:62] Unable to authenticate the request due to an error: [x509: subject with cn=front-proxy-client is not allowed, x509: certificate signed by unknown authority]
I0613 03:59:24.781005 1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (696.518µs) 401 [[kube-controller-manager/v1.13.6 (linux/amd64) kubernetes/abdda3f/system:serviceaccount:kube-system:resourcequota-controller] 177.245.72.64:18907]
W0613 03:59:26.187626 1 x509.go:172] x509: subject with cn=front-proxy-client is not in the allowed list: [aggregator]
E0613 03:59:26.187716 1 authentication.go:62] Unable to authenticate the request due to an error: [x509: subject with cn=front-proxy-client is not allowed, x509: certificate signed by unknown authority]
I0613 03:59:26.187866 1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (822.578µs) 401 [[kube-apiserver/v1.13.6 (linux/amd64) kubernetes/abdda3f] 177.245.72.64:18907]
W0613 03:59:29.071907 1 x509.go:172] x509: subject with cn=front-proxy-client is not in the allowed list: [aggregator]
E0613 03:59:29.071978 1 authentication.go:62] Unable to authenticate the request due to an error: [x509: subject with cn=front-proxy-client is not allowed, x509: certificate signed by unknown authority]
I0613 03:59:29.072126 1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (669.407µs) 401 [[kube-apiserver/v1.13.6 (linux/amd64) kubernetes/abdda3f] 10.103.236.179:36320]
I0613 03:59:29.923974 1 request.go:897] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
I0613 03:59:29.924301 1 round_trippers.go:386] curl -k -v -XPOST -H "Content-Type: application/json" -H "Accept: application/json, */*" -H "User-Agent: adapter/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtb25pdG9yaW5nIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImN1c3RvbS1tZXRyaWNzLWFwaXNlcnZlci10b2tlbi1yd2I1bSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjdXN0b20tbWV0cmljcy1hcGlzZXJ2ZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI2ZGE5M2UwYS04ZDhmLTExZTktOTBkYy1mODBmNDFmMjdkYTEiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6bW9uaXRvcmluZzpjdXN0b20tbWV0cmljcy1hcGlzZXJ2ZXIifQ.xyuVdgLY7GxpozNMPGqUFpOhe2xTlKFtAH62xmgoSRjw3dx2LAMQwdYVcPRJJhEnYL5fadsQpENCbbO21v229RJFd3ZSuNbFzqjCf5Zi_SP8c2XIGyPQtkOnBxJK1RfcisLsAxt-FfFP-m5OZ33okRKXVyb6tZj3qK08YPHdD9WlVYSpdlTg8aK_GPlwWbmSftn4A4K7iGXzKb936trjO9SdT3aTz2sYY7PzkzKAt1w2M48Vge8P0UJvUnD1mGZ3T2fUYFGMtBmQe598Cx3wDssVjw2Nm8_QFtkGkgzIW2HvIkFwblNcjztF5-6qcMu4HkoSZjSS76w0BlJhrj2nag" 'https://50.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews'
I0613 03:59:29.935204 1 round_trippers.go:405] POST https://50.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 10 milliseconds
I0613 03:59:29.935264 1 round_trippers.go:411] Response Headers:
I0613 03:59:29.935281 1 round_trippers.go:414] Content-Length: 260
I0613 03:59:29.935295 1 round_trippers.go:414] Date: Thu, 13 Jun 2019 04:01:00 GMT
I0613 03:59:29.935308 1 round_trippers.go:414] Content-Type: application/json
I0613 03:59:29.935406 1 request.go:897] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
I0613 03:59:29.935612 1 authorization.go:73] Forbidden: "/", Reason: ""
I0613 03:59:29.935817 1 wrap.go:42] GET /: (12.205904ms) 403 [[Go-http-client/2.0] 10.103.236.179:36294]
W0613 03:59:30.791929 1 x509.go:172] x509: subject with cn=front-proxy-client is not in the allowed list: [aggregator]
E0613 03:59:30.791994 1 authentication.go:62] Unable to authenticate the request due to an error: [x509: subject with cn=front-proxy-client is not allowed, x509: certificate signed by unknown authority]
I0613 03:59:30.792143 1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (696.288µs) 401 [[kube-controller-manager/v1.13.6 (linux/amd64) kubernetes/abdda3f/system:serviceaccount:kube-system:generic-garbage-collector] 177.245.72.64:18907]
W0613 03:59:32.541963 1 x509.go:172] x509: subject with cn=front-proxy-client is not in the allowed list: [aggregator]
E0613 03:59:32.542042 1 authentication.go:62] Unable to authenticate the request due to an error: [x509: subject with cn=front-proxy-client is not allowed, x509: certificate signed by unknown authority]
I0613 03:59:32.542186 1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (672.224µs) 401 [[kube-controller-manager/v1.13.6 (linux/amd64) kubernetes/abdda3f/system:serviceaccount:kube-system:generic-garbage-collector] 177.245.72.64:18907]
I0613 03:59:34.996385 1 authorization.go:73] Forbidden: "/", Reason: ""
I0613 03:59:34.996529 1 wrap.go:42] GET /: (386.223µs) 403 [[Go-http-client/2.0] 177.253.180.64:49696]
Looking at the log is like a certificate issue.
My cluster was deployed manually, not using kubeadm. When I created the cluster, I generated the following certificate:
admin.csr
admin-key.pem
admin.pem
apiserver.csr
apiserver-key.pem
apiserver.pem
ca.csr
ca-key.pem
ca.pem
controller-manager.csr
controller-manager-key.pem
controller-manager.pem
front-proxy-ca.csr
front-proxy-ca-key.pem
front-proxy-ca.pem
front-proxy-client.csr
front-proxy-client-key.pem
front-proxy-client.pem
kubelet-key.pem
kubelet.pem
sa.key
sa.pub
scheduler.csr
scheduler-key.pem
scheduler.pem
Then I tried to change the ca certificate in the Makefile, then re-execute the make certs, and finally redeploy the custom-metrics-api, but still have this problem, is there a solution?
Kubernetes Version:
[root@k8s-master01 k8s-prom-hpa]# kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.6", GitCommit:"abdda3f9fefa29172298a2e42f5102e777a8ec25", GitTreeState:"clean", BuildDate:"2019-05-08T13:53:53Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.6", GitCommit:"abdda3f9fefa29172298a2e42f5102e777a8ec25", GitTreeState:"clean", BuildDate:"2019-05-08T13:46:28Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Hello,
I'm running kubernetes 1.10, and followed instruction here and installed metrics server, custom-metrics-api, and prometheus. I'm able to do autoscaling based on http_requests, but i can not do that based on container_network_receive_bytes_total, i noticed that http_requests metrics exposed at "/apis/custom.metrics.k8s.io/v1beta1", see output below:
[xyz@master k8s-prom-hpa]$ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1|jq .|grep http_requests
"name": "pods/http_requests_count",
"name": "pods/http_requests",
"name": "namespaces/http_requests",
"name": "jobs.batch/http_requests",
"name": "pods/http_requests_sum",
"name": "jobs.batch/http_requests_bucket",
"name": "jobs.batch/http_requests_count",
"name": "jobs.batch/http_requests_sum",
"name": "namespaces/http_requests_count",
"name": "namespaces/http_requests_sum",
"name": "pods/http_requests_bucket",
"name": "namespaces/http_requests_bucket",
but this is the case for "container_network_receive_bytes_total"
[xyz@master k8s-prom-hpa]$ kubectl get --raw container_network_receive_bytes_total|jq .|grep network
"name": "pods/network_udp_usage",
"name": "pods/network_tcp_usage",
In Prometheus , i can see "container_network_receive_bytes_total" with data, actually all the metrics start with "container_" in Prometheus didn't get exposed at /apis/custom.metrics.k8s.io/v1beta1..., any help is really appreciated
my system information:
[xyz@master k8s-prom-hpa]$ kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-12T14:14:26Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
[xyz@master k8s-prom-hpa]$ kubectl api-versions
admissionregistration.k8s.io/v1beta1
apiextensions.k8s.io/v1beta1
apiregistration.k8s.io/v1
apiregistration.k8s.io/v1beta1
apps/v1
apps/v1beta1
apps/v1beta2
authentication.k8s.io/v1
authentication.k8s.io/v1beta1
authorization.k8s.io/v1
authorization.k8s.io/v1beta1
autoscaling/v1
autoscaling/v2beta1
batch/v1
batch/v1beta1
certificates.k8s.io/v1beta1
custom.metrics.k8s.io/v1beta1
events.k8s.io/v1beta1
extensions/v1beta1
metrics.k8s.io/v1beta1
networking.k8s.io/v1
policy/v1beta1
rbac.authorization.k8s.io/v1
rbac.authorization.k8s.io/v1beta1
stable.example.com/v1
storage.k8s.io/v1
storage.k8s.io/v1beta1
v1
[yaoky@master k8s-prom-hpa]$ kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default php-apache-8699449574-8hlqf 1/1 Running 0 3h
default podinfo-65fc48f955-9xwdd 1/1 Running 0 22h
default podinfo-65fc48f955-fnzx4 1/1 Running 0 22h
default vtc1-deployment-6f864dbf54-lpq2d 1/1 Running 0 22h
default vtc2-deployment-7f65859fc9-nbt2x 1/1 Running 0 22h
kube-system etcd-master 1/1 Running 1 23h
kube-system heapster-cbfb87c9c-sqknr 1/1 Running 0 23h
kube-system kube-apiserver-master 1/1 Running 0 23h
kube-system kube-controller-manager-master 1/1 Running 2 23h
kube-system kube-dns-86f4d74b45-mgb44 3/3 Running 3 23h
kube-system kube-proxy-52ctt 1/1 Running 0 23h
kube-system kube-proxy-cgbb4 1/1 Running 1 23h
kube-system kube-proxy-fw46f 1/1 Running 0 23h
kube-system kube-proxy-rb4l5 1/1 Running 0 23h
kube-system kube-scheduler-master 1/1 Running 2 23h
kube-system metrics-server-6fbfb84cdd-cs7hk 1/1 Running 0 23h
kube-system monitoring-grafana-69df66f668-87qkl 1/1 Running 0 23h
kube-system monitoring-influxdb-78d4c6f5b6-jk59b 1/1 Running 0 23h
kube-system weave-net-4qvhg 2/2 Running 0 23h
kube-system weave-net-6mg8p 2/2 Running 0 23h
kube-system weave-net-9bvqg 2/2 Running 0 23h
kube-system weave-net-z9mmz 2/2 Running 3 23h
monitoring custom-metrics-apiserver-7dd968d85-sw7fl 1/1 Running 0 22h
monitoring prometheus-7dff795b9f-6jpd5 1/1 Running 0 22h
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.