KOPS 1.5.0-beta2, Kubernetes 1.5.2, AWS: Only api cert error TODO from deploy, self-hosted setup. about kube-prometheus

[dmcnaught]

Yes, Kubernetes is listed now as a target and in alerts 😄 , so I'll update the docs too.

[dmcnaught]

Same with KOPS 1.5.0-beta2, Kubernetes 1.4.8 (w prometheus-operator:v0.1.1)

[dmcnaught]

Running ku -n kube-system create -f manifests/k8s/self-hosted/ updates to:

Alerts
K8SNodeDown (1 active)
K8SKubeletDown (1 active)
K8SKubeletNodeExporterDown (1 active)
K8SApiserverDown (1 active)

Targets
kube-controller-manager
Endpoint	State	Labels	Last Scrape	Error
http://10.101.91.106:10252/metrics
DOWN	instance="10.101.91.106:10252"	11.231s ago	context deadline exceeded
kube-dns
Endpoint	State	Labels	Last Scrape	Error
http://100.96.1.2:10054/metrics
UP	instance="100.96.1.2:10054"	4.934s ago	
http://100.96.1.2:10055/metrics
UP	instance="100.96.1.2:10055"	3.823s ago	
http://100.96.1.4:10054/metrics
UP	instance="100.96.1.4:10054"	1.97s ago	
http://100.96.1.4:10055/metrics
UP	instance="100.96.1.4:10055"	12.165s ago	
kube-scheduler
Endpoint	State	Labels	Last Scrape	Error
http://10.101.91.106:10251/metrics
DOWN	instance="10.101.91.106:10251"	24.405s ago	context deadline exceeded
kube-state-metrics
Endpoint	State	Labels	Last Scrape	Error
http://100.96.1.66:8080/metrics
UP	instance="100.96.1.66:8080"	13.793s ago	
kubelets
Endpoint	State	Labels	Last Scrape	Error
https://10.101.90.245:10250/metrics
UP	instance="ip-10-101-90-245.ec2.internal"	13.104s ago	
https://10.101.91.106:10250/metrics
DOWN	instance="ip-10-101-91-106.ec2.internal"	13.514s ago	context deadline exceeded
kubernetes
Endpoint	State	Labels	Last Scrape	Error
https://10.101.91.106:443/metrics
DOWN	instance="10.101.91.106:443"	12.117s ago	Get https://10.101.91.106:443/metrics: x509: certificate is valid for 100.64.0.1, not 10.101.91.106
node-exporter
Endpoint	State	Labels	Last Scrape	Error
http://10.101.90.245:9100/metrics
UP	instance="10.101.90.245:9100"	1.398s ago	
http://10.101.91.106:9100/metrics
DOWN	instance="10.101.91.106:9100"	14.259s ago	context deadline exceeded
prometheus
Endpoint	State	Labels	Last Scrape	Error
http://100.96.1.68:9090/metrics
UP	instance="100.96.1.68:9090"	14.541s ago	

[dmcnaught]

Opening up 10250 and 9100 in the master SG to the nodes SG fixed the node exporter and kubelets communications errors. Then just have two errors:

K8SSchedulerDown (1 active)
K8SControllerManagerDown (1 active)

Running kubectl -n kube-system create -f manifests/k8s/self-hosted/
opening up ports again to master: 10252, 10251 leaves only one target error:

kubernetes
Endpoint	State	Labels	Last Scrape	Error
https://10.52.71.194:443/metrics
DOWN	instance="10.52.71.194:443"	3.711s ago	Get https://10.52.71.194:443/metrics: x509: certificate is valid for 100.64.0.1, not 10.52.71.194

and one alert:

K8SApiserverDown (1 active)

Related comment: #23 (comment)

[brancz]

Yeah that's correct, best would be to inspect the cert as described in #23 (comment) and set the serverName to kubernetes. In #33 I chose to ignore cert issues for now, however want to apply exactly what I mentioned to that, but wanted to get the general config generation in place first with #33. Could you maybe try out #33, and see if it works?

[dmcnaught]

#33 tried out and feedback sent in Slack 😄

[brancz]

@dmcnaught can this be closed as the #49 is merged?