learnk8s / spring-boot-k8s-hpa

Autoscaling Spring Boot with the Horizontal Pod Autoscaler and custom metrics on Kubernetes

Autoscaling Spring Boot with the Horizontal Pod Autoscaler and custom metrics on Kubernetes

Prerequisites

You should have minikube installed.

You should start minikube with at least 4GB of RAM:

minikube start \
  --memory 4096 \
  --extra-config=controller-manager.horizontal-pod-autoscaler-upscale-delay=1m \
  --extra-config=controller-manager.horizontal-pod-autoscaler-downscale-delay=2m \
  --extra-config=controller-manager.horizontal-pod-autoscaler-sync-period=10s

If you're using a pre-existing minikube instance, you can resize the VM by destroying it an recreating it. Just adding the --memory 4096 won't have any effect.

You should install jq — a lightweight and flexible command-line JSON processor.

You can find more info about jq on the official website.

Installing Custom Metrics Api

Deploy the Metrics Server in the kube-system namespace:

kubectl create -f monitoring/metrics-server

After one minute the metric-server starts reporting CPU and memory usage for nodes and pods.

View nodes metrics:

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" | jq .

View pods metrics:

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/pods" | jq .

Create the monitoring namespace:

kubectl create -f monitoring/namespaces.yaml

Deploy Prometheus v2 in the monitoring namespace:

kubectl create -f monitoring/prometheus

Deploy the Prometheus custom metrics API adapter:

kubectl create -f monitoring/custom-metrics-api

List the custom metrics provided by Prometheus:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .

Package the application

You package the application as a container with:

eval $(minikube docker-env)
docker build -t spring-boot-hpa .

Deploying the application

Deploy the application in Kubernetes with:

kubectl create -f kube/deployment

You can visit the application at http://minkube_ip:32000

(Find the minikube ip address via minikube ip)

You can post messages to the queue by via http://minkube_ip:32000/submit?quantity=2

You should be able to see the number of pending messages from http://minkube_ip:32000/metrics and from the custom metrics endpoint:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/messages" | jq .

Autoscaling workers

You can scale the application in proportion to the number of messages in the queue with the Horizontal Pod Autoscaler. You can deploy the HPA with:

kubectl create -f kube/hpa.yaml

You can send more traffic to the application with:

while true; do curl -d "quantity=1" -X POST http://minkube_ip:32000/submit ; sleep 4; done

When the application can't cope with the number of incoming messages, the autoscaler increases the number of pods in 3 minute intervals.

You may need to wait three minutes before you can see more pods joining the deployment with:

kubectl get pods

The autoscaler will remove pods from the deployment every 5 minutes.

You can inspect the event and triggers in the HPA with:

kubectl get hpa spring-boot-hpa

Notes

The configuration for metrics and metrics server is configured to run on minikube only.

You won't be able to run the same YAML files for metrics and custom metrics server on your cluster or EKS, GKE, AKS, etc.

Also, there are secrets checked in the repository to deploy the Prometheus adapter.

In production, you should generate your own secrets and (possibly) not check them into version control.

If you wish to run metrics and custom metrics server in production, you should check out the following resources:

• Metrics server
• How to install Prometheus and the Promtheus Adapter

spring-boot-k8s-hpa's Issues

error: unable to recognize "monitoring/metrics-server/metrics-server-deployment.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"

Hi,

I am getting this error:

error: unable to recognize "monitoring/metrics-server/metrics-server-deployment.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"

while I executed this command:

kubectl create -f monitoring/metrics-server

I am using Minikube version: v1.5.2

Thanks.

Am trying to understand how to run web and worker separately

So I have looked at your code. I see the way you are running backend and worker separately is by using the environment variable then using that environment variable to set up JMS listeners.
But with that approach, it means in the backend you will still run the backend and worker altogether. Since Spring application will start web by default.

Do you have the real application request data per second

Do you have the real application request data per second ,I want to use arima model to predict it

Broker localhost not started

Hello I'm trying to implement this example, but when I try to create the image using docker build -t spring-boot-hpa . I get this WARN:

WARN 168 --- [ main] o.apache.activemq.broker.BrokerRegistry : Broker localhost not started so using embedded-broker instead

and then:

WARN 168 --- [ActiveMQ Task-1] o.a.a.t.failover.FailoverTransport : Failed to connect to [vm://embedded-broker?create=false] after: 10 attempt(s) continuing to retry.

so the image never is created, so before to try to create image I install ActiveMQ and the service is running, but the problem persist

can you help me please ?

It doesn't work in EKS

the 'custom-metrics-apiserver' cannot run in EKS:
$ kubectl get po -n monitoring
NAME READY STATUS RESTARTS AGE
custom-metrics-apiserver-55c757b95d-x5zdg 0/1 CrashLoopBackOff 6 9m51s
prometheus-8669f7f7d8-slmqq 1/1 Running 0 11m

I checked the log. It has the following error error:
Error: cluster doesn't provide client-ca-file

Just past all below:
$ kubectl logs custom-metrics-apiserver-55c757b95d-x5zdg -n monitoring
I0606 10:15:51.477908 1 round_trippers.go:417] curl -k -v -XGET -H "Accept: application/json, /" -H "User-Agent: adapter/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtb25pdG9yaW5nIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImN1c3RvbS1tZXRyaWNzLWFwaXNlcnZlci10b2tlbi1oNXZjYiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjdXN0b20tbWV0cmljcy1hcGlzZXJ2ZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJhNWU0YjQxZC04ODQzLTExZTktODVmMS0wMjdhMzdkNDczMWUiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6bW9uaXRvcmluZzpjdXN0b20tbWV0cmljcy1hcGlzZXJ2ZXIifQ.qzcldE0DaAoaROLHg-mfWY5BegiN9YQwH4qcJGj3-wjWCt9NN3QECLRfdizbFCnglBmc5OmZw8bPc3n7mODBKLRIzqfwryh9RXfVD-heNuB1SnFYp3aTOyDM81k02m2eMcMxULtsfPi1OzyIx_TXqBdxvNiAWgSE9zdtKT9j_DQq2GHAqPLRiSDFI4HepDleyO0CXBGI0xvFiCVuy8J5o7D1aSHlQ232YE4w9ZaB4NgERIRwrQzHZbvGrk6pQn-PEvKFqdHICtR1f8Fgwi3LJtD3NGiichR4zXJNNCtdqxZRM8gJNG7aT-gF_sKH9iYL36EmVef0BRLu4FqSlrBQGw" https://172.20.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication
I0606 10:15:51.494442 1 round_trippers.go:436] GET https://172.20.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 16 milliseconds
I0606 10:15:51.494464 1 round_trippers.go:442] Response Headers:
I0606 10:15:51.494469 1 round_trippers.go:445] Audit-Id: 277f9a0b-bdbe-429f-a08c-2e1e8b6e9231
I0606 10:15:51.494473 1 round_trippers.go:445] Content-Type: application/json
I0606 10:15:51.494478 1 round_trippers.go:445] Content-Length: 1648
I0606 10:15:51.494482 1 round_trippers.go:445] Date: Thu, 06 Jun 2019 10:15:51 GMT
I0606 10:15:51.494518 1 request.go:836] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"extension-apiserver-authentication","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication","uid":"2e07486e-836e-11e9-85f1-027a37d4731e","resourceVersion":"56","creationTimestamp":"2019-05-31T06:34:45Z"},"data":{"requestheader-allowed-names":"["front-proxy-client"]","requestheader-client-ca-file":"-----BEGIN CERTIFICATE-----\nMIIC0DCCAbigAwIBAgIBADANBgkqhkiG9w0BAQsFADAZMRcwFQYDVQQDEw5mcm9u\ndC1wcm94eS1jYTAeFw0xOTA1MzEwNjM0MDNaFw0yOTA1MjgwNjM0MDNaMBkxFzAV\nBgNVBAMTDmZyb250LXByb3h5LWNhMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB\nCgKCAQEAtpd8oUCa2DYD+O1LSRlaePTLAbrUzdwkxyAssPKNTZgAA3uau4m7HDAW\nAhzpD3JvCohFKTBQ5Dz9kDKesxlQJ9sUVVECuBtskvpNy1GWJPXkDOYRyVDqPmfo\nWDyyGLJwcl34Ztzya5ybcG0nrXXEUElZ+CHvA3jHG30GLAF6kMJHkSFiT943zQml\nnxAsLsb0PYPtrACjYmh3B20Tn3NVuoQ2xPe2lmgjgzY45Ebj9ezjEv0Qli3sdsGC\nW5L+3f/y40vogjSHbgTnhBhedzNhJYco/WFDWxJBvawkKuBuhAhVA3WYxySRffPu\nEdSWK2wqL56cn9vLY0CBU+ZHsoVm1QIDAQABoyMwITAOBgNVHQ8BAf8EBAMCAqQw\nDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAL25JNmj4gvczywkS\nMQ97URsfAYjMCne7TMYxe01bfL+uFnXBRu4DeMA9Q+NQyYmneBiuKJRvhuP0l9U+\nTUWPpiFUBqljQI3azpd0A2E7RWLwUuhcMY37vQxpA+n2D3bfJwx4muQg1RmigX22\n4+tUsmNyzVVAj8H/knk/4oTls5VNSXhMVJ3qyHGmKjUWa+deHHiPT3mBY0S4SP35\nIqL42alcGEju3Vmak4cih848FbgFPaKv2SGpDsaCJNsKWTEs+utAmVuyaCY/iuNj\nyG4wNHpV13GYQ1FLefjCB2uqdAzFD+Wdow6ZvtZXSscnexcmRUNTk+i4j69dtfG4\ntDc0IA==\n-----END CERTIFICATE-----\n","requestheader-extra-headers-prefix":"["X-Remote-Extra-"]","requestheader-group-headers":"["X-Remote-Group"]","requestheader-username-headers":"["X-Remote-User"]"}}
Error: cluster doesn't provide client-ca-file
Usage:
[flags]

Flags:
--alsologtostderr log to standard error as well as files
--authentication-kubeconfig string kubeconfig file pointing at the 'core' kubernetes server with enough rights to create tokenaccessreviews.authentication.k8s.io.
--authentication-skip-lookup If false, the authentication-kubeconfig will be used to lookup missing authentication configuration from the cluster.
--authentication-token-webhook-cache-ttl duration The duration to cache responses from the webhook token authenticator. (default 10s)
--authorization-kubeconfig string kubeconfig file pointing at the 'core' kubernetes server with enough rights to create subjectaccessreviews.authorization.k8s.io.
--authorization-webhook-cache-authorized-ttl duration The duration to cache 'authorized' responses from the webhook authorizer. (default 10s)
--authorization-webhook-cache-unauthorized-ttl duration The duration to cache 'unauthorized' responses from the webhook authorizer. (default 10s)
--bind-address ip The IP address on which to listen for the --secure-port port. The associated interface(s) must be reachable by the rest of the cluster, and by CLI/web clients. If blank, all interfaces will be used (0.0.0.0). (default 0.0.0.0)
--cert-dir string The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored. (default "apiserver.local.config/certificates")
--client-ca-file string If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate.
--contention-profiling Enable lock contention profiling, if profiling is enabled
--discovery-interval duration interval at which to refresh API discovery information (default 10m0s)
--enable-swagger-ui Enables swagger ui on the apiserver at /swagger-ui
--lister-kubeconfig string kubeconfig file pointing at the 'core' kubernetes server with enough rights to list any described objets
--log-flush-frequency duration Maximum number of seconds between log flushes (default 5s)
--log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log_dir string If non-empty, write log files in this directory
--logtostderr log to standard error instead of files (default true)
--metrics-relist-interval duration interval at which to re-list the set of all available metrics from Prometheus (default 10m0s)
--profiling Enable profiling via web interface host:port/debug/pprof/ (default true)
--prometheus-url string URL and configuration for connecting to Prometheus. Query parameters are used to configure the connection (default "https://localhost")
--rate-interval duration period of time used to calculate rate metrics from cumulative metrics (default 5m0s)
--requestheader-allowed-names stringSlice List of client certificate common names to allow to provide usernames in headers specified by --requestheader-username-headers. If empty, any client certificate validated by the authorities in --requestheader-client-ca-file is allowed.
--requestheader-client-ca-file string Root certificate bundle to use to verify client certificates on incoming requests before trusting usernames in headers specified by --requestheader-username-headers
--requestheader-extra-headers-prefix stringSlice List of request header prefixes to inspect. X-Remote-Extra- is suggested. (default [x-remote-extra-])
--requestheader-group-headers stringSlice List of request headers to inspect for groups. X-Remote-Group is suggested. (default [x-remote-group])
--requestheader-username-headers stringSlice List of request headers to inspect for usernames. X-Remote-User is common. (default [x-remote-user])
--secure-port int The port on which to serve HTTPS with authentication and authorization. If 0, don't serve HTTPS at all. (default 443)
--stderrthreshold severity logs at or above this threshold go to stderr (default 2)
--tls-ca-file string If set, this certificate authority will used for secure access from Admission Controllers. This must be a valid PEM-encoded CA bundle. Altneratively, the certificate authority can be appended to the certificate provided by --tls-cert-file.
--tls-cert-file string File containing the default x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If HTTPS serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to /var/run/kubernetes.
--tls-private-key-file string File containing the default x509 private key matching --tls-cert-file.
--tls-sni-cert-key namedCertKey A pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: "example.crt,example.key" or "foo.crt,foo.key:*.foo.com,foo.com". (default [])
-v, --v Level log level for V logs (default 0)
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging

panic: cluster doesn't provide client-ca-file

goroutine 1 [running]:
main.main()
/go/src/github.com/directxman12/k8s-prometheus-adapter/cmd/adapter/adapter.go:41 +0x114

packaging and running this on minikube / windows possible ?

I am trying to package and deploy this in windows - but no luck,
appreciate your input on whether it's possible?

when I run docker build -t spring-boot-hpa . its created the image successfully

when I deploy in minikube using kubectl create -f kube/deployment

get the following

backend-7f8577cdd5-tbs4w 0/1 ContainerCreating 0 67m
frontend-559bd7c667-5gnh7 0/1 ContainerCreating 0 67m
queue-58cf455966-64qn2 0/1 ContainerCreating 0 67m