jetstack / kube-lego Goto Github PK
View Code? Open in Web Editor NEWDEPRECATED: Automatically request certificates for Kubernetes Ingress resources from Let's Encrypt
License: Apache License 2.0
DEPRECATED: Automatically request certificates for Kubernetes Ingress resources from Let's Encrypt
License: Apache License 2.0
LEGO_URL
like nearly all ACME clients should default to Let's Encrypt production backend by default. Lego itself does that. However the YML given as example should override it with the staging environment.
This not only is desirable to make it consistent but also because it's easy to find Let's Encrypt staging URL, harder to find their production URL (as it's considered the default on all clients).
Background:
I have two existing GCE ingresses running without kube-lego. This is what the health checks look like:
$ gcloud compute http-health-checks list
NAME HOST PORT REQUEST_PATH
k8s-be-30804--eeba4b4d12737265 30804 /login
k8s-be-32516--eeba4b4d12737265 32516 /
k8s-be-32742--eeba4b4d12737265 32742 /healthz
The /login
path is for a legacy service that does not expose a dedicated, unauthenticated health(z) endpoint. /login
is used b/c it doesn't require authentication and returns a 200. The deployment behind this ingress exposes /login
as a readinessProbe
.
Issue:
Now I want to test kube-lego (0.1.2) with the echoserver sample. After deploying the echoserver ingress (and updating dns), I see a lot of errors in the kube-lego logs indicating that the reachability test is failing with wrong status code '502'
. Sure enough, looking at the GCE load balancer, I can see that none of my nodes for the /.well-known/acme-challenge/*
backend are healthy.
gcloud compute backend-services get-health k8s-be-32044--eeba4b4d12737265 | grep healthState
- healthState: UNHEALTHY
- healthState: UNHEALTHY
- healthState: UNHEALTHY
When I look a the health checks again, I see that there are two /login
paths. That doesn't seem right. The health check for the /.well-known/acme-challenge/*
should be /healthz
, right?
$ gcloud compute http-health-checks list
NAME HOST PORT REQUEST_PATH
k8s-be-30804--eeba4b4d12737265 30804 /login
k8s-be-32044--eeba4b4d12737265 32044 /login
k8s-be-32243--eeba4b4d12737265 32243 /
k8s-be-32516--eeba4b4d12737265 32516 /
k8s-be-32742--eeba4b4d12737265 32742 /healthz
$ gcloud compute http-health-checks describe k8s-be-32044--eeba4b4d12737265
checkIntervalSec: 1
creationTimestamp: '2016-09-07T07:12:18.014-07:00'
description: kubernetes L7 health check from readiness probe.
healthyThreshold: 1
host: ''
id: '304508149891440301'
kind: compute#httpHealthCheck
name: k8s-be-32044--eeba4b4d12737265
port: 32044
requestPath: /login
selfLink: https://www.googleapis.com/compute/v1/projects/imm-gce/global/httpHealthChecks/k8s-be-32044--eeba4b4d12737265
timeoutSec: 1
unhealthyThreshold: 10
If I manually change the path for the k8s-be-32044--eeba4b4d12737265
health check to /healthz
and then wait a few minutes, the backend for /.well-known/acme-challenge/*
becomes healthy and kube-lego is able to get a certificate.
I can't figure out how the health check for the challenge endpoint is getting created with /login
instead of /healthz
. Is it somehow picking it up from the pre-existing ingress? Or am I doing some wrong here? Any ideas? Thanks!
Looking in the logs for the kube-lego pod, i'm seeing "Error during getting secret: resource name may not be empty" appear quite frequently. kube-lego is able to fetch a certificate, but because of this error it appears to not store it and thus tries to fetch a new certificate, eventually hitting LE's rate limit on certificate issuance.
2016-11-20T12:17:22.492732131Z time="2016-11-20T12:17:22Z" level=info msg="requesting certificate for domain1.kingj.net,domain2.kingj.net" context="ingress_tls" name=domain1-ingress namespace=default
2016-11-20T12:17:23.207569614Z time="2016-11-20T12:17:23Z" level=debug msg="testing reachablity of http://domain2.kingj.net/.well-known/acme-challenge/_selftest" context=acme domain=domain2.kingj.net
2016-11-20T12:17:23.207887947Z time="2016-11-20T12:17:23Z" level=debug msg="testing reachablity of http://domain1.kingj.net/.well-known/acme-challenge/_selftest" context=acme domain=domain1.kingj.net
2016-11-20T12:17:24.507967398Z time="2016-11-20T12:17:24Z" level=debug msg="got authorization: &{URI:https://acme-v01.api.letsencrypt.org/acme/challenge/REMOVED Status:valid Identifier:{Type: Value:} Challenges:[] Combinations:[]}" context=acme domain=domain1.kingj.net
2016-11-20T12:17:24.508068395Z time="2016-11-20T12:17:24Z" level=info msg="authorization successful" context=acme domain=domain1.kingj.net
2016-11-20T12:17:24.508862168Z time="2016-11-20T12:17:24Z" level=debug msg="got authorization: &{URI:https://acme-v01.api.letsencrypt.org/acme/challenge/REMOVED Status:valid Identifier:{Type: Value:} Challenges:[] Combinations:[]}" context=acme domain=domain2.kingj.net
2016-11-20T12:17:24.508943840Z time="2016-11-20T12:17:24Z" level=info msg="authorization successful" context=acme domain=domain2.kingj.net
2016-11-20T12:17:25.726043026Z time="2016-11-20T12:17:25Z" level=info msg="successfully got certificate: domains=[domain1.kingj.net domain2.kingj.net] url=https://acme-v01.api.letsencrypt.org/acme/cert/REMOVED" context=acme
2016-11-20T12:17:25.726140077Z time="2016-11-20T12:17:25Z" level=debug msg="certificate pem data:"REMOVED" context=acme
2016-11-20T12:17:25.726308003Z time="2016-11-20T12:17:25Z" level=warning msg="Error during getting secret: resource name may not be empty" context=kubelego
2016-11-20T12:17:25.726400451Z time="2016-11-20T12:17:25Z" level=error msg="Error while process certificate requests: resource name may not be empty" context=kubelego
2016-11-20T12:17:25.726414706Z time="2016-11-20T12:17:25Z" level=debug msg="worker: done processing true" context=kubelego
The ingress i'm requesting a certificate for, the nginx LB and kube-lego are all part of the default namespace.
Hi!
This is a question regarding the best way to handle multiple domains for one service using the nginx ingress-controller. Currently I add the new TLS domain names to the ingress and duplicate the rules for each domain. Could this be written in a less verbose manner?
When adding a new domain name to the list a new SAN-cert that will be issued, hence it would make more sense to apply few bulk changes instead small, multiple changes, to minimize the risk of being blocked from making further certificate requests.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: mobile-web-ingress
namespace: production
annotations:
kubernetes.io/tls-acme: "true"
kubernetes.io/ingress.class: "nginx"
spec:
tls:
- hosts:
- m.domain.org
- m.domain.com
- m.domain.net
secretName: mobile-web-tls
rules:
- host: m.domain.org
http:
paths:
- path: /
backend:
serviceName: mobile-web-svc
servicePort: 80
- host: m.domain.com
http:
paths:
- path: /
backend:
serviceName: mobile-web-svc
servicePort: 80
- host: m.domain.net
http:
paths:
- path: /
backend:
serviceName: mobile-web-svc
servicePort: 80
Hello.
I'm trying to use kube-lego with nginx controller.
I used echoserver as explained in the example.
However the HTTP-01 challenge fails :
time="2016-09-21T18:33:14Z" level=warning msg="Error while obtaining certificate: Errors while obtaining cert: map[mydomain.com:acme: Error 403 - urn:acme:error:unauthorized - Invalid response from http://mydomain.com/.well-known/acme-challenge/OHTuEazqigcmn1x-kSXMmzWnHNRJTrLFNfB0kZL7Py8: \"CLIENT VALUES:\r\nclient_address=('10.x.x.x', 47151) (10.x.x.x)\r\ncommand=GET\r\npath=/.well-known/acme-challenge/OHTuEazqigcmn1x-k\"\nError Detail:\n\tValidation for mydomain.com:80\n\tResolved to:\n\t\t104.y.y.y\n\tUsed: 104.y.y.y\n\n]" context=acme
It looks normal as the echoserver catches the request and respond something not related to the token specified in the url.
How is that possible ? Is there something that I forgot or something missing in the doc ?
I only see error 502, no matter what I try.
My ingress:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress
annotations:
kubernetes.io/tls-acme: "true"
kubernetes.io/ingress.class: "gce"
spec:
tls:
- hosts:
- redacted.io
- www.redacted.io
secretName: tls
- hosts:
- staging.redacted.io
secretName: tls-staging
rules:
- host: staging.redacted.io
http:
paths:
- path: /*
backend:
serviceName: web-http-staging
servicePort: 80
- host: redacted.io
http:
paths:
- path: /*
backend:
serviceName: web-http
servicePort: 80
- host: www.redacted.io
http:
paths:
- path: /*
backend:
serviceName: web-http
servicePort: 80
And one of my services (The other looks exactly the same):
apiVersion: v1
kind: Service
metadata:
name: web-http-staging
labels:
app: web
tier: frontend
spec:
type: NodePort
ports:
# the port that this service should serve on
- name: http
port: 80
protocol: TCP
- name: https
port: 443
protocol: TCP
selector:
app: web-http-staging
tier: frontend
and lastly, my pod listens on 80 and 443. If I curl the internal service IP from a node, I get the correct response (200). Therefore the Load Balancer must fail.
The .well-known path is present in the GCE load balancer, and there are four backends -- and half of them has the health status 0/4, while the last two has 4/4. I have no idea why they are reporting has unhealthy, since none of my Pods have health checks and they are reporting 200, if you connect directly to the service.
Help is greatly appreciated. Thank you.
thank you for writing and publishing such a nice piece of software ๐
however, I'd prefer to not see my private keys in the log, or at least not on info-lvl. could you please change this?
2016/09/16 ..:..:.. [INFO][my.host.name] Server responded with a certificate.
time="2016-09-16T...." level=info msg="Got certs={my.host.name https://acme-v01.api.letsencrypt.org/acme/cert/..... https://acme-v01.api.letsencrypt.org/acme/reg/..... -----BEGIN RSA PRIVATE KEY-----\nMIIEowIBAAKCAQEAu4iFuMTP/YxdVeZBfbrbbAdyvm .......
i'm using version jetstack/kube-lego:0.1.2
Filter ingress objects based on namespace (which is equal to the namespace where kube-lego runs)
Requested in #33
I'm getting the follow error (and all that is in the logs) when the pod starts. I assume this is when kube-lego tries to talk to my kube-api
2016-10-18T00:28:29.478223138Z time="2016-10-18T00:28:29Z" level=info msg="kube-lego 0.1.2-03c268d7 starting" context=kubelego
2016-10-18T00:28:29.513084276Z time="2016-10-18T00:28:29Z" level=fatal msg="the server has asked for the client to provide credentials" context=kubelego
I am currently trying to setup a small bare-metal one-node "cluster". For this I would like to use kube-lego. I have read the docs, but up until now, kube-lego is starting and then halting after
2016-10-05T20:24:41.039507691Z time="2016-10-05T20:24:41Z" level=info msg="server listening on http://:8080/" context=acme
I am using the "usual" nginx-ingress controller, but without a service (like found in your example). The config looks like:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: {{template "fullname" . }}
spec:
replicas: 1
template:
metadata:
labels:
# Required for the auto-create kube-lego-nginx service to work.
app: "{{ .Chart.Name }}"
spec:
containers:
- name: kube-lego
image: jetstack/kube-lego:canary
imagePullPolicy: Always
ports:
- containerPort: 8080
env:
- name: LEGO_KUBE_API_URL
value: http://lego.xxx.yyy:32767
- name: LEGO_EMAIL
valueFrom:
configMapKeyRef:
name: {{template "fullname" . }}
key: lego.email
- name: LEGO_URL
valueFrom:
configMapKeyRef:
name: {{template "fullname" . }}
key: lego.url
- name: LEGO_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: LEGO_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
readinessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
timeoutSeconds: 1
My ingress is configured quite similar to the one found in your examples:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: {{ template "fullname" . }}
labels:
app: "{{ .Chart.Name }}"
annotations:
kubernetes.io/tls-acme: "true"
spec:
tls:
- secretName: {{ .Chart.Name }}-tls
hosts:
- {{ .Values.hostName }}
rules:
- host: {{ .Values.hostName }}
http:
paths:
- path: /
backend:
serviceName: {{ template "fullname" . }}
servicePort: 80
For http, everything is working fine and now I would like to setup https. Any help on this one?
This is more of a support request, then a bug, I know, but I was unsure how to reach you.
When I create more than one ingress resource with tls host entries, they aren't added to the rules of the kube-lego ingress config.
I have to manually add the hosts to make sure the /.well-known/...
is proxied to the kube-lego service.
Since gcr.io/google_containers/nginx-ingress-controller:0.8.1
all the required changes are merged
(Warning: This report is a bit fuzzy)
I had kube-lego setup to update the certificates if they had 30 days or less to expire. Looking at the kubectl logs
output, this was performed correctly on kube-lego side. It was reporting that the certificate it was seeing had 70 days to go, which matched the timestamp of the secret associated with it.
However, no matter how long I waited, the actual certificate as seen by my external checker was the old one, with 20 days left to go.
I started looking for clues, and I found https://code.google.com/p/google-cloud-platform/issues/detail?id=105
This suggests to use kubectl replace --force -f secret.json
, so I did kubectl get -o json secret my-certificate
and fed that backt o the kubectl replace
command. at which point the certificate seemed to have been updated. So maybe GCE HTTP(s) Load Balancers require us to destroy the resource first and to re-create it, in order to force a reload?
I'm not 100% sure if this was the correct symptom/fix yet, but at this moment I highly suspect that this line here needs to be Delete+Create instead of update
Again, sorry about the fuzzy report, but I'm also trying to figure out what is going on here :/
Thanks for the nice tool. I am currently using it with staging url but for some services, I would prefer to get the cert from production LE CA rather than staging. So ability to choose the provider for each ingress with annotation can be quite handy. So the scenario will be like this
default: use LEGO_URL configured value for all the ingress where no annotation is specified to override it.
Have a support for annotation as:
`kubernetes.io/tls-acme-lego-url: "https://acme-staging.api.letsencrypt.org/directory"
the value can be staging or prod depending upon use case. This can help in avoiding the rate limiting when you testing some stuff and use production once you done.
When using the example for NGINX on GCE, everything seems to work, but I am not getting the headers from the original client. It appears that the values for
X-Real-IP & X-Forwarded-For are the private ip addr of the gke ingress controller.
In nginx examples, I saw a command
kubectl apply -f lego/service.yaml
But i did't find this file
It would be helpful if kube-lego
binary spit out an INFO line that told us which endpoint it's connecting to.
Anyway, these two screenshots illustrate the issue. I was under the impression that the staging endpoint didn't enforce limits? Is it possible kube-lego
isn't actually reading the LEGO_URL
environment variable?
Hi,
For various reasons I need to have the different paths for a host spread across different ingress objects (mainly to enable http basic-auth on some paths only, since there's no way to set this at the "rule" level and it can only be done using ingress-level annotations)
I couldn't find a way to make kube-lego play nice with this. I do want to share the same certificate/secret for the same host across these different ingress objects, but it complains with :
the secret xxxxxx/default is used multiple times. These linked TLS ingress elements where ignored:
And it then ignores all TLS ingress elements, not just additional duplicates but even the first definition it encounters.
Is there a workaround I'm missing ? Could kube-lego handle duplicates more gracefully maybe (gather the list of hosts/secrets needed across all ingresses, de-duplicate if needed, and request) ?
Hi!
I got an environment up running with two namespaces, production and staging. It is working fine with kube-lego and the nginx-ingress thanks to this project.
The other day I started a web socket server in production and all was nice and dandy. I set sessionAffinity to ClientIP for the services involved and it went smooth. When creating the second web socket server in staging, it didn't go that well. Actually, I'm not certain were the error is at the moment but I suspect when with two nginx-ingress replicas the sessionAffinity breaks. Setting the replicas of nginx-ingress to 1 will make it work.
Anyone who have had the same experience and have a recipe to get it working to have redundant nginx-ingresses?
Like gce
Hi,
I was a bit confused recently why kube-lego
is not working on my setup. After some debugging, I've noticed that it replaced all selectors I've defined in service.yml
with just app=kube-lego
.
Is it really needed? For example, I'm using a bit different naming and had to add extra label just to let it work. If not, at least it would be great to document this somewhere.
Should code be belowใin tls.go?
if !i.newCertNeeded(i.ingress.kubelego.LegoMinimumValidity()) {
i.Log().Infof("no cert request needed")
return nil
}
$ kubectl describe ing proding --namespace prodenv
Name: proding
Namespace: prodenv
Address:
Default backend: default-http-backend:80 (10.48.2.7:8080)
TLS:
proding-tls terminates www.{foo}.com,{foo}.com,www.{foo}.design,{foo}.design
Rules:
Host Path Backends
---- ---- --------
www.{foo}.com
/ jackserver:80 (<none>)
{foo}.com
/ jackserver:80 (<none>)
www.{foo}.design
/ pylonserver:80 (<none>)
{foo}.design
/ pylonserver:80 (<none>)
Annotations:
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
15m 15m 1 {nginx-ingress-controller } Normal CREATE prodenv/proding
15m 15m 1 {nginx-ingress-controller } Normal CREATE ip: 104.196.29.121
15m 15m 1 {nginx-ingress-controller } Normal UPDATE prodenv/proding
14m 14m 1 {nginx-ingress-controller } Normal DELETE ip: 104.196.29.121
14m 14m 1 {nginx-ingress-controller } Normal CREATE prodenv/proding
14m 14m 1 {nginx-ingress-controller } Normal UPDATE prodenv/proding
14m 14m 1 {nginx-ingress-controller } Normal CREATE ip: 104.196.29.121
9m 9m 1 {nginx-ingress-controller } Normal CREATE prodenv/proding
9m 9m 1 {nginx-ingress-controller } Normal CREATE ip: 104.196.29.121
9m 9m 1 {nginx-ingress-controller } Normal UPDATE prodenv/proding
8m 8m 1 {nginx-ingress-controller } Normal CREATE prodenv/proding
8m 8m 1 {nginx-ingress-controller } Normal DELETE ip: 104.196.29.121
8m 8m 1 {nginx-ingress-controller } Normal UPDATE prodenv/proding
8m 8m 1 {nginx-ingress-controller } Normal CREATE ip: 104.196.29.121
2m 2m 1 {nginx-ingress-controller } Normal CREATE prodenv/proding
2m 2m 1 {nginx-ingress-controller } Normal UPDATE prodenv/proding
2m 2m 1 {nginx-ingress-controller } Normal CREATE ip: 104.196.29.121
2m 2m 1 {nginx-ingress-controller } Normal CREATE prodenv/proding
2m 2m 1 {nginx-ingress-controller } Normal CREATE ip: 104.196.29.121
2m 2m 1 {nginx-ingress-controller } Normal UPDATE prodenv/proding
1m 1m 1 {nginx-ingress-controller } Normal DELETE ip: 104.196.29.121
The backends were reachable initially but then the wonkyness started.
I am happy to provide any debugging information.
We use kube2sky for DNS inside our K8S cluster.
It was down for some reason. but I haven't noticed that. When I created an ingress which required a certificate kube-lego did not handle the DNS problem quite well:
time="2016-12-09T17:16:54Z" level=info msg="requesting certificate for domain.foo.com" context="ingress_tls" name=https namespace="ece8168e-f5c7-4f41-9469-702f1eb2e4ec"
time="2016-12-09T17:16:54Z" level=info msg="creating new secret" context=secret name=kube-lego-account namespace="a12ddb33-13d4-43e4-9cfe-bf7e5b90935d"
time="2016-12-09T17:17:39Z" level=info msg="creating new secret" context=secret name=https namespace="ece8168e-f5c7-4f41-9469-702f1eb2e4ec"
time="2016-12-09T17:17:39Z" level=error msg="Error while process certificate requests: Secret \"https\" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value]" context=kubelego
Looks like if DNS is down kubelego fails silently and then tries to create an empty secret.
Would it be possible to build a Health Check into Kube-Lego itself? I feel like this would be a simplistic enough task, and it would allow the kubernetes server to use liveliness probes with kube-lego. If someone points me in the right direction, perhaps I could help with this.
Hi!
To start with, thanks for your good work with kube-lego!
I've setup kube-lego with gce and it works fine. The certificates are requested and deployed for two sites, one mobile and one desktop site. However, only the mobile site is reachable, the desktop site returns default backend - 404.
The setup files I've used are https://gist.github.com/johnparn/ce0e025e8c015de812c0b84ef8b1faf9
Containers for both mobile and desktop and mobile are exposed on port 80. The only difference that I've spotted is that in the GCE Load Balancer for mobile service there is a path rule with All unmatched (default) for that particular host name.
This rule is obviously missing in GCE LB for desktop and I believe this is the problem.
However, I tried creating a corresponding rule for the desktop LB but I don't seem to be able to create All unmatched (default) rule for the desktop host, well not by using the GUI. And I want to to make sure in case I have to rerun the scripts that the rule actually is created.
Any insights appreciated!
// John
Kube-lego's log show below and not work. It work again after pod restart :
E1121 12:01:09.104331 1 runtime.go:58] Recovered from panic: &runtime.TypeAssertionError{interfaceString:"interface {}", concreteString:"cache.DeletedFinalStateUnknown", assertedString:"*extensions.Ingress", missingMethod:""} (interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *extensions.Ingress)
/go/src/github.com/jetstack/kube-lego/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go:52
/go/src/github.com/jetstack/kube-lego/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go:40
/usr/local/go/src/runtime/asm_amd64.s:472
/usr/local/go/src/runtime/panic.go:443
/usr/local/go/src/runtime/iface.go:182
/go/src/github.com/jetstack/kube-lego/pkg/kubelego/watch.go:76
/go/src/github.com/jetstack/kube-lego/vendor/k8s.io/kubernetes/pkg/controller/framework/controller.go:178
<autogenerated>:25
/go/src/github.com/jetstack/kube-lego/vendor/k8s.io/kubernetes/pkg/controller/framework/controller.go:248
/go/src/github.com/jetstack/kube-lego/vendor/k8s.io/kubernetes/pkg/controller/framework/controller.go:122
/go/src/github.com/jetstack/kube-lego/vendor/k8s.io/kubernetes/pkg/controller/framework/controller.go:97
/go/src/github.com/jetstack/kube-lego/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go:66
/go/src/github.com/jetstack/kube-lego/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go:67
/go/src/github.com/jetstack/kube-lego/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go:47
/go/src/github.com/jetstack/kube-lego/vendor/k8s.io/kubernetes/pkg/controller/framework/controller.go:97
/usr/local/go/src/runtime/asm_amd64.s:1998
E1122 01:55:41.157331 1 runtime.go:58] Recovered from panic: &runtime.TypeAssertionError{interfaceString:"interface {}", concreteString:"cache.DeletedFinalStateUnknown", assertedString:"*extensions.Ingress", missingMethod:""} (interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *extensions.Ingress)
/go/src/github.com/jetstack/kube-lego/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go:52
/go/src/github.com/jetstack/kube-lego/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go:40
/usr/local/go/src/runtime/asm_amd64.s:472
/usr/local/go/src/runtime/panic.go:443
/usr/local/go/src/runtime/iface.go:182
/go/src/github.com/jetstack/kube-lego/pkg/kubelego/watch.go:76
/go/src/github.com/jetstack/kube-lego/vendor/k8s.io/kubernetes/pkg/controller/framework/controller.go:178
<autogenerated>:25
/go/src/github.com/jetstack/kube-lego/vendor/k8s.io/kubernetes/pkg/controller/framework/controller.go:248
/go/src/github.com/jetstack/kube-lego/vendor/k8s.io/kubernetes/pkg/controller/framework/controller.go:122
/go/src/github.com/jetstack/kube-lego/vendor/k8s.io/kubernetes/pkg/controller/framework/controller.go:97
/go/src/github.com/jetstack/kube-lego/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go:66
/go/src/github.com/jetstack/kube-lego/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go:67
/go/src/github.com/jetstack/kube-lego/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go:47
/go/src/github.com/jetstack/kube-lego/vendor/k8s.io/kubernetes/pkg/controller/framework/controller.go:97
/usr/local/go/src/runtime/asm_amd64.s:1998
k8s 's version:
Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.6", GitCommit:"ae4550cc9c89a593bcda6678df201db1b208133b", GitTreeState:"clean", BuildDate:"2016-08-26T18:13:23Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.6", GitCommit:"ae4550cc9c89a593bcda6678df201db1b208133b", GitTreeState:"clean", BuildDate:"2016-08-26T18:06:06Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
From looking at the issues, it seems that 1.2 can fail to renew certs (or possibly always causes an 8 hour expiration?) because it was using CheckInterval instead of MinimalValidity. I see that this is fixed in the canary tag, but that appears to be tracking master, and I don't want to depend on a mutable tag.
Any thoughts on cutting a new release with the fixed validity issue?
edit: Though I suppose if you set checkinterval to 5 minutes[0], you shouldn't have more than 5 minutes of invalid certs.
[0]: I think this minimum value is too high, the check seems like a lightweight thing, I don't think setting it to something outrageously low like 10 seconds would really cause problems, I may be wrong
edit: I found a couple of bugs that I have patches for on my fork, will submit prs soon
We have 3 different stages of our app. They are all running on a different domain.
Each of them already obtained a ssl certificate and saved it as a secret.
This is a screenshot showing the secrets:
But for only one of the stages the ssl cert is actually set in the LoadBalancer. All others are only available through HTTP and not HTTP&HTTPS.
Here are all SSL certs available in the LoadBalancer settings:
Any idea why it only adds one certificate to the Google Cloud Engine Network settings?
Hello.
After adding the kube lego deployment these lines are printed :
time="2016-09-21T16:30:18Z" level=warning msg="wrong status code '401'" context=acme host=my-host.com
time="2016-09-21T16:30:18Z" level=warning msg="Error while obtaining certificate: reachabily test failed for this cert" context=acme
time="2016-09-21T16:30:23Z" level=debug msg="testing reachablity of http://my-host.com/.well-known/acme-challenge/_selftest" context=acme host=my-host.com
I'm using nginx controller with http auth.
Can this be the cause ? How can pass the reachabily test ?
Hi!
We've been working on a new web site that will replace an old one. When launching we will change the DNS to point to the containers at GKE, running kube-lego. When doing this which is the best way?
Alternative 1
Alternative 2
Alternative 2 assumes that kube-lego keeps trying to verify the new IP-address until it succeeds. Is that the way it works? This would likely also mean that you can prepare the site in advance and the switch will mean less downtime.
In case it is a SAN cert, will kube-lego continue requesting new certs even if one of the domains is not pointing at the GKE server at the moment and will that exhaust the rate limit at Let's encrypt.
First off, this is awesome. Thank you so much for providing this amazing tool. It took me a couple hours, but I've got it up and running (using nginx
), and everything is working great.
In my particular use case, I have an application that powers many different domains, say aaa.com
, bbb.com
, ccc.com
, etc. Adding and removing domains needs to be automated.
What's the best (or recommended) way to handle this type of scenario? I'm thinking that creating separate ingresses within the application's namespace for each domain would be the best approach. (The ingresses would be created programmatically using the Kubernetes API.) I've manually tested this, but I'm not sure how scalable this approach is, or if there are other issues around having many ingresses.
I'm relatively new to Kubernetes and Let's Encrypt, any help is greatly appreciated. Thanks!
kube-lego/examples/nginx/20-nginx-configmap.yaml contains a lot of overrides of the default values.
It's worth putting a comment for each to mark:
Are there plans to support the DNS Challenge as well as the current challenge?
kubernetes.io/tls-acme="false"
)coming from #12
I followed the GCE example but continually see logs like this in the kube-lego pod:
time="2016-08-14T16:50:19Z" level=debug msg="testing reachablity of http://foo.redacted.com/.well-known/acme-challenge/_selftest" context=acme host=foo.redacted.com
time="2016-08-14T16:50:19Z" level=warning msg="wrong status code '404'" context=acme host=foo.redacted.com
My DNS record is pointed at the Ingress external IP for foo.redcated.com
. I can curl it via http, but only the html index is accessible: all of its css js and images 404.
Are there any suggested debugging steps? I previously had this working but wanted to try out the GCE solution.
Edit: I also see a bunch of wrong status code '502'
logs after killing the kube-lego pod and watching the new one come up.
Edit 2: looks like the css/js/images issue was fixed by updating - path: /
to - path: /*
in my ingress object.
I would love a use this project for our Kubernetes cluster running in AWS. Unfortunately it doesn't seem that this project supports AWS ElasticLoadBalancers yet. I know that lego
itself supports Route53 which is Amazons DNS provider, we just need the load balancer integration.
Whould this be possible to implement?
Version of kube-lego:
2016-10-24T13:22:49.775587542Z time="2016-10-24T13:22:49Z" level=info msg="kube-lego 0.1.2-a823e819 starting" context=kubelego
2016-10-24T13:22:49.804969421Z time="2016-10-24T13:22:49Z" level=info msg="connected to kubernetes api v1.4.3" context=kubelego
Logs for one domain:
2016-10-24T13:23:22.062415223Z time="2016-10-24T13:23:22Z" level=info msg="cert expires soon so renew" context="ingress_tls" expire_time=2016-11-14 16:27:00 +0000 UTC name=kubedash namespace=kube-system
2016-10-24T13:23:22.062443256Z time="2016-10-24T13:23:22Z" level=info msg="requesting certificate for kubedash.test.gelato.tech" context="ingress_tls" name=kubedash namespace=kube-system
2016-10-24T13:23:25.336489023Z time="2016-10-24T13:23:25Z" level=info msg="authorization successful" context=acme domain=kubedash.test.gelato.tech
2016-10-24T13:23:25.336529610Z time="2016-10-24T13:23:25Z" level=warning msg="authorization failed for some domains" context=acme failed_domains=[ kubedash.test.gelato.tech]
And one more error from logs:
2016-10-24T13:36:36.966341172Z time="2016-10-24T13:36:36Z" level=error msg="Error while process certificate requests: error getting certificate: 400 urn:acme:error:malformed: Error creating new cert :: at least one DNS name is required, error getting certificate: 400 urn:acme:error:malformed: Error creating new cert :: at least one DNS name is required, error getting certificate: 400 urn:acme:error:malformed: Error creating new cert :: at least one DNS name is required, error getting certificate: 400 urn:acme:error:malformed: Error creating new cert :: at least one DNS name is required, error getting certificate: 400 urn:acme:error:malformed: Error creating new cert :: at least one DNS name is required, error getting certificate: 400 urn:acme:error:malformed: Error creating new cert :: at least one DNS name is required, error getting certificate: 400 urn:acme:error:malformed: Error creating new cert :: at least one DNS name is required, error getting certificate: 400 urn:acme:error:malformed: Error creating new cert :: at least one DNS name is required, error getting certificate: 400 urn:acme:error:malformed: Error creating new cert :: at least one DNS name is required" context=kubelego
So I had this running with the default setup. It fetched the certs and was able to see the site up and running! Then I wanted to change some names and reorganize the files so I tore down the cluster and rebuilt it. I haven't been able to get it running a second time! I've gone ahead and created a gist with the appropriate files.
I tried some of the troubleshooting steps from #15 but was again unable to get the certs to authenticate.
Any help would be greatly appreciated @simonswine!
When kube-lego starts up and tries to obtain the cert, I see 3 repetitions of these logs (domain masked):
time="2016-06-06T12:54:05Z" level=debug msg="testing reachablity of http://foo.com/.well-known/acme-challenge/_selftest" context=acme host=foo.com
time="2016-06-06T12:54:05Z" level=warning msg="wrong status code '503'" context=acme host=foo.com
This results in:
time="2016-06-06T12:54:06Z" level=info msg="initialize lego acme connection" context=acme
2016/06/06 12:54:06 [INFO][foo.com] acme: Obtaining bundled SAN certificate
t
Error 400 - urn:acme:error:badNonce - JWS has invalid anti-replay nonce tlGC7MNn-udU86kjs_DOK1pTWynV5P3kYjNmDuVIruo]" context=acme
Are those related? I saw many others with similar JWS errors and read possible solutions (retry ~6 times, change email address) but still no luck.
File a PR against nginx-ingress-controller
In the screencast example, it shows two ingress resources: echoserver
and kube-lego
. Each one has 2 external IP addresses. Which one do you use for the demo.kube-lego.jetstack.net
A record?
It'd be helpful to include this in the docs.
Maybe it's a misunderstanding on my side but I just activated kube-lego for a gce ingress with a manually preinstalled certificate which expires in 6 days.
This is the ingress configuration:
# ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress
namespace: default
annotations:
kubernetes.io/tls-acme: "true"
kubernetes.io/ingress.class: "gce"
spec:
tls:
- hosts:
- sub.example.com
secretName: tls-secret
backend:
serviceName: endpoint
servicePort: api
rules:
- host: sub.example.com
http:
paths:
- path: /something
backend:
serviceName: some-service
servicePort: http
I set explicitly LEGO_MINIMUM_VALIDITY
to "720h".
The log of the kube-lego pod is showing this:
time="2016-08-22T20:00:15Z" level=info msg="cert expires in 6.6 days, no renewal needed" context="ingress_tls" expire_time=2016-08-29 09:53:00 +0000 UTC name=ingress namespace=default
So did I misunderstood the minimum validity, and if so when is the cert renewed and how can I control the interval?
Greetings
When the kube-lego
ingress is created, it uses both nginx-ingress-controller
and loadbalancer-controller
:
Name: kube-lego
Namespace: default
Address: 5.6.7.8,1.2.3.4
Default backend: default-http-backend:80 (10.84.0.3:8080)
Rules:
Host Path Backends
---- ---- --------
echo.example.com
/.well-known/acme-challenge kube-lego:8080 (<none>)
Annotations:
backends: {"k8s-be-31590--64893abc10ca8449":"HEALTHY"}
forwarding-rule: k8s-fw-default-kube-lego--64893abc10ca8449
ssl-redirect: false
target-proxy: k8s-tp-default-kube-lego--64893abc10ca8449
url-map: k8s-um-default-kube-lego--64893abc10ca8449
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
8m 8m 1 {nginx-ingress-controller } Normal CREATE default/kube-lego
8m 8m 1 {loadbalancer-controller } Normal ADD default/kube-lego
8m 7m 2 {nginx-ingress-controller } Normal CREATE ip: 1.2.3.4
7m 7m 1 {loadbalancer-controller } Normal CREATE ip: 5.6.7.8
7m 7m 1 {loadbalancer-controller } Warning Status Operation cannot be fulfilled on ingresses.extensions "kube-lego": the object has been modified; please apply your changes to the latest version and try again
8m 2m 7 {nginx-ingress-controller } Normal UPDATE default/kube-lego
Probably it should be created with annotation kubernetes.io/ingress.class: "nginx"
to disable GLBC.
Are there any plans on supporting multiple clusters hosting same domain? In our case we have Kubernetes cluster(s) in multiple regions and use DNS-based for traffic routing. This has the issue that only single cluster would be able to obtain the certificate with the current kube-lego integration.
I'm currently writing a patch that allows kube-lego to pass-thru the challenge request to other clusters (query DNS for list of the clusters (a bit of torn between SRV and A records) & try to retrieve the challenge response from each. If anyone returns 200, return that to the client). Is this design something that could be upstreamed or is this something that's should to be addressed in some other way/simply not in the scope of the project?
The kube-lego
container has LEGO_URL
set to production:
Containers:
kube-lego:
Container ID: docker://...
Image: jetstack/kube-lego:0.1.2
Image ID: docker://...
Port: 8080/TCP
State: Running
Ready: True
Restart Count: 0
Readiness: http-get http://:8080/healthz delay=5s timeout=1s period=10s #success=1 #failure=3
Environment Variables:
LEGO_EMAIL: [email protected]
LEGO_NAMESPACE: default (v1:metadata.namespace)
LEGO_POD_IP: (v1:status.podIP)
LEGO_URL: https://acme-v01.api.letsencrypt.org/directory
However its retrieving the staging TLS certificate:
time="2016-08-31T11:51:50Z" level=info msg="CREATE foo/ingress" context=kubelego
time="2016-08-31T11:51:50Z" level=info msg="creating new secret" context=secret name=foo-tls-certificate namespace=foo
time="2016-08-31T11:51:50Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=ingress namespace=foo
time="2016-08-31T11:51:50Z" level=info msg="requesting certificate for foo.example.com" context="ingress_tls" name=ingress namespace=foo
time="2016-08-31T11:51:50Z" level=debug msg="testing reachablity of http://foo.example.com/.well-known/acme-challenge/_selftest" context=acme host=foo.example.com
time="2016-08-31T11:51:50Z" level=warning msg="wrong status code '502'" context=acme host=foo.example.com
time="2016-08-31T11:52:08Z" level=debug msg="testing reachablity of http://foo.example.com/.well-known/acme-challenge/_selftest" context=acme host=foo.example.com
time="2016-08-31T11:52:08Z" level=warning msg="wrong status code '503'" context=acme host=foo.example.com
time="2016-08-31T11:52:16Z" level=debug msg="testing reachablity of http://foo.example.com/.well-known/acme-challenge/_selftest" context=acme host=foo.example.com
2016/08/31 11:52:17 [INFO][foo.example.com] acme: Obtaining bundled SAN certificate
time="2016-08-31T11:52:18Z" level=warning msg="Error while obtaining certificate: Errors while obtaining cert: map[foo.example.com:acme: Error 400 - urn:acme:error:badNonce - JWS has invalid anti-replay nonce *************************************]" context=acme
time="2016-08-31T11:52:34Z" level=debug msg="testing reachablity of http://foo.example.com/.well-known/acme-challenge/_selftest" context=acme host=foo.example.com
2016/08/31 11:52:34 [INFO][foo.example.com] acme: Obtaining bundled SAN certificate
2016/08/31 11:52:35 [INFO][foo.example.com] acme: Trying to solve HTTP-01
2016/08/31 11:52:35 [INFO][foo.example.com] The server validated our request
2016/08/31 11:52:35 [INFO][foo.example.com] acme: Validations succeeded; requesting certificates
2016/08/31 11:52:36 [INFO][foo.example.com] Server responded with a certificate.
time="2016-08-31T11:52:36Z" level=info msg="Got certs={foo.example.com https://acme-staging.api.letsencrypt.org/acme/cert/***************** https://acme-staging.api.letsencrypt.org/acme/reg/123456 -----BEGIN RSA PRIVATE KEY-----\n*************************
Note: I kept Nginx Ingress controller hsts-include-subdomains
to the default (true
).
Should I expect to receive any e-mail at [email protected]? Should I delete the token or something else if I change from staging to production?
When a secret already exists with the the name configured in the TLS directive for ingress a duplicate or replacement secret cannot be created (error Invalid value: \"kubernetes.io/tls\": field is immutable"
).
As a result, kube-lego repeatedly requests a cert for the same subdomain, meaning you only have a few seconds before hitting the Duplicate Certificate rate limit
time="2016-09-16T15:14:58Z" level=debug msg="testing reachablity of http://sub.example.com/.well-known/acme-challenge/_selftest" context=acme host=sub.example.com
time="2016-09-16T15:14:58Z" level=info msg="initialize lego acme connection" context=acme
2016/09/16 15:14:58 [INFO][sub.example.com] acme: Obtaining bundled SAN certificate
2016/09/16 15:14:58 [INFO][sub.example.com] acme: Trying to solve HTTP-01
2016/09/16 15:14:58 [INFO][sub.example.com] The server validated our request
2016/09/16 15:14:58 [INFO][sub.example.com] acme: Validations succeeded; requesting certificates
time="2016-09-16T15:14:59Z" level=warning msg="Error while obtaining certificate: Errors while obtaining cert: map[sub.example.com:acme: Error 429 - urn:acme:error:rateLimited - Error creating new cert :: Too many certificates already issued for exact set of domains: sub.example.com]" context=acme
This could be avoided in two ways:
Especially marked ingress ressources (annotation: kubernetes.io/ingress.class: gce
), should be set up in a way, that no nginx is needed. (Single ressource, kube-lego svc in same NS)
(related to #6)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.