Problems getting up and running… about haproxy-acme-validation-plugin

[hedefalk]

The first log line is from your plugin I see and seems to indicate "happy path", right? But the second one says it still serves a 502. I guess there's something funky with my haproxy.cfg anyways. I'm pretty new to it… I might as well post the entire thing here if you can spot something:

global
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    # Default ciphers to use on SSL-enabled listening sockets.
    # For more information, see ciphers(1SSL). This list is from:
    #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
    ssl-default-bind-options no-sslv3

    tune.ssl.default-dh-param 2048

    lua-load /etc/haproxy/acme-http01-webroot.lua

defaults
    log global
    mode    http

    option forwardfor
        option http-server-close

    option  httplog
    option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http


frontend http
    mode http
    bind *:80

    # Letsencrypt: https://github.com/janeczku/haproxy-acme-validation-plugin
    acl url_acme_http01 path_beg /.well-known/acme-challenge/
    http-request use-service lua.acme-http01 if METH_GET url_acme_http01

    #redirect scheme https code 301 if { hdr(Host) -i repo.woodenstake.se } !{ ssl_fc }
    #redirect scheme https code 301 if { hdr(Host) -i jenkins.woodenstake.se } !{ ssl_fc }


frontend https
    mode http

    bind *:443 ssl crt /etc/haproxy/startssl2.pem
#bind *:443 ssl crt /etc/letsencrypt/live/jenkins.woodenstake.se/haproxy.pem crt /etc/haproxy/startssl2.pem


    # Define hosts based on domain names
    acl host_jenkins hdr(host) -i jenkins.woodenstake.se
    acl host_repo hdr(host) -i repo.woodenstake.se
    acl host_jenkins_nas hdr(host) -i jenkins-nas.woodenstake.se
    acl host_transmission hdr(host) -i transmission.woodenstake.se

    use_backend jenkins if host_jenkins
    use_backend nexus if host_repo
    use_backend jenkins_nas if host_jenkins_nas
    use_backend transmission_nas if host_transmission

backend jenkins
    mode http
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    option httpchk HEAD / HTTP/1.1\r\nHost:localhost
    server jenkins_backend 127.0.0.1:8080

backend nexus
    mode http
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    option httpchk HEAD / HTTP/1.1\r\nHost:localhost
    server nexus_backend 127.0.0.1:8081

backend jenkins_nas
    mode http
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    option httpchk HEAD / HTTP/1.1\r\nHost:localhost
    server jenkins_nas_backend 10.0.1.30

backend transmission_nas
    mode http
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    option httpchk HEAD / HTTP/1.1\r\nHost:localhost
    server jenkins_nas_backend 10.0.1.30:8181

[hedefalk]

Feels like a spaghetti of http status codes I see here:

Ok, so I see in the logs that this lua service clearly sets 200. Then I also see a 502 (bad gateway) on the next line in haproxy log. But then the LE client tells me it gets a 400 (bad request). Just doesn't make any sense :´(

[hedefalk]

If I just put a file there manually and curl it myself I get the same:

~ $ curl -v jenkins.woodenstake.se/.well-known/acme-challenge/eBY2kFvH_GVcBMCMZM10eqEa3wR40M0ZLwy2y6Uk4V8
*   Trying 89.253.88.3...
* Connected to jenkins.woodenstake.se (89.253.88.3) port 80 (#0)
> GET /.well-known/acme-challenge/eBY2kFvH_GVcBMCMZM10eqEa3wR40M0ZLwy2y6Uk4V8 HTTP/1.1
> User-Agent: curl/7.41.0
> Host: jenkins.woodenstake.se
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 400 Bad request
< Cache-Control: no-cache
< Connection: close
< Content-Type: text/html
<
<html><body><h1>400 Bad request</h1>
Your browser sent an invalid request.
</body></html>

HTTP/1.0 502 Bad Gateway
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html><body><h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.
</body></html>

* Closing connection 0

haproxy.log looks pretty much identical…

[lukastribus]

It don't think it can access the necessary file because of chroot, can you try without it?

[hedefalk]

@lukastribus Pretty sure that's not it. I added a log just before writing the body:

applet:send(response)
core.Info("[acme] sent response: " .. response)

and I can see the file content in the haproxy log. So there's something later that makes the request fail.

[janeczku]

@hedefalk this looks like something happening after the Lua handler returns.
Are you running the latest stable (v1.6.2) ?
To debug this, i would recommend starting with a minimal config like this one: https://github.com/janeczku/haproxy-acme-validation-plugin/blob/master/haproxy.cfg.example

[hedefalk]

@janeczku

Yes, running latest stable 1.6.2.

I found something. I did what you suggested and used your stripped down config. And it worked. I then re-inserted my personal config and tested each change. What makes it break is that I have:

option forwardfor

under defaults. See http://cbonte.github.io/haproxy-dconv/configuration-1.6.html#4-option%20forwardfor.

This since I need "X-Forwarded-For" for most of my backends to tell the "real client IP". I guess what happens is that this directive is applied at the end which is after you already did AppletHTTP:start_response()and wrote to the body with applet:send(response) which makes stuff blow silently since you can't set new headers after this:

"AppletHTTP.start_response()
This function indicates to the HTTP engine that it can process and send the response headers. After this called we cannot add headers to the response; We cannot use the AppletHTTP:send() function if the AppletHTTP:start_response() is not called."

That's at least a guess.

I guess I could instead move my usage of

option forwardfor

to be under each individual backend and be done. I'll report back!

[hedefalk]

By the way, yesterday I continued googling for options and found:

https://coolaj86.com/articles/lets-encrypt-with-haproxy/

I guess you've seen that? Could I ask if you could elaborate on pros/cons of each approach? He's using the standalone server instead with alternative port. There are options for these under "testing":

--tls-sni-01-port TLS_SNI_01_PORT
                        Port number to perform tls-sni-01 challenge. Boulder
                        in testing mode defaults to 5001. (default: 443)
  --http-01-port HTTP01_PORT
                        Port used in the SimpleHttp challenge. (default: 80)

I think the whole thing is pretty confusing - what port to run the standalone server on and what port letsencrypt will challenge you on are obviously two entirely different things. It seems that the way he does it is to set a high port with --tls-sni-01-port but then he gets the challenge on 443 anyways. I can't seem to read that from the documentation so I'm wandering if this is likely to break in the future.

[hedefalk]

I moved the option forwardfor to each individual backend and now I have green beautiful adressbars on all my subdomains :)

Multiply domains on a single run didn't work with the client though. Multiple -dthat is. Don't know if that is a known…

Now I'm gonna look into your script and setup a cronjob. Thanks a lot for this!

[janeczku]

@hedefalk

By the way, yesterday I continued googling for options and found:
https://coolaj86.com/articles/lets-encrypt-with-haproxy/
I think the whole thing is pretty confusing

Exactly 😁
Thats way i wrote the plugin.

I didn't test multiple domains. Would be interesting to know what exactly doesn't work. Do you still have the client log?

[hedefalk]

@janeczku Reported here certbot/certbot#1946 (comment)

I might have been mistaken - sounds like I might have gotten SAN certificate for all subdomains, but under a folder in webroot for the first subdomain. Maybe. Can't test right now since I've reached some quota.

[janeczku]

@hedefalk Use --staging flag in the letsencrypt command for testing purposes. There is no rate limit there.

[hedefalk]

@janeczku But what does that entail? Any downsides? Those are pretty important sites for me… I'm testing but it's still production so to say :)

[janeczku]

Use the staging server to obtain test (invalid) certs

[janeczku]

Ah then you should not do that! 😄