crash with ssh connection about haproxy

[TimWolla]

Something does not add up here: With the configuration you posted HAProxy is not even involved in the SSH connection, because the relevant frontend is commented out. Commenting it in is not possible, because OpenSSH is already listening on port 22, thus HAProxy can't bind to 22.

Can you create a self-contained Dockerfile that does not rely on external services (such as keyrock)?

[agaldemas]

Hello,
I reproduced the issue after removing any external service, even any container except portainer,

• with version 1.8 as root image, there's no crash, but seems that the thread die,
• with 1.9 got the invalid pointer error reported.

I discover this issue while trying to use an haproxy-wi container to connect to an haproxy container (through ssh).

I started with 1.8 from dockerhub image, as I didn't manage to connect to the container through ssh,
I tried version 1.9 and haproxy crashed each time I tried to connect via ssh to the container (through port 22)

May be there is a particularity in my docker configuration (under MacOS), I will try on a virtualbox VM
May be my post-installation in Dockerfile break something ?
Or service ssh restart before starting haproxy in docker-entrypoint.sh ?

[TimWolla]

I reproduced the issue after removing any external service, even any container except portainer,

Can you provide that self-contained Dockerfile with a list of exact steps, please? Ideally I should be able to just docker build the image, docker run the container and then connect to reproduce the issue.

[agaldemas]

Here is the Dockerfile(.txt)
Dockerfile.txt
just change the version of the base haproxy image, 1.8 or 1.9

But, think it can be considered as a kind of corner case...
Because as an haproxy newbie, but as experimented senior SW dev&architect, I din't read a lot before starting testing,
and later I realized it's not a useful idea to install openssh on the same container, instead of having a specific one for it, beside haproxy, which is a smarter and better practice ;O).

Anyway sometimes strange corner case bugs, can reveal a true hidden issue,

You have the docker file, the docker-compose file part above,
to reproduce the issue, try to ssh the haproxy with ssh root@localhost:2222, or even form the haproxy host itself
answer 'screencast' the password (or whatever, it doesn't matter), at the end of process on my configuration the container die in version 1.9 with invalid pointer exception reported.
Waitng for your feedback, Have a nice test

[TimWolla]

Waitng for your feedback, Have a nice test

What is the HAProxy configuration? As outlined in my initial comment I am unable to reproduce the issue with the configuration you provided above, because it relies on external services and does not start because it is unable to resolve then.

[agaldemas]

here is the haproxy.conf:

global
    #log /dev/log local0
    #log /dev/log local1 notice
    #chroot /var/lib/haproxy
    #stats socket /run/haproxy/admin.sock mode 660 level admin
    #stats timeout 30s
    user root
    group root
    daemon

    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    # Default ciphers to use on SSL-enabled listening sockets.
    # For more information, see ciphers(1SSL). This list is from:
    #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
    ssl-default-bind-options no-sslv3

defaults
    log global
    mode http
    #option httplog
    #option dontlognull
    timeout connect 5000
    timeout client  50000
    timeout server  50000

any other block is commented.
I you don't reproduce it, probably it's an issue with my configuration (Docker under MacOS),
On which environment are you trying to reproduce ?

[TimWolla]

Okay. HAProxy clearly is not involved within the connection here (it does not even listen to any ports at all) and indeed I'm unable to reproduce the issue with the haproxy:1.9 image using Docker 18.09.2 on Ubuntu.

The only thing that might be able to give some insight is the core dump or attaching to the running process with gdb and waiting for the crash to get a backtrace.

[agaldemas]

I know that's why it is so weird crash ! It has been a long time I used gdb, never with Docker Can you give me hints for that Then I could generate the backtrace and send it to you CU Le jeu. 14 févr. 2019 à 11:57, Tim Düsterhus <[email protected]> a écrit :

…

Okay. HAProxy clearly is not involved within the connection here (it does not even listen to any ports at all) and indeed I'm unable to reproduce the issue with the haproxy:1.9 image using Docker 18.09.2 on Ubuntu. The only thing that might be able to give some insight is the core dump or attaching to the running process with gdb and waiting for the crash to get a backtrace. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#41 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ATWJo8R2uNwRkEj5_lva7EqKYvfvtgIKks5vNUEagaJpZM4a0O9-> .

[TimWolla]

I guess the easiest way is finding the PID of the worker process and then attaching to it (it's the one with the higher PID). In my example I killed the worker with kill -SEGV for a demonstration. I'm not sure whether you can do this easily on your Mac, because Docker runs in lightweight VM if I understand it correctly.

[timwolla@/t/crasher]ps aux |grep haproxy
root     19648  0.0  0.0   1048     4 ?        Ss   13:29   0:00 /dev/init -- /docker-entrypoint.sh haproxy -f /usr/local/etc/haproxy/haproxy.cfg
root     19740  0.0  0.0  43548  4428 ?        S    13:29   0:00 haproxy -W -db -f /usr/local/etc/haproxy/haproxy.cfg
root     19793  0.0  0.0  43548  2756 ?        S    13:29   0:00 haproxy -W -db -f /usr/local/etc/haproxy/haproxy.cfg
timwolla 19925  0.0  0.0  15776   968 pts/19   S+   13:30   0:00 grep --color=auto haproxy
[timwolla@/t/crasher]sudo gdb -p 19793
*snip*

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
0x00007f4ed48e70a3 in epoll_wait () from target:/lib/x86_64-linux-gnu/libc.so.6
(gdb) cont
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0x00007f4ed48e70a3 in epoll_wait () from target:/lib/x86_64-linux-gnu/libc.so.6
(gdb) bt full
#0  0x00007f4ed48e70a3 in epoll_wait () from target:/lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x000055b3bd5c677d in _do_poll (p=<optimized out>, exp=<optimized out>) at src/ev_epoll.c:156
        timeout = 1000
        status = <optimized out>
        fd = <optimized out>
        count = <optimized out>
        updt_idx = <optimized out>
        old_fd = <optimized out>
#2  0x000055b3bd6681c2 in run_poll_loop () at src/haproxy.c:2652
        next = <optimized out>
        exp = <optimized out>
#3  run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:2684
        ptif = <optimized out>
        ptdf = <optimized out>
        start_lock = 0
#4  0x000055b3bd5c4283 in main (argc=<optimized out>, argv=<optimized out>) at src/haproxy.c:3313
        tids = 0x55b3bddbda90
        threads = 0x55b3bddbdab0
        i = <optimized out>
        old_sig = {__val = {0, 139976553102080, 24, 139976570649344, 140733718668488, 140733718668440, 5, 139976549830500, 140733718668488, 94230472160944, 94230472237504, 139976564790982, 335544638, 
            367767529555655936, 5, 139976570649360}}
        blocked_sig = {__val = {18446744067199990583, 18446744073709551615 <repeats 15 times>}}
        err = <optimized out>
        retry = <optimized out>
        limit = {rlim_cur = 4030, rlim_max = 4030}
        errmsg = '\000' <repeats 24 times>, "\t\037q\325N\177\000\000 \"O\037\377\177\000\000\005\000\000\000N\177\000\000A\000\000\000\000\000\000\000p\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\005\000\000\000\061", '\000' <repeats 19 times>, "[\000\000"
        pidfd = <optimized out>

[TimWolla]

This appears to be a duplicate of #29. /cc @lukastribus

[capflam]

Old enough without activity to be closed now.