ocsp update over cli crashing haproxy about haproxy

[pa77777]

after last patch 2.8.6 I see some fixes on ocsp in changlogs

    - BUG/MINOR: ssl: Duplicate ocsp update mode when dup'ing ckch
    - MINOR: ssl: Use OCSP_CERTID instead of ckch_store in ckch_store_build_certid
    - BUG/MEDIUM: ocsp: Separate refcount per instance and per store
    - BUG/MINOR: ssl: Reenable ocsp auto-update after an "add ssl crt-list"
    - REGTESTS: ssl: Add OCSP related tests
    - BUG/MEDIUM: ssl: Fix crash when calling "update ssl ocsp-response" when an update is ongoing

but when I send update ssl ocsp-response haproxy crashes totaly

haproxy[2501588]: FATAL: bug condition "ocsp->refcount_instance > 0" matched at src/ssl_ocsp.c:397
haproxy[2501588]:   call trace(15):
haproxy[2501588]:   |       0x4953b6 [0f 0b 0f 1f 84 00 00 00]: main+0x26376
haproxy[2501588]:   |       0x48468b [48 8b 7b 38 e8 cc 80 2e]: ssl_sock_free_cert_key_and_chain_contents+0x16b/0x1c7
haproxy[2501588]:   |       0x485143 [48 8b 7d 00 e8 c4 80 fe]: ckch_store_free+0x53/0x71
haproxy[2501588]:   |       0x488309 [5b 48 8d 75 28 bf 90 cd]: ckch_store_replace+0x119/0x12f
haproxy[2501588]:   |       0x48838f [48 c7 03 00 00 00 00 48]: main+0x1934f
haproxy[2501588]:   |       0x5db763 [85 c0 74 22 49 8b 45 40]: main+0x16c723
haproxy[2501588]:   |       0x68868c [44 8b 05 35 9f 5c 00 45]: task_run_applet+0x13c/0xdd3
haproxy[2501588]:   |       0x63b07a [44 0f b6 44 24 18 48 89]: run_tasks_from_lists+0x38a/0x9fa
haproxy[2501588]:   |       0x63ba76 [29 44 24 1c 8b 4c 24 1c]: process_runnable_tasks+0x386/0x6f7
haproxy[2501588]:   |       0x608237 [83 3d 22 9c 8b 00 01 0f]: run_poll_loop+0x127/0x5a8
haproxy[2501588]:   |       0x6088a7 [48 8b 1d 92 ae 64 00 48]: main+0x199867
haproxy[2501588]:   | 0x7f1968c401ca [64 48 89 04 25 30 06 00]: libpthread:+0x81ca
haproxy[2501588]:   | 0x7f19688ace73 [48 89 c7 b8 3c 00 00 00]: libc:clone+0x43/0x5e
haproxy[2501577]: [NOTICE]   (2501577) : haproxy version is 2.8.6-f6bd011
haproxy[2501577]: [NOTICE]   (2501577) : path to executable is /usr/local/haproxy/sbin/haproxy
haproxy[2501577]: [ALERT]    (2501577) : Current worker (2501588) exited with code 132 (Illegal instruction)
haproxy[2501577]: [ALERT]    (2501577) : exit-on-failure: killing every processes with SIGTERM
haproxy[2501577]: [WARNING]  (2501577) : All workers exited. Exiting... (132)

[Delagen]

I have crash on start with ocsp error when binary was compiled with libressl. Recompiled with openssl, and error was gone

[wtarreau]

Thank you. Can you please issue "where" in gdb so that we know precisely where it crashed ? It mentions line 2087 in log.c but that's a variable declaration so I suspect it's in fact the next line with switch(tmp->type). But if so, it would mean that a list entry found in the list is wrong or possibly corrupted, which is quite confusing. Maybe please also issue "p *tmp" in gdb, in case it can print something about this strange entry.

[wtarreau]

Oh I'm seeing tmp=0 in your back trace. That's definitely not good. I don't see how it can happen but suspect the list itself is null. I'm investigating.

[wlallemand]

I think that's the problem we already discovered with @rlebreton a while ago, there are a few problems when combining ocsp-update and the CLI currently.

[wtarreau]

Apparently it died when exploiting the log-format-sd, is this defined in your config ?

[wtarreau]

I'd be interested in showing the proxy's contents, e.g. "p *sess->fe". I'm seeing how this could cause this if sess->fe->logformat_sd is NULL, except that my understanding of the code is that this proxy is the httpclient's and the httpclient uses the standard alloc_new_proxy() call which performs a LIST_INIT() on that field. So either I'm missing something, or it got overwritten at some point, I don't know.

[wtarreau]

So I verified in gdb and the proxy doesn't have a NULL logformat_sd list. Hence it must have been destroyed later (maybe a free before detaching an element for example).

[wlallemand]

I pushed @rlebreton fixes related to 'set ssl cert' + 'ocsp-update', it's possible that this problem is related.

[wtarreau]

Many thanks for this. As I mentioned in the 3.0-dev4 announce, we had a somewhat similar report but were unable to reproduce it. Probably that your backtrace will be helpful!

[wtarreau]

OK that's useful to know as well, thanks!

[pa77777]

       Message: Process 1867936 (haproxy) of user 992 dumped core.

                Stack trace of thread 1867937:
                #0  0x000000000049504e ssl_sock_free_ocsp (haproxy + 0x9504e)
                #1  0x00000000004841a8 ssl_sock_free_cert_key_and_chain_contents (haproxy + 0x841a8)
                #2  0x0000000000484c23 ckch_store_free (haproxy + 0x84c23)
                #3  0x0000000000487e90 ckch_store_replace (haproxy + 0x87e90)
                #4  0x0000000000487f2f cli_io_handler_commit_cert (haproxy + 0x87f2f)
                #5  0x00000000005d14d7 cli_io_handler (haproxy + 0x1d14d7)
                #6  0x0000000000677df7 task_run_applet (haproxy + 0x277df7)
                #7  0x000000000062e5d5 run_tasks_from_lists (haproxy + 0x22e5d5)
                #8  0x000000000062ef09 process_runnable_tasks (haproxy + 0x22ef09)
                #9  0x00000000005fc757 run_poll_loop (haproxy + 0x1fc757)
                #10 0x00000000005fcdc7 run_thread_poll_loop (haproxy + 0x1fcdc7)
                #11 0x00007f1475a9f802 start_thread (libc.so.6 + 0x9f802)
                #12 0x00007f1475a3f450 __clone3 (libc.so.6 + 0x3f450)

                Stack trace of thread 1867936:
                #0  0x00007f1475b4e84e epoll_wait (libc.so.6 + 0x14e84e)
                #1  0x0000000000473261 _do_poll (haproxy + 0x73261)
                #2  0x00000000005fc718 run_poll_loop (haproxy + 0x1fc718)
                #3  0x00000000005fcdc7 run_thread_poll_loop (haproxy + 0x1fcdc7)
                #4  0x00000000004716df main (haproxy + 0x716df)
                #5  0x00007f1475a3feb0 __libc_start_call_main (libc.so.6 + 0x3feb0)
                #6  0x00007f1475a3ff60 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x3ff60)
                #7  0x0000000000471f15 _start (haproxy + 0x71f15)
                ELF object binary architecture: AMD x86-64
                

(gdb) t a a bt full

Thread 2 (Thread 0x7f1475cc1a40 (LWP 1867936)):
#0  0x00007f1475b4e84e in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
#1  0x0000000000473261 in _do_poll (p=<optimized out>, exp=<optimized out>, wake=<optimized out>) at src/ev_epoll.c:232
        timeout = 132
        status = <optimized out>
        fd = <optimized out>
        count = <optimized out>
        updt_idx = <optimized out>
        wait_time = 132
        old_fd = <optimized out>
#2  0x00000000005fc718 in run_poll_loop () at src/haproxy.c:3045
        next = <optimized out>
        wake = <optimized out>
        __func__ = {<optimized out> <repeats 14 times>}
#3  0x00000000005fcdc7 in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3169
        ptaf = <optimized out>
        ptif = 0xa42750 <per_thread_init_list>
        ptdf = <optimized out>
        ptff = <optimized out>
        init_left = 0
        init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}
--Type <RET> for more, q to quit, c to continue without paging--c
        init_cond = {__data = {{__wseq = 3, __wseq32 = {__low = 3, __high = 0}}, {__g1_start = 1, __g1_start32 = {__low = 1, __high = 0}}, __g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 4, __wrefs = 0, __g_signals = {0, 0}}, __size = "\003\000\000\000\000\000\000\000\001", '\000' <repeats 23 times>, "\004", '\000' <repeats 14 times>, __align = 3}
#4  0x00000000004716df in main (argc=<optimized out>, argv=<optimized out>) at src/haproxy.c:3859
        err = <optimized out>
        retry = <optimized out>
        limit = {rlim_cur = 65535, rlim_max = 65535}
        pidfd = <optimized out>
        intovf = <optimized out>
        msg = <optimized out>

Thread 1 (Thread 0x7f14751ff640 (LWP 1867937)):
#0  ssl_sock_free_ocsp (ocsp=0x7f147424ad00) at src/ssl_ocsp.c:397
No locals.
#1  0x000000000049686a in ssl_sock_free_ocsp (ocsp=<optimized out>) at src/ssl_ocsp.c:391
        __lk_r = <optimized out>
        __set_r = <optimized out>
        __msk_r = <optimized out>
        ret = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
#2  0x00000000004841a8 in ssl_sock_free_cert_key_and_chain_contents (data=0x7f147562de30) at src/ssl_ckch.c:738
        ocsp = <optimized out>
        certid = "0K0\t\006\005+\016\003\002\032\005\000\004\024H\332ɠ\373+\323-O\360\336h\322\365g\267\065\371\263\304\004\024\024.\263\027\267XVˮP\t@\346\037\257\235\213\024\302\306\002\022\004w\030\315\346\035\356Y|\006\214\025\367\355\260\210\345\n", '\000' <repeats 50 times>
        certid_length = 77
#3  0x0000000000484c23 in ckch_store_free (store=0x7f1475645730) at src/ssl_ckch.c:896
        inst = <optimized out>
        inst_s = 0x7f14756456f0
#4  0x0000000000487e90 in ckch_store_replace (old_ckchs=0x7f1475645730, new_ckchs=new_ckchs@entry=0x7f147439a610) at src/ssl_ckch.c:2124
        entry = <optimized out>
        ckchi = <optimized out>
        ckchis = <optimized out>
#5  0x0000000000487f2f in cli_io_handler_commit_cert (appctx=0x7f1474401280) at src/ssl_ckch.c:2211
        ctx = 0x7f1474401330
        sc = <optimized out>
        y = 8
        old_ckchs = <optimized out>
        new_ckchs = <optimized out>
        ckchi = <optimized out>
#6  0x00000000005d14d7 in cli_io_handler (appctx=0x7f1474401280) at src/cli.c:1115
        ctx = <optimized out>
        msg = <optimized out>
        sev = <optimized out>
        sc = 0x7f1474496060
        req = <optimized out>
        res = 0x7f147449d070
        bind_conf = 0x7f14741f6380
        reql = <optimized out>
        len = <optimized out>
        __func__ = {<optimized out> <repeats 15 times>}
#7  0x0000000000677df7 in task_run_applet (t=0x7f1474419320, context=<optimized out>, state=<optimized out>) at src/applet.c:454
        app = <optimized out>
        sc = <optimized out>
        sco = 0x7f1474496000
        rate = <optimized out>
        count = 46
        did_send = 0
        __FUNCTION__ = "task_run_applet"
#8  0x000000000062e5d5 in run_tasks_from_lists (budgets=budgets@entry=0x7f14751f2a80) at src/task.c:634
        process = <optimized out>
        tl_queues = <optimized out>
        t = 0x7f1474419320
        budget_mask = <optimized out>
        profile_entry = 0x0
        done = <optimized out>
        queue = <optimized out>
        state = <optimized out>
        ctx = <optimized out>
        __func__ = {<optimized out> <repeats 21 times>}
#9  0x000000000062ef09 in process_runnable_tasks () at src/task.c:876
        tt = 0xc87000 <ha_thread_ctx+512>
        lrq = <optimized out>
        grq = <optimized out>
        t = <optimized out>
        max = {0, 184, 0, 0}
        max_total = <optimized out>
        tmp_list = <optimized out>
        queue = <optimized out>
        max_processed = <optimized out>
        lpicked = <optimized out>
        gpicked = <optimized out>
        heavy_queued = 1
        budget = <optimized out>
        not_done_yet = <optimized out>
#10 0x00000000005fc757 in run_poll_loop () at src/haproxy.c:2970
        next = <optimized out>
        wake = <optimized out>
        __func__ = {<optimized out> <repeats 14 times>}
#11 0x00000000005fcdc7 in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3169
        ptaf = <optimized out>
        ptif = 0xa42750 <per_thread_init_list>
        ptdf = <optimized out>
        ptff = <optimized out>
        init_left = 0
        init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}
        init_cond = {__data = {{__wseq = 3, __wseq32 = {__low = 3, __high = 0}}, {__g1_start = 1, __g1_start32 = {__low = 1, __high = 0}}, __g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 4, __wrefs = 0, __g_signals = {0, 0}}, __size = "\003\000\000\000\000\000\000\000\001", '\000' <repeats 23 times>, "\004", '\000' <repeats 14 times>, __align = 3}
#12 0x00007f1475a9f802 in start_thread () from /lib64/libc.so.6
No symbol table info available.
#13 0x00007f1475a3f450 in clone3 () from /lib64/libc.so.6
No symbol table info available.

[wtarreau]

Please note that 2.9.6 and 2.8.7 were both emitted a few hours ago to address the regression above. For now they just revert the fix as it used to affect more users than the initial issue (basically all those touching OCSP from the CLI instead of just those with the ocsp-update mechanism).