stubby dies when connectivity is lost about stubby

[saradickinson]

What platform and what version of stubby are you using?

[ArchangeGabriel]

Linux (ArchLinux), 0.1.4.

[saradickinson]

Sorry - and what version of getdns?

[ArchangeGabriel]

1.2.0

[saradickinson]

OK - there are a bunch of stability improvements in the develop branch including a fix for a segfault. You could try building from develop or hang on as we'll be releasing a 1.2.1 version in the next week or 2.

If that doesn't help if you are able to send us the core dump that would be great.

[ArchangeGabriel]

I’ll wait for the next release, I’ve kept the coredump just in case.

[ArchangeGabriel]

Hum… Now I get segfaults all the time, I can’t get it to work at all (I had to revert to network provided DNS instead). I will have to try getdns-git then.

[ArchangeGabriel]

Hum by the way, I have lines in dmesg:

stubby[573]: segfault at 3c ip 00007fe59e002e75 sp 00007fff6cafb940 error 4 in libgetdns.so.6.2.0[7fe59dfe4000+67000]

EDIT: Pressed enter while not focused, sorry about that.

[saradickinson]

The latest release is now available - please try that:
getdns 1.2.1rc, stubby 0.1.5
https://getdnsapi.net/releases/getdns-1-2-1-rc1/

[ArchangeGabriel]

Still coredumping without even answering any request. Here are the coredumps:
https://paste.xinu.at/ohbw6/

[ArchangeGabriel]

So after changing getdns to 1.2.1 final, it seems to be working again. The only commit difference seems to be getdnsapi/getdns@260416a.

I’ll wait until Wednesday to be sure that it works under all my network environments and especially when roaming, keeping you updated.

[ArchangeGabriel]

Should I backport a43be56?

[ArchangeGabriel]

Still getting segfaults. Maybe I should try backporting this commit.

[ArchangeGabriel]

OK, still coredumping anyway. Do you want new coredumps?

[wtoorop]

Hi Br uno,
I am very much interested in the coredumps, thanks for willing to provide them.
Besides the coredump I also need the stubby binary and possible also the libgetdns.so to be able to inspect the backtrace. Did you compile stubby and getdns with debugging symbols? (i.e. with -g in CFLAGS).
Thank you very much for willing to provide all this feedback! This is definitely very valuable for us!

[ArchangeGabriel]

So here are the new coredumps: https://paste.xinu.at/1ux/.

I did not compile with debugging symbols yet (I’ve compiled in release mode for ArchLinux official repos), I’m doing so right now and replacing the one on my system (I’ll upload them here too).

Just in case they could still be usefull, current stubby binary is at https://paste.xinu.at/xdZ/, /usr/lib/libgetdns.so.6.2.1 is at https://paste.xinu.at/R0K/.

[ArchangeGabriel]

OK I’ve found a way to reliably coredump stubby (drill www.hellobank.fr, not sure why this specific domain does this, the same one without www. works OK), so I already have everything required.

stubby binary with debug symbols: https://paste.xinu.at/oTvQ1w/
libgetdns.so.6.2.1 library with debug symbols: https://paste.xinu.at/RRx/
core.stubby.993.21629c238c924720863f9b509cc73910.28639.1510485185000000.lz4: https://paste.xinu.at/2uELr/

[ArchangeGabriel]

(Well it does not coredumps systematically, but at the very least never resolves the domain)

[ArchangeGabriel]

Got a coredump on roaming: I left work where I’m plugged to the network for home where I’m on Wi-Fi, so upon resuming from suspend, I had no network. I suspect stubby died somehow because of this.

Binaries are still the same, new coredump at https://paste.xinu.at/OPeZ/.

[ArchangeGabriel]

So as a matter of fact stubby still segfaults even without roaming, and quite often. Do you want more coredumps? I can provides some more.

[ArchangeGabriel]

This is my current boot:

[ 2514.980376] stubby[28639]: segfault at 44 ip 00007f4de0535075 sp 00007ffe72953180 error 4 in libgetdns.so.6.2.1[7f4de0516000+67000]
[28548.167065] stubby[3314]: segfault at 44 ip 00007fe462b00075 sp 00007ffc2fc1b830 error 4 in libgetdns.so.6.2.1[7fe462ae1000+67000]
[73184.214193] stubby[17732]: segfault at 44 ip 00007f0f90f58075 sp 00007ffd3b6f9520 error 4 in libgetdns.so.6.2.1[7f0f90f39000+67000]
[89433.147344] stubby[28216]: segfault at 44 ip 00007f3ab3934075 sp 00007ffd9574d970 error 4 in libgetdns.so.6.2.1[7f3ab3915000+67000]
[141723.762015] stubby[31407]: segfault at 44 ip 00007f4f634be075 sp 00007ffdba1a3c00 error 4 in libgetdns.so.6.2.1[7f4f6349f000+67000]
[230376.219121] stubby[25578]: segfault at 44 ip 00007f502b32b075 sp 00007fffb2787e00 error 4 in libgetdns.so.6.2.1[7f502b30c000+67000]
[230480.599009] stubby[4538]: segfault at 44 ip 00007f2b8f894075 sp 00007ffd2236f570 error 4 in libgetdns.so.6.2.1[7f2b8f875000+67000]
[260869.761059] stubby[4878]: segfault at 44 ip 00007fd5609fa075 sp 00007ffc664e18a0 error 4 in libgetdns.so.6.2.1[7fd5609db000+67000]

The first two are available in previous posts, the next three I think I’ve deleted them, but I still have the last three if you want.

[wtoorop]

Thanks Bruno, sorry for the late response, last week I was busy at ietf.
I will dive into the core dumps first thing tomorrow!

[wtoorop]

Yes please. The two dumps you provided via paste.xinu.at with debugging symbols, both pointed to the same location in dnssec.c. They were probably caused by failure to fetch all data to do DNSSEC validation in a timely manner. Indeed this code path needs to be reviewed more carefully.

For now, a patch might be this:
0001-BOGUS-answer-because-unable-to-fetch-root-DNSKEY.patch.txt

I am still interested in the last three core dumps since they might be caused by something else...
Thank you for your support!

[ArchangeGabriel]

You’ve got 4 for the same price: https://paste.xinu.at/QPSpFj/ ;)

I’m applying the patch on my local build, will see if things change.

[wtoorop]

Thanks! They all point to line 3167 in dnssec.c:

willem@makaak:~/sink/core-dumps$ gdb stubby core.stubby.993.21629c238c924720863f9b509cc73910.4878.1511186950000000 
GNU gdb (Ubuntu 8.0.1-0ubuntu1) 8.0.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from stubby...done.
[New LWP 4878]

warning: .dynamic section for "/lib64/ld-linux-x86-64.so.2" is not at the expected address (wrong library or version mismatch?)

warning: Could not load shared library symbols for 14 libraries, e.g. /usr/lib/libyaml-0.so.2.
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
Core was generated by `/usr/bin/stubby'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fd5609fa075 in ?? ()
(gdb) set solib-search-path .
Reading symbols from /home/willem/sink/core-dumps/libgetdns.so.6.2.1...done.
(gdb) bt
#0  0x00007fd5609fa075 in check_chain_complete (chain=0x55d18d1e3360) at ./dnssec.c:3167
#1  0x00007fd560a1d0d9 in poll_timeout_cb (event=<optimized out>) at ./extension/poll_eventloop.c:314
#2  poll_eventloop_run_once (loop=loop@entry=0x55d18d034c20, blocking=blocking@entry=1)
    at ./extension/poll_eventloop.c:346
#3  0x00007fd560a1d67d in poll_eventloop_run_once (blocking=1, loop=0x55d18d034c20)
    at ./extension/poll_eventloop.c:491
#4  poll_eventloop_run (loop=0x55d18d034c20) at ./extension/poll_eventloop.c:499
#5  0x000055d18c5bdca2 in main (argc=<optimized out>, argv=<optimized out>) at stubby.c:838

So the patch might work, but since I haven't reproduced the issue myself yet, it might also introduce new issues. At least I now know where to start. Thanks again for that.

[ArchangeGabriel]

OK, so it has been much more stable lately. I’ve just had my first coredump right now, and the message in dmesg is a bit different:

stubby[599]: segfault at 555fc9a8f34a ip 00007efe5f3998f9 sp 00007ffc5dea3940 error 4 in libgetdns.so.6.2.1[7efe5f37a000+67000]

New coredump: https://paste.xinu.at/BSo8z/

[wtoorop]

Yes! Thanks for finding this bug too! I was able to reproduce. We should have found this already were it not for broken error detection in our valgrind unit tests. Your report made another issue apparent too!
I have a fix on the getdns develop branch, or you could apply this patch:
0001-Access-of-freed-memory-in-stub-DNSSEC-cleanup-code.patch.txt

[ArchangeGabriel]

Added to my local build, thanks. Note that I’m stressing stubby far less than before because I’m now using Unbound which then relays to stubby, so that I get the benefits of a validating resolver with caching and the advantages of DNS over TLS on port 443 with reuse of connection thanks to stubby.

[wtoorop]

In that case you might want to turn of validation in Stubby as well?
Or perhaps use the dnssec_return_all_statuses extension to be on the safe side...

[ArchangeGabriel]

You mean I should change dnssec_return_status: GETDNS_EXTENSION_TRUE? What are other options (outside of nothing — would that be enough for Unbound?)?

[wtoorop]

Yes, leaving it out should be enough for Unbound.

[wtoorop]

Actually stubby would always do DNSSEC validation when behind an unbound forwarder, because it sets the dnssec_return_all_statuses extension. PR #53 combined with getdnsapi/getdns#363 will resolve this.

[wtoorop]

Both forementioned PRs have been merged, so issue is resolved.

[ArchangeGabriel]

Yes indeed. It works like a charm now, I’m starting to spreed my Unbound+Stubby configuration around me. We would need more DNS servers answering on port 443, but that’s an other topic.