BUG: error querying OCSP responder about testssl.sh

[drwetter]

Thx, @polarathene .

Slept over it . The current supplied binary is statically linked. My understanding would be that the system call gethostbyname is supplied by the binary then and musl libc isn't involved.

A longer while back there were problems on Arch Linux with some DNS related system calls before. I compiled the Linux binaries on a legacy VM. That could be related. In any case that doesn't help to get this off the table immediately.

OTOH This seems hard to fix properly, unless we would check in testssl.sh whether we're in a container and use for this call e.g. /usr/bin/openssl1.1 . Which seems to me like an ugly hack.

Opinions?

[drwetter]

Minor update...

It looks a bit like the static openssl we're providing needs glibc style nss libs. In an alpine 3.17 container:

573e7112c954:/home/testssl# strace /home/testssl/bin/openssl.Linux.x86_64  ocsp  -no_nonce -header Host r3.o.lencr.org -url http://r3.o.lencr.org -issuer /tmp/testssl.S5ZCUF/hostcert_issuer.pem -verify_other /tmp/testssl.S5ZCUF/intermediatecerts.pem -CAfile <(cat etc/*pem) -cert /tmp/testssl.S5ZCUF/host_certificate.pem -text

stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=674, ...}) = 0
open("/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=674, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f95012c1000
read(3, "### /etc/resolv.conf is a symlin"..., 4096) = 674
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0x7f95012c1000, 4096)            = 0
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(3)                                = 0
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(3)                                = 0
open("/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=205, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f95012c1000
read(3, "# musl itself does not support N"..., 4096) = 205
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0x7f95012c1000, 4096)            = 0
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib/x86_64-linux-gnu", 0x7ffff06b2ac0) = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib/x86_64-linux-gnu", 0x7ffff06b2ac0) = -1 ENOENT (No such file or directory)
open("/lib/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib", {st_mode=S_IFDIR|0755, st_size=17, ...}) = 0
open("/usr/lib/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib", {st_mode=S_IFDIR|0755, st_size=258, ...}) = 0
open("/lib/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
write(2, "Error connecting BIO\n", 21Error connecting BIO
)  = 21
write(2, "Error querying OCSP responder\n", 30Error querying OCSP responder
) = 30
write(2, "37206080:error:2006A066:BIO rout"..., 10637206080:error:2006A066:BIO routines:BIO_get_host_ip:bad hostname lookup:b_sock.c:146:host=r3.o.lencr.org
) = 106
exit_group(1)                           = ?
+++ exited with 1 +++
573e7112c954:/home/testssl# cat /etc/nsswitch.conf
# musl itself does not support NSS, however some third-party DNS
# implementations use the nsswitch.conf file to determine what
# policy to follow.
# Editing this file is not recommended.
hosts: files dns
573e7112c954:/home/testssl#

So as one can see it looks at the name service switch, finds files and dns. It tries then to find the files library, then the dns lib. Nothing worked. So it ended up with return code 1.

Alpine doesn´t provide those libraries. The error is the same using 3.16 or edge.

Possible solutions

• provide an extra openssl in the container (3.0 has none, but 3.1dev) and use it for those calls if testssl.sh runs in the container (ugly)
• switch per default to an alpine openssl (definitely not for 3.0, for 3.2rc2 I am due to it's rc status not fond of it)
• provide the libs (nope. won´t work good enough on every platform)
• live with the error (not really good either)

[drwetter]

I tried in the meantime to generate a libnss_dns.a to be included into openssl as a dirty hack but gave up as I couldn't get it to fly and it was ugly (too) in the first place

[drwetter]

In 3.0 this works for me with the recent commit when using within the container

testssl.sh -q -S --phone-out --openssl=/usr/bin/openssl testssl.sh

I hope this workaround is a solution which works for the time the 3.0 branch is still alive.

For 3.1dev: expect a solution this week.

[dcooper16]

Hi @fogs,

Thanks for reporting. My environment is different, but I got the same result. My environment is:

Ubuntu 22.04.2 LTS
Linux 5.19.0-35-generic x86_64
.../testssl.sh/bin/openssl.Linux.x86_64

I can reproduce the problem at the command line:

> ./bin/openssl.Linux.x86_64 ocsp -no_nonce -header Host r3.o.lencr.org -url http://r3.o.lencr.org -issuer /tmp/testssl.L9uzZU/hostcert_issuer.pem -verify_other /tmp/testssl.L9uzZU/intermediatecerts.pem -CAfile etc/Apple.pem -cert /tmp/testssl.L9uzZU/host_certificate.pem -text
Error configuring OpenSSL
31987776:error:25066067:DSO support routines:DLFCN_LOAD:could not load the shared library:dso_dlfcn.c:187:filename(libproviders.so): libproviders.so: cannot open shared object file: No such file or directory
31987776:error:25070067:DSO support routines:DSO_load:could not load the shared library:dso_lib.c:233:
31987776:error:0E07506E:configuration file routines:MODULE_LOAD_DSO:error loading dso:conf_mod.c:271:module=providers, path=providers
31987776:error:0E076071:configuration file routines:MODULE_RUN:unknown module name:conf_mod.c:212:module=providers

However, everything works if I use the built-in OpenSSL:

/usr/bin/openssl ocsp -no_nonce -header Host=r3.o.lencr.org -url http://r3.o.lencr.org -issuer /tmp/testssl.SOO9Of/hostcert_issuer.pem -verify_other /tmp/testssl.SOO9Of/intermediatecerts.pem -CAfile etc/Apple.pem -cert /tmp/testssl.SOO9Of/host_certificate.pem -text

I found something at acmesh-official/acme.sh#4048 that suggests the issue is with the openssl.cnf that comes with the distribution.

[dcooper16]

It seems that my testing was incorrect, as my environment at the command line was incorrect. I can't reliably reproduce things at the moment. However, I believe that if my environment is set up the same as as testssl.sh sets it up the result is:

> .../testssl.sh/bin/openssl.Linux.x86_64 ocsp -no_nonce -header Host r3.o.lencr.org -url http://r3.o.lencr.org -issuer /tmp/testssl.4EWHlq/hostcert_issuer.pem -verify_other /tmp/testssl.4EWHlq/intermediatecerts.pem -CAfile etc/Apple.pem -cert /tmp/testssl.4EWHlq/host_certificate.pem -text
OCSP Request Data:
    Version: 1 (0x0)
    Requestor List:
        Certificate ID:
          Hash Algorithm: sha1
          Issuer Name Hash: 48DAC9A0FB2BD32D4FF0DE68D2F567B735F9B3C4
          Issuer Key Hash: 142EB317B75856CBAE500940E61FAF9D8B14C2C6
          Serial Number: 04913746C0218A9C801267BA49AC6A12362C
Segmentation fault (core dumped)

[drwetter]

Hi David,

./bin/openssl.Linux.x86_64 ocsp -no_nonce -header Host r3.o.lencr.org -url http://r3.o.lencr.org -issuer /tmp/testssl.L9uzZU/hostcert_issuer.pem -verify_other /tmp/testssl.L9uzZU/intermediatecerts.pem -CAfile etc/Apple.pem -cert /tmp/testssl.L9uzZU/host_certificate.pem -text
Error configuring OpenSSL
31987776:error:25066067:DSO support routines:DLFCN_LOAD:could not load the shared library:dso_dlfcn.c:187:filename(libproviders.so): libproviders.so: cannot open shared object file: No such file or directory
31987776:error:25070067:DSO support routines:DSO_load:could not load the shared library:dso_lib.c:233:
31987776:error:0E07506E:configuration file routines:MODULE_LOAD_DSO:error loading dso:conf_mod.c:271:module=providers, path=providers
31987776:error:0E076071:configuration file routines:MOD

OPENSSL_CONF="" is missing, otherwise the system provided openssl.cnf will be used

[drwetter]

... but I believe you found the problem, David.

For 3.1dev where we have openssl in the container it works using docker run --rm -ti drwetter/testssl.sh --phone-out -S testssl.sh , whereas w/o openssl it does not.

For 3.0 there's no openssl in the container. I added it later as we needed that for STARTTLS injection.

So it should probably suffice adding OPENSSL_CONF="" in front of that call you used when the supplied openssl is used -- outside the container $OPENSSL might not be equal to $HOME/bin/openssl.Linux.x86_64

[drwetter]

not quite:

prompt> docker run --rm -ti drwetter/testssl.sh:3.0  --debug=2  --phone-out -S testssl.sh
[..]
OCSP URI                     http://r3.o.lencr.org, error querying OCSP responder
WARNING: can't open config file:
Error connecting BIO
Error querying OCSP responder
37165120:error:2006A066:BIO routines:BIO_get_host_ip:bad hostname lookup:b_sock.c:146:host=r3.o.lencr.org
OCSP Request Data:
Version: 1 (0x0)
Requestor List:
Certificate ID:
Hash Algorithm: sha1
Issuer Name Hash: 48DAC9A0FB2BD32D4FF0DE68D2F567B735F9B3C4
Issuer Key Hash: 142EB317B75856CBAE500940E61FAF9D8B14C2C6
Serial Number: 04913746C0218A9C801267BA49AC6A12362C

OCSP stapling                offered, not revoked
[..]

prompt> testssl.sh --debug=2 --phone-out  -S testssl.sh
[..]
# of certificates provided   3
In pwnedkeys.com DB          not in database
Certificate Revocation List  --
OCSP URI                     http://r3.o.lencr.org, not revoked
OCSP stapling                offered, not revoked
OCSP must staple extension   --
DNS CAA RR (experimental)    0 issue "letsencrypt.org" 0 issuewild ";"
[..]

[drwetter]

hmm... 🤔

b_sock.c:146 is referring to BIO_gethostbyname which calls gethostbyname which comes from the libc. Not sure what a openssl.cnf can fix there. Educated guess: the musl libc is to blame. It works outside the container

[drwetter]

@polarathene : what's your take on this?

[polarathene]

@polarathene : what's your take on this?

Not really my area of expertise, but as you mention it's probably due to musl.

• Has this feature ever worked on Alpine images before that don't provide an OpenSSL binary from apk?
• Is the binary you provide completely portable, or does it have external dependencies that link to glibc? (the extent of my knowledge of this sort of problem 😛 )

I am assuming it's never worked. I know that this can be a common problem of running binaries built with glibc (linked?) and copied over into an Alpine image to run.

I found something at acmesh-official/acme.sh#4048 that suggests the issue is with the openssl.cnf that comes with the distribution.

Does not affect apk installed OpenSSL packages, nor any change running without those and just rm /etc/ssl/openssl.cnf

---

Probably not surprising output given maintainer comments above 😅

$ docker run --rm -it --entrypoint ash --user "0:0" drwetter/testssl.sh:3.0
$ testssl.sh --debug=2 --phone-out -S testssl.sh

No engine or GOST support via engine with your /usr/bin/openssl1.1

###########################################################
    testssl.sh       3.0.8 from https://testssl.sh/
[..]
 Using "OpenSSL 1.0.2-bad (1.0.2k-dev)" [~179 ciphers]
 on 69c122d608cc:/home/testssl/bin/openssl.Linux.x86_64
 (built: "Sep  1 14:03:44 2022", platform: "linux-x86_64")
[..]
 OCSP URI                     http://r3.o.lencr.org, error querying OCSP responder
WARNING: can't open config file: 
Error connecting BIO
Error querying OCSP responder
43751488:error:2006A066:BIO routines:BIO_get_host_ip:bad hostname lookup:b_sock.c:146:host=r3.o.lencr.org
OCSP Request Data:
    Version: 1 (0x0)
    Requestor List:
        Certificate ID:
          Hash Algorithm: sha1
          Issuer Name Hash: 48DAC9A0FB2BD32D4FF0DE68D2F567B735F9B3C4
          Issuer Key Hash: 142EB317B75856CBAE500940E61FAF9D8B14C2C6
          Serial Number: 04913746C0218A9C801267BA49AC6A12362C

 OCSP stapling                offered, not revoked
 OCSP must staple extension   --
[..]

vs with OpenSSL 3.0 package

$ docker run --rm -it --entrypoint ash --user "0:0" drwetter/testssl.sh:3.0
$ apk add openssl
$ testssl.sh --debug=2 --phone-out -S --openssl /usr/bin/openssl testssl.sh

No engine or GOST support via engine with your /usr/bin/openssl1.1

###########################################################
    testssl.sh       3.0.8 from https://testssl.sh/
[..]
 Using "OpenSSL 3.0.8 7 Feb 2023 (Library: OpenSSL 3.0.8 7 Feb 2023)" [~76 ciphers]
 on 69c122d608cc:/usr/bin/openssl
 (built: "Feb  7 16:19:55 2023", platform: "linux-x86_64")
[..]
 OCSP URI                     http://r3.o.lencr.org, not revoked
 OCSP stapling                offered, not revoked
 OCSP must staple extension   --
[..]

vs with OpenSSL 1.1.x package:

$ docker run --rm -it --entrypoint ash --user "0:0" drwetter/testssl.sh:3.0
$ apk add openssl1.1-compat
$ testssl.sh --debug=2 --phone-out -S --openssl /usr/bin/openssl1.1 testssl.sh

No engine or GOST support via engine with your /usr/bin/openssl1.1

###########################################################
    testssl.sh       3.0.8 from https://testssl.sh/
[..]
 Using "OpenSSL 1.1.1t  7 Feb 2023" [~76 ciphers]
 on 69c122d608cc:/usr/bin/openssl1.1
 (built: "Feb  7 16:22:38 2023", platform: "linux-x86_64")
[..]
 OCSP URI                     http://r3.o.lencr.org, not revoked
 OCSP stapling                offered, not revoked
 OCSP must staple extension   --
[..]

[drwetter]

If anybody wants to check whether it's the legacy platform which is to blame: https://github.com/drwetter/openssl-1.0.2.bad is your friend

[polarathene]

unless we would check in testssl.sh whether we're in a container and use for this call e.g. /usr/bin/openssl1.1 . Which seems to me like an ugly hack.

Either native openssl binaries work fine AFAIK. I don't think it's specific to a container in that sense, you should have the same issue on an Alpine host?

It looks a bit like the static openssl we're providing needs glibc style nss libs.

This is what came to mind, but I was of the understanding if the binary was built with static linking that it did not matter if glibc vs musl on the system. Alpine has some history of DNS issues apparently, and perhaps is related to your findings? 🤷‍♂️

Alpine doesn´t provide those libraries.

Same question, if the openssl binary you supply is meant to be static why would this matter?

I have had to mess with NSS in the past on a linux system (glibc though) to configure it to properly handle mdns lookups with some Avahi setup I had and remember I sunk a bit of time troubleshooting that.

If the image just needs to install something extra or tweak config, that could be done, but AFAIK it's capable of handling DNS just fine, it's only an issue with the foreign binary you're providing it?

---

provide an extra openssl in the container (3.0 has none, but 3.1dev) and use it for those calls if testssl.sh runs in the container (ugly)

You can update the 3.0 Dockerfile to provide openssl, but IIRC you still need to specify --openssl /path/to/binary to testssl.sh in this case, so in addition to that, add help / docs (or if you can detect the failure and inline the advice output perhaps) that state the feature may require --openssl.

If you want it as a container only behaviour, then there are a few options that can be done there, some that don't require making any cahnges to testssl.sh script, but I'm pretty sure this would still be an issue running 3.0 on host with musl instead of glibc? Not just in a container.

switch per default to an alpine openssl (definitely not for 3.0, for 3.2rc2 I am due to it's rc status not fond of it)

I doubt anyone is extending the image, you could consider switching the base image? Wasn't that already done in the past for 3.0 or 3.1 before changing to Alpine?

Alternatively, you can publish both base images, this is often done with a tag like 3.0-debian. Then you can mention in DockerHub or anywhere else relevant that Debian is available as an alternative.

Alpine provides the size benefit, but often at the expense of compatibility and performance (both in runtime speed and memory usage AFAIK). Debian is often a base image offered as an alternative (but it lags behind in package versions and much slower release cadence).

[polarathene]

I think I got my answer about the static build not being able to prevent this 😅

• On GNU/Linux libc isn't an interface to the NSS service, libc is the NSS service.
• libc provides the standard POSIX sockets APIs, which include DNS functions such as gethostbyname() and getaddrinfo()
• DNS queries that use glibc (so everything basically) parse /etc/nsswitch.conf and follow the path of NSS modules which can do whatever they want to produce a name.
• dig is not affected by alpine’s decision here because dig does not use gethostbyname().

Source

Common advice is to build with static linking to musl instead of glibc apparently, but for NSS, this has some implications about expected behaviour (for most uses of testssl.sh probably not an issue though?).

So technically I guess one other option is to go that route and distribute that binary.

---

I'm pretty sure the Docker ENTRYPOINT can just run testssl.sh --openssl /usr/bin/openssl if you don't need to default to static build in the container 🤷‍♂️ (I totally understand why you'd rather avoid that implicit opt-in)

[drwetter]

So technically I guess one other option is to go that route and distribute that binary.

At some point of time actually it's the best to get rid of that binary. I feel no so tempted to recompile with musl libc, bc of that and also bc I sense there will be one or the other obstacle when compiling.

[drwetter]

Either native openssl binaries work fine AFAIK. I don't think it's specific to a container in that sense, you should have the same issue on an Alpine host?

probably. I have none though :-)

This is what came to mind, but I was of the understanding if the binary was built with static linking that it did not matter if glibc vs musl on the system

The NSS calls come probably from the static glibc but it expects a Gnu libc system . The static compile process doesn´t include those external libs. In my limited experience with libs it seem just normal that some system calls are coming from a lib and end up in another lib. And udring compile time one doesn´t know what the system is configure to use for DNS lookups.

I have had to mess with NSS in the past on a linux system (glibc though) to configure it to properly handle mdns lookups with some Avahi setup I had and remember I sunk a bit of time troubleshooting that.

See ;-) On my desktop system I have a couple of nss_mdns libs . Openssl could probably also use that when /etc/nsswitch.conf contains that. /usr/lib64/libnss_wins.so.2 on my system btw is provided by Samba. Oh, and btw, see nss_ldap(5). Got the picture?

you could consider switching the base image? Wasn't that already done in the past for 3.0 or 3.1 before changing to Alpine?

a) for 3.0: probably not . For 3.1dev: not sure. Just pulled the bullseye-slim image. It´s 80MB. Was just trying to figure out how you ended up at 135 MB. But using your Dockerfile in #2299 I got Error response from daemon: dockerfile parse error line 10: unknown instruction: APT-GET

b) cbe38cc, for size reasons

[drwetter]

I´ll sleep over it, see #2299 (comment)

[polarathene]

Just pulled the bullseye-slim image. It´s 80MB. Was just trying to figure out how you ended up at 135 MB.

You seem to have figured it out in the referenced issue, but just to clarify that is the base image weight 80MB, extra size comes from all the extra additions in the Dockerfile after that.

---

But using your Dockerfile in #2299 I got Error response from daemon: dockerfile parse error line 10: unknown instruction: APT-GET

Possibly due to your version of Docker?

If you have the release from earlier this year (v23) it should build it without errors like that I think. AFAIK it's due to the HereDoc syntax which I think may only be supported with BuildKit (should now be default in v23, but was opt-in for prior releases requiring an ENV DOCKER_BUILDKIT=1 or docker buildx).

---

b) cbe38cc, for size reasons

Yeah, that's usually the motivator for using Alpine.

Until it becomes more trouble than it's worth due to compatibility issues like witnessed here, and potential degraded performance and higher memory usage / leaks (different allocator), and the DNS issues others cite.

Eventually if you experience those problems, you can try work around them or consider if the 100MB or so savings is actually a problem.

IIRC, the size in transit to pull is actually lower due to compression (DockerHub tags page usually reports the compressed size). Debian slim image has 80MB, but is only 30MB compressed. Often the concern for size is about network transfer more than disk, plus if any other images use that same base image that weight is no longer relevant.

[drwetter]

Update: I missed yesterday night to commit also the link from openssl1.1 to openssl. --> see PR #2342 merged.