Cannot build --with-secure-transport from source macOS 14.4 about curl

[bagder]

Works

[bagder]

(You should remove sudo from the two first commands, but that's unrelated to this problem)

[icing]

Thanks for the report. I am still on macOS 14.3.1 where this works in curl 8.6.0 and the current dev version.

Have to upgrade macOS and test what changed there....

[MSBlueVn]

(You should remove sudo from the two first commands, but that's unrelated to this problem)

didn't help.

test what changed there....

from what I gathered, they moved to libressl from openssl, but I am not sure when this change happened.

[bagder]

they moved to libressl from openssl, but I am not sure when this change happened.

Apple's build of curl has used libressl for a while, yes. So if you want to build a similar build as they do, then don't build with secure transport.

Your command line asks to build with secure transport as the TLS backend, which is another TLS library. One that Apple has deprecated since a while back.

If you check the config.log file, and probably search for "error" from the bottom up or something, does that help to explain this error?

[MSBlueVn]

Apple clang version 15.0.0 (clang-1500.1.0.2.5)
Target: arm64-apple-darwin23.4.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
configure:4906: $? = 0
configure:4895: gcc -v >&5
Apple clang version 15.0.0 (clang-1500.1.0.2.5)
Target: arm64-apple-darwin23.4.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
configure:4906: $? = 0
configure:4895: gcc -V >&5
clang: error: argument to '-V' is missing (expected 1 value)
clang: error: no input files
configure:4906: $? = 1
configure:4895: gcc -qversion >&5
clang: error: unknown argument '-qversion'; did you mean '--version'?
clang: error: no input files
configure:4906: $? = 1
configure:4895: gcc -version >&5
clang: error: unknown argument '-version'; did you mean '--version'?
clang: error: no input files
configure:4906: $? = 1
configure:4925: checking whether the C compiler works
configure:4947: gcc    conftest.c  >&5
configure:4951: $? = 0
configure:5001: result: yes
configure:5004: checking for C compiler default output file name
configure:5006: result: a.out
configure:5012: checking for suffix of executables
configure:5019: gcc -o conftest    conftest.c  >&5
configure:5023: $? = 0
configure:5046: result:
configure:5067: checking whether we are cross compiling
configure:5075: gcc -o conftest    conftest.c  >&5
configure:5079: $? = 0
configure:5086: ./conftest
configure:5090: $? = 0
configure:5105: result: no
configure:5110: checking for suffix of object files
configure:5132: gcc -c   conftest.c >&5
configure:5136: $? = 0
configure:5158: result: o
configure:5162: checking whether the compiler supports GNU C
configure:5181: gcc -c   conftest.c >&5
configure:5181: $? = 0
configure:5191: result: yes
configure:5202: checking whether gcc accepts -g
configure:5222: gcc -c -g  conftest.c >&5
configure:5222: $? = 0
configure:5264: result: yes
configure:5284: checking for gcc option to enable C11 features
configure:5299: gcc  -c -g -O2  conftest.c >&5
conftest.c:23:14: warning: a function definition without a prototype is deprecated in all versions of C and is not supported in C2x [-Wdeprecated-non-prototype]
static char *e (p, i)
             ^
1 warning generated.
configure:5299: $? = 0
configure:5317: result: none needed
configure:5433: checking whether gcc understands -c and -o together
configure:5455: gcc -c conftest.c -o conftest2.o
configure:5458: $? = 0
configure:5455: gcc -c conftest.c -o conftest2.o
configure:5458: $? = 0
configure:5470: result: yes
configure:5493: checking how to run the C preprocessor
configure:5519: gcc -E  conftest.c
configure:5519: $? = 0
configure:5534: gcc -E  conftest.c
conftest.c:9:10: fatal error: 'ac_nonexistent.h' file not found
#include <ac_nonexistent.h>
         ^~~~~~~~~~~~~~~~~~
1 error generated.
configure:5534: $? = 1
configure: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME "curl"
| #define PACKAGE_TARNAME "curl"
| #define PACKAGE_VERSION "-"
| #define PACKAGE_STRING "curl -"
| #define PACKAGE_BUGREPORT "a suitable curl mailing list: https://curl.se/mail/"
| #define PACKAGE_URL ""
| /* end confdefs.h.  */
| #include <ac_nonexistent.h>
configure:5561: result: gcc -E
configure:5575: gcc -E  conftest.c
configure:5575: $? = 0
configure:5590: gcc -E  conftest.c
conftest.c:9:10: fatal error: 'ac_nonexistent.h' file not found
#include <ac_nonexistent.h>
         ^~~~~~~~~~~~~~~~~~
1 error generated.
configure:5590: $? = 1

this is the only block I did not see in the output before.

[icing]

Just updated to macOS 14.4 and ./configure --with-secure-transport continues to work. I am on a x86 machine, though.

[bagder]

I too just updated to macOS 14.4 and ./configure --with-secure-transport continues to work. On an arm64 version.

$ ./src/curl -V
curl 8.7.0-DEV (aarch64-apple-darwin23.4.0) libcurl/8.7.0-DEV SecureTransport zlib/1.2.12 libidn2/2.3.7 nghttp2/1.59.0
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS HSTS HTTP2 HTTPS-proxy IDN IPv6 Largefile libz NTLM SSL threadsafe UnixSockets

[MSBlueVn]

ok so what is different on our machines? both arm and intel are stuck on 14.4
Did you maybe have openssl installed? instead of libressl?
What logs can I provide that will help us find the issue?

[bagder]

Did you maybe have openssl installed? instead of libressl?

That does not matter. The configure check in question just checks that you have selected at least one valid TLS library. When you select --with-secure-transport it won't care about libressl, openssl or any other TLS library than secure transport.

What logs can I provide that will help us find the issue?

I think you should double-check that the --with-secure-transport option you use has no funny unicode letter or anything somewhere.

Perhaps try a daily snapshot to see if the problem is rather in your autotools set.

[MSBlueVn]

the code has not changed in the last 6 months. so I am not sure what is the issue.
How would the daily snapshots help? should I check my autotools version then?

[bagder]

If the configure script from a daily snapshot works, it would suggest the issue is with your autotools setup.

[bagder]

If the configure script from a daily snapshot does not work either, we need to continue digging.

[MSBlueVn]

`
configure: Configured to build curl/libcurl:

Host setup: aarch64-apple-darwin23.4.0
Install prefix: /usr/local
Compiler: gcc
CFLAGS: -Qunused-arguments -O2 -Wno-pointer-bool-conversion -Werror=partial-availability
CPPFLAGS:
LDFLAGS: -framework CoreFoundation -framework CoreServices -framework SystemConfiguration -framework CoreFoundation -framework CoreServices -framework Security
LIBS: -lldap -lz

curl version: 8.7.0-20240314
SSL: enabled (Secure Transport)
SSH: no (--with-{libssh,libssh2})
zlib: enabled
brotli: no (--with-brotli)
zstd: no (--with-zstd)
GSS-API: no (--with-gssapi)
GSASL: no (libgsasl not found)
TLS-SRP: no (--enable-tls-srp)
resolver: POSIX threaded
IPv6: enabled
Unix sockets: enabled
IDN: no (--with-{libidn2,winidn})
Build docs: enabled (--disable-docs)
Build libcurl: Shared=yes, Static=yes
Built-in manual: enabled
--libcurl option: enabled (--disable-libcurl-option)
Verbose errors: enabled (--disable-verbose)
Code coverage: disabled
SSPI: no (--enable-sspi)
ca cert bundle: no
ca cert path:
ca fallback:
LDAP: enabled (ancient OpenLDAP)
LDAPS: enabled
RTSP: enabled
RTMP: no (--with-librtmp)
PSL: no (--with-libpsl)
Alt-svc: enabled (--disable-alt-svc)
Headers API: enabled (--disable-headers-api)
HSTS: enabled (--disable-hsts)
HTTP1: enabled (internal)
HTTP2: no (--with-nghttp2)
HTTP3: no (--with-ngtcp2 --with-nghttp3, --with-quiche, --with-openssl-quic, --with-msh3)
ECH: no (--enable-ech)
WebSockets: no (--enable-websockets)
Protocols: DICT FILE FTP FTPS GOPHER GOPHERS HTTP HTTPS IMAP IMAPS IPFS IPNS LDAP LDAPS MQTT POP3 POP3S RTSP SMB SMBS SMTP SMTPS TELNET TFTP
Features: AsynchDNS HSTS HTTPS-proxy IPv6 Largefile NTLM SSL UnixSockets alt-svc libz threadsafe

WARNING: A selected TLS library (Secure-Transport) does not support TLS 1.3!
`

[MSBlueVn]

my autotools versions are:
autoreconf (GNU Autoconf) 2.71
automake (GNU automake) 1.16.5
libtool (GNU libtool) 2.4.7
if that helps

[bagder]

This implies that the problem is not in curl. I don't know why you get this problem, as I believe the tools/versions you show should all work fine.

You can probably bisect your way to finding out exactly what tool/version to blame.

[MSBlueVn]

but the versions I stated were the latest I found online, question is, what is different between my mac, and yours. I do not install these tools, I use xcode to get most of the basic tools, and I only replace curl (and now also libressl, because apple did not patch a vulnerability for months now apparently) to get the best functionality in my scripts.

So, what steps do I take to find the culprit? can you point me in the right direction?

[bagder]

again, I don't think the versions are the problem - I believe there's something strange in your particular install. I build with that exact set on my Linux box. On my m1 mac the only difference is that I use autoconf 2.72 (from homebrew).

The machine that makes the daily snapshots has

• autoconf 2.69
• automake 1.16.3
• libtool 2.4.6

So, what steps do I take to find the culprit? can you point me in the right direction?

If I were to do it, I would investigate the bad generated configure vs the working and how the check for the TLS libraries is so broken that it does not work - because it is a very basic check. Maybe by diffing parts of the working vs non-working scripts.

But it's not an easy nor quick task. Maybe ask the autotools team to see if they recognize this issue or have advice on how to proceed.

[MSBlueVn]

So you think the issue is with autotools?
It's not just one machine, btw, it's all the machines in my org that are getting this script and are stuck on 8.5 even though they try to update.

[bagder]

Clearly the configure script you generate is not working, and at the same time using the exact same set of sources, numerous others generate a fine configure script. So yes, I believe your set of autotools is somehow broken.

[MSBlueVn]

My bad, I forgot the machine I was using to give you outputs was not an average machine. I had installed autotools manually on there, since they do not respond on macOS out of the box.
so this autoreconf -fi command does not exists on mos of my macs, and the ./configure of the snapshots fails with the same error about the TLS

[MSBlueVn]

@bagder can you please reopen this? see my last comment.

[bagder]

I don't understand. In this comment it obviously worked for you. What changed?

[MSBlueVn]

I forgot that the computer I used was not a vanilla mac that is representative of the rest of the macs in my org. It has the autotools installed, among other tools that do not come with macOS or xcode cli.

[bagder]

If you run the configure script we ship on your machine, does that work?

[MSBlueVn]

When you say your machine, do you mean my test machine or any other mac in the company?

[bagder]

1. You first reported an issue with configure when you generated it yourself.
2. You then confirmed that using configure from a daily snapshot worked.
3. You now want to re-open the issue because of unclear reasons.

I want you to explain the problem.

[MSBlueVn]

1. Still valid. 2. Was tested on a test machine that has autotools installed hence should be disregarded. 3. Yes, due to it being closed because of #2. Basically, all the machines in our org do not have autotools, and fail to configure even from snapshots. Is this clear now?

On March 19, 2024 3:17:49 PM GMT+02:00, Daniel Stenberg ***@***.***> wrote: 1. You first reported an issue with configure when you generated it yourself. 2. You then confirmed that using configure from a daily snapshot worked. 3. You now want to re-open the issue because of unclear reasons. I want you to explain the problem. -- Reply to this email directly or view it on GitHub: #13126 (comment) You are receiving this because you authored the thread. Message ID: ***@***.***>

-- -- NOTICE: This email and all attachments are confidential and may contain information that is privileged, confidential, or exempt from disclosure under law. If you are not the intended recipient, delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.

[bagder]

2. Was tested on a test machine that has autotools installed hence should be disregarded.

Autotools being installed or present in a machine does not matter. Autotools is not used by configure.

If the configure scripts works or does not work based on you installing the autotools on the machine or not, then something is seriously wrong.

[MSBlueVn]

Again, only in the snapshots not in the release. And these are out of box macs no brew or other changes where it fails for snapshots as well. We're talking nearly 400 computers, so it's not a small number.

On March 20, 2024 12:56:08 AM GMT+02:00, Daniel Stenberg ***@***.***> wrote: > 2. Was tested on a test machine that has autotools installed hence should be disregarded. Autotools being installed or present in a machine does not matter. Autotools is not used by configure. If the configure scripts works or does not work based on you installing the autotools on the machine or not, then something is seriously wrong. -- Reply to this email directly or view it on GitHub: #13126 (comment) You are receiving this because you authored the thread. Message ID: ***@***.***>

-- -- NOTICE: This email and all attachments are confidential and may contain information that is privileged, confidential, or exempt from disclosure under law. If you are not the intended recipient, delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.

[bagder]

Again, only in the snapshots not in the release.

Are you saying you downloaded 8.6.0 and tried its configure script and it did not work? You did not mention this before.

Still, if the snapshot version works, then if there was a bug in the 8.6.0 version it seems to have been fixed. Has it not?

[MSBlueVn]

I'll try again. Disregard the report of it working. Configure is not working not for the snapshot and not the normal 8.6.0 release. So configure fails the same for everything.

On March 20, 2024 4:51:31 PM GMT+02:00, Daniel Stenberg ***@***.***> wrote: > Again, only in the snapshots not in the release. Are you saying you downloaded 8.6.0 and tried its configure script and it did not work? You did not mention this before. Still, if the snapshot version works, then if there was a bug in the 8.6.0 version it seems to have been fixed. Has it not? -- Reply to this email directly or view it on GitHub: #13126 (comment) You are receiving this because you authored the thread. Message ID: ***@***.***>

-- -- NOTICE: This email and all attachments are confidential and may contain information that is privileged, confidential, or exempt from disclosure under law. If you are not the intended recipient, delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.

[bagder]

So nobody else has this problem and you see it on 400 machines. It still makes me say that the problem is likely to be something in your environment. I don't know what more we can do to help you find it. Any suggestions?

[MSBlueVn]

Can anyone try with a vanilla macos? I suspect you have other binaries that are not built into macos. Aside from that, we tried logs, and it worked to install 8.5 on those 400 machines.

On March 20, 2024 5:09:13 PM GMT+02:00, Daniel Stenberg ***@***.***> wrote: So nobody else has this problem and you see it on 400 machines. It still makes me say that the problem is likely to be something in your environment. I don't know what more we can do to help you find it. Any suggestions? -- Reply to this email directly or view it on GitHub: #13126 (comment) You are receiving this because you authored the thread. Message ID: ***@***.***>

-- -- NOTICE: This email and all attachments are confidential and may contain information that is privileged, confidential, or exempt from disclosure under law. If you are not the intended recipient, delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.

[bagder]

Can anyone try with a vanilla macos? I suspect you have other binaries that are not built into macos.

No other binaries would affect configure's check that a TLS library has been selected so it does not matter. Everyone has other binaries installed if they ever used their machine for something.

I would guess that a few users out there have built curl 8.6.0 successfully on macOS, just like me and @icing. And so have numerous mac CI jobs for all PRs we have merged. I think it is reasonable to suspect that your machines are the outliers here, not all others.

The configure script is a plain shell script. It works and runs on numerous operating systems equally fine. It works virtually everywhere. Except on your machines.

Aside from that, we tried logs, and it worked to install 8.5 on those 400 machines.

I don't know what "tried logs" mean.

The check that a TLS library has been selected was the same in the 8.5.0 configure as it is in the 8.6.0 configure. That check was introduced in 68d89f2 (shipped in 7.77.0) and has mostly remained the same since.

I propose you add debug outputs in the configure script and try to figure out what makes the test fail. It's a mystery to me.

[MSBlueVn]

What I mean is I shared the output from the logs earlier in this thread, and we found nothing.
Can you please elaborate what exactly you mean by add debug outputs? I do not want to miss something.

[bagder]

The failing output shows that the script thinks there was no TLS backend specified even though there was.

The script is a shell script that compares arguments and set a variable if the correct option is used.

I suggest that you debug why that shell script does not do what it is set out to do. Maybe by sprinkling debug outputs in it in suitable places.

[ryandesign]

@MSBlueVn Can you provide the complete config.log file somewhere? The portion you posted above doesn't contain anything that seems relevant.

[MSBlueVn]

Whatever this was, I just had more than 10 users use the same script, and update to 8.7.1 successfully. Might have been a bug in 8.6 that was fixed. Will update later on.