(regression) x86_64-macos -> aarch64-macos cross compilation failing after update to Zig master about mach

[slimsag]

Huh, strangely with (nearly) the same Zig version on my Intel MBP everything is fine:

$ zig build test -Dtarget=aarch64-macos
warning(link): unable to resolve dependency /System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
warning(link): unable to resolve dependency /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
warning: created /Users/slimsag/Desktop/hexops/mach/glfw/zig-cache/o/83c2edab7d4451bb28605a31fc98a00d/test but skipping execution because it is non-native

$ zig build test -Dtarget=aarch64-macos.11
warning(link): unable to resolve dependency /System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
warning(link): unable to resolve dependency /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
warning: created /Users/slimsag/Desktop/hexops/mach/glfw/zig-cache/o/b1636399f5403b7ee98b9329a4d4703f/test but skipping execution because it is non-native

$ zig build test -Dtarget=aarch64-macos.12
warning(link): unable to resolve dependency /System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
warning(link): unable to resolve dependency /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
warning: created /Users/slimsag/Desktop/hexops/mach/glfw/zig-cache/o/ca237d10517e108bb0bc18d25be67e1b/test but skipping execution because it is non-native

[slimsag]

Could be a difference in the XCode versions.

Or possibly the macOS version even? GitHub actions aren't running Monterey yet (but maybe we can opt in? see github/roadmap#240 ) while my system is on Monterey.

[slimsag]

Prior tests were with latest XCode for macOS 12, and CLT 11.3 SDK (surprisingly updating XCode doesn't update CLT)

Same thing with no XCode/CLT, though: it just works.

[slimsag]

Looks like macOS 12 on GitHub Actions won't be supported until at least early 2022, yuck.

actions/runner-images#3649

Also tried what is described in github/roadmap#240 but no dice, if there is a beta program for Monterey we're not in it.

[slimsag]

With set_sysroot = true we don't run into the issue. This implies that Zig is picking up system x86_64 libraries (maybe during dependency resolution?) and trying to link those in by accident, instead of properly resolving the ones in the -iframework and -F parameters first.

[slimsag]

Looks like macos-latest refers to macOS 11 with GitHub actions, and includes XCode 13.1 / CLT 13.0.0

https://github.com/actions/virtual-environments/blob/main/images/macos/macos-11-Readme.md

[slimsag]

What we know so far..

When cross compiling x86_64-macos -> aarch64-macos:

1. On GitHub actions (macOS 11, XCode 13) if we do not set sysroot, we get mismatched cpu architecture: expected Arch.aarch64, found Arch.x86_64
2. On Intel MBP (macOS 12, with both XCode 13 AND no XCode) if we do not set sysroot, linking works fine.

The only fundamental difference here is macOS 11 vs macOS 12.

[slimsag]

OK, finally have a complete explanation.

Context: We do not wish to set sysroot, doing so prevents someone wishing to use e.g. mach-glfw alone from including libraries not found in the mach native SDKs (e.g. Vulkan is not included in our sdk-macos-11.3), we just want to use the sdk-macos-11.3 only for libraries mach-glfw needs by default - things not present on the native system.

Framework .tbd stubs define re-exported libraries:

https://github.com/hexops/sdk-macos-11.3/blob/6d49d06e1280f0f4a207720cf67df44eed41e4db/root/System/Library/Frameworks/Foundation.framework/Foundation.tbd#L24-L25

When we link -framework Foundation without setting sysroot, it finds our sdk-macos-11.3 copy because we've added it to the framework include path.

On macOS 12+, it simply fails to find these reexported libraries:

warning(link): unable to resolve dependency /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
warning(link): unable to resolve dependency /usr/lib/libobjc.A.dylib

This is fine on macOS 12+, because we also explicitly link CoreFoundation and objc separately (not with absolute paths like in those reexported libraries.)

But on macOS 11 and earlier, these libraries in fact exist on the system. /usr/lib/libobjc.A.dylib is a real x86_64 dylib, and that's a mismatch.

[slimsag]

Here is where in the Zig linker code that these re-exported libraries are resolved: https://sourcegraph.com/github.com/ziglang/zig@aa61e03f244a72ea01f05c3ceea7c5fb5aadf1ff/-/blob/src/link/MachO.zig?L1267:4

One option I looked into here: what if we made these paths relative (/usr/lib/libobjc.A.dylib -> usr/lib/libobjc.A.dylib) in our copy only, then included sdk-macos-11.3/root on the lib path. The hope being that usr/lib/libobjc.A.dylib doesn't resolve the system copy, but it would resolve to the sdk-macos-11.3/root/usr/lib/libobjc.A.dylib copy.

The problem with that is that when resolving dependencies, Zig's linker doesn't consider the framework/library search path - it expects absolute paths (rightfully so perhaps?) so unless we could specify fully qualified paths, that's a no-go.

[slimsag]

Another option is to say we don't support cross compilation to Mac unless you're on macOS 12+. But this is icky, particularly because we're relying on some subtle behavior here which could change (that these libs do not exist on the system.)

[slimsag]

Another option to consider: maybe we send a PR to zld to have it respect framework/lib include dirs when resolving dependencies, if an absolute filepath is not found. At first thought, that would seem specific to what Mach is doing but I think Zig would need to solve this too when it starts distributing tbd stubs with the package manager?

[slimsag]

Another valid relatively low-effort option: strip all dependency sections from the .tbd files we distribute.

• Pro: Fixes the issue on macOS 11
• Pro: Doesn't rely on side effects on macOS 12.
• Con: Requires specifying all dependencies manually.

Maybe this option is not too bad - the number of dependencies for system libs seems small in general, there's no harm in specifying all dependencies manually, and if you're using system_sdk.zig you are already specifying libs to link.

[slimsag]

Another valid relatively low-effort option: strip all dependency sections from the .tbd files we distribute.

I plan to go this route. However, this is a non-backwards-compatible change to the sdk-macos repos. Thus, I am first going to ensure we have SDK version pinning and automatic updates for SDKs.