Rust 1.66.0 not building about pantry.core

[jhheider]

1.66.1 is the same: https://github.com/teaxyz/pantry.core/actions/runs/3888889687/jobs/6636907235

[jhheider]

I believe this is now impacting deno.land builds, so we need to figure this out. https://github.com/teaxyz/pantry.core/actions/runs/3914013979/jobs/6690589969

[jhheider]

@jonchang any ideas here? the failure states are basically all the same: it can't find the atomics it wants for linux-aarch64.

= note: ld.lld: error: undefined reference due to --no-allow-shlib-undefined: __aarch64_ldadd8_relax
          >>> referenced by /root/.tea/rust-lang.org/src/v1.66.1/build/aarch64-unknown-linux-gnu/stage0-rustc/aarch64-unknown-linux-gnu/release/deps/librustc_driver-c4bd77c3b6122f7a.so

          ld.lld: error: undefined reference due to --no-allow-shlib-undefined: __aarch64_ldadd8_rel
          >>> referenced by /root/.tea/rust-lang.org/src/v1.66.1/build/aarch64-unknown-linux-gnu/stage0-rustc/aarch64-unknown-linux-gnu/release/deps/librustc_driver-c4bd77c3b6122f7a.so

          ld.lld: error: undefined reference due to --no-allow-shlib-undefined: __aarch64_cas1_acq
          >>> referenced by /root/.tea/rust-lang.org/src/v1.66.1/build/aarch64-unknown-linux-gnu/stage0-rustc/aarch64-unknown-linux-gnu/release/deps/librustc_driver-c4bd77c3b6122f7a.so

          ld.lld: error: undefined reference due to --no-allow-shlib-undefined: __aarch64_cas1_rel
          >>> referenced by /root/.tea/rust-lang.org/src/v1.66.1/build/aarch64-unknown-linux-gnu/stage0-rustc/aarch64-unknown-linux-gnu/release/deps/librustc_driver-c4bd77c3b6122f7a.so

I've tried llvm 13, 14, and 15, RUSTFLAGS=-Ctarget-feature=-outline-atomics, CFLAGS=-Wl,-moutline-atomics, including gnu.org/gcc~12 (which has libatomic under lib64/), passing LDFLAGS=-L(gnu.org/gcc/lib64 and /usr/lib/aarch64-unknown-linux-gnu, which also has a copy), and this error is sticky.

Only blocking linux-aarch64. I'm tempted to just pull the pre-built bins for that arch to unblock this ticket.

[jhheider]

Most recent wild failure: https://github.com/teaxyz/pantry.core/actions/runs/3888889687/jobs/6636907235

[mxcl]

Only blocking linux-aarch64. I'm tempted to just pull the pre-built bins for that arch to unblock this ticket.

I'm fine with using relocatable pre-built binaries (we're doing it for bun) though really before doing that we should implement it properly and SHA check and maybe even insist on the binaries being signed.

The other option is to implement support for some platforms not being built being ok. I dunno how to signal to CI/CD that we are ok with this tho.

[jonchang]

I was doing some research into this two weeks ago and I believe the failure is actually due to how we're building LLVM (or possibly just rust) on aarch64. I believe that we actually need to set -mno-outline-atomics due to some instruction set changes in different versions of ARM, but I haven't dug too deeply into that. Alternatively, it could be that we haven't enabled some LLVM feature that builds in these intrinsics.

[jonchang]

It now occurs to me (after searching) that tea doesn't appear to pass any -march= flags (or restrict them) when bottling. So it's possible that this is an instance where rust is trying to build something for a newer cpu than can actually run on the CI hardware.

[jhheider]

That'd be an interesting regression. The hardware is an Apple M1max, so there's not a lot newer in the arm space.

[mxcl]

It now occurs to me (after searching) that tea doesn't appear to pass any -march= flags (or restrict them) when bottling

Great point.

With brew initially this stuff was added in order to pursue MAXIMUM PERFORMANCE. CPUs were less capable in 2009. Over time with bottling the parameters were changed to match oldest supported arches.

I'm no bin tools expert, but I don't think any of them apply optimizations for specific CPU types by default, even if -O3 were applied (which we don’t do either).

Still rust may be doing something for sure.

And either way, we should probably do a superenv type thing and ensure that cc etc. only get called with performance flags that make sense for our bottles.

wistful thinking: would be wonderful to do bottles per CPU.

---

Anyway I don't think this can be the issue here since we only have one self-hosted runner. The same machine is building and running the tests.

[jhheider]

https://bugzilla.redhat.com/show_bug.cgi?id=1830472 seems like the most promising path of enquiry, especially after noting that our build scripts build rust-1.66.1 just fine on ubuntu-latest but not debian:buster-slim. I don't seem to be able to shoe-horn in the libgcc.{a,so} identities from gnu.org/gcc~12. @jonchang thoughts? Rabbit hole? Any tricks?

[jhheider]

Looks like outline-atomics were added in gcc-9.4 and debian buster ships with 8.3. The redhat bug report suggests that some part of the system linking is provided by libgcc, so I'm still trying to see if we can get ours to take precedence, since we provide 12.

[jhheider]

cross-rs/cross#1067 (comment)

I suppose one option would be bumping the version of debian we're building aarch64 with; risks are that we're going in the wrong direction for glibc compatibility.

[jhheider]

https://users.rust-lang.org/t/build-rustc-from-source-with-libgcc-9-3-1/88051

[mxcl]

I suppose one option would be bumping the version of debian we're building aarch64 with; risks are that we're going in the wrong direction for glibc compatibility.

Depends on the state of Linux on aarhc64. I believe eg. Raspberry Pi is aarch64 and if so that would be something to look at, what's its glibc requirements?

[jhheider]

Well, there's definitely arm builds of glibc going way back. The issue here is with a specific feature (out-of-line atomics) that rust apparently requires as of 1.66.0 which requires a newer libgcc. I'd hate to lose backwards compatibility for a single package, but I'd also hate to not get rust updates.

[magnusviri]

Can tea eat its own dog food? That is, is it versioning libgcc and can it use different versions for different packages?

[jhheider]

Working on that, actually. It's quite not-trivial, of course, but we're here for the challenge.

[jhheider]

After all that, rust-lang.org=1.67.0 built just fine. I'll try another rebuild of 1.66.0/1, just in case the fix was external to their codebase, but it seems like they've probably updated their llvm bootstrap internals in a way that fixed this for us.

[mxcl]

#pkglife