Cannot run tests in CI about rules_ll

[SpamDoodler]

Personally, I think we actually want to have a custom build Bazel. This would give us even better control over the whole build environment.
Concerning the issues you raised, I think it's ok to prioritize the nix workflow for now, since I don't see a huge drawback with using nix. The few MB for libc++ should be no problem either, since storage is quite inexpensive.
I think it is the spirit of rules_ll to provide the most advanced toolchain possible, so I'm happy to prioritize remote execution over those minor inconveniences.

[jaroeichler]

Statically linking libc++ is fine, those few MB are neglectable in comparison to the cache and nix environment size. We should keep dynamical linking in mind if image size becomes an issue in the future.

Yes, we should aim for a custom build Bazel, this aligns well with the rest of the rules_ll project. Could you go into further detail on how you want to handle the patching and building of Bazel?

[aaronmondal]

@jaroeichler I initially tried just patching the RPATHs with patchelf, but then bazel refuses to operate. Probably for security reasons.

My current plan is:

• Write a nix package that builds bazel via the non-upstream llvm toolchain from nixpkgs and statically link libc++ into it.
• Distribute that Bazel in a way that is compatible with Bazelisk's custom release mechanism outlined here.
• Fetch the binary in our remote execution images via bazelisk.

If things work as i intend, we'd end up with remote execution images that no longer require libstdc++ or any gcc-toolchain parts. If we can reference these custom Bazel binaries in .bazelversion this approach should also be portable to non-nix users as long as the LLVM toolchain parts are statically linked.

As an interesting sidenote we could also try to statically link libmusl into that release to create a fat binary that is independent of the host's glibc version. But let's leave this for later when things actually work 😅

[aaronmondal]

Ok remote execution works, so we could run tests in CI. But that might be really expensive. A single build with near perfect cache reuse (which we basically always have) still needs ~2GB of artifacts to operate (makes sense, building a single target requires the tools from the ll_toolchain, which is roughly that size). At 1 commit per day this is ~60GB just for the main branch. This does not include any PR testing etc. This is also a minimum value. For instance, updating LLVM alone which requires a full cache rebuild and a few revisions might be many times larger than that.

We probably still need a fraction of the resources that others would need for a similar setup, but it's still a big setup. We it might be better off hosting our own remote exec cluster.

• #91