urbit / vere

An implementation of the Urbit runtime

Urbit Runtime

Urbit is a personal server stack built from scratch. This repository contains Urbit's runtime environment, the lowest layer of the Urbit stack, which includes the Nock virtual machine, I/O drivers, event log, and snapshotting system.

Getting Started

For basic Urbit usage instructions, head over to urbit.org. For a high-level overview of the salient aspects of Urbit's architecture, visit developers.urbit.org. You might also be interested in joining the urbit-dev mailing list.

Packages

Urbit's runtime is broken down into a few separate layers, each of which is defined in its own package:

• pkg/c3: A set of basic utilities for writing Urbit's style of C.
• pkg/ent: A cross-platform wrapper for getentropy(2).
• pkg/ur: An implementation of jam and cue, Urbit's bitwise noun serialization and deserialization algorithms, respectively.
• pkg/noun: The Nock virtual machine and snapshotting system.
• pkg/vere: The I/O drivers, event log, and main event loop.

Build

See INSTALL.md.

Contributing

Contributions of any form are more than welcome. Please take a look at our contributing guidelines for details on our git practices, coding styles, how we manage issues, and so on.

vere's Issues

Build and release Docker images

Use bazelbuild/rules_docker to build a linux/amd64 Docker image automatically when new releases are tagged. Upload (release) images with a GitHub Actions workflow.

Reference
https://github.com/urbit/urbit/blob/master/.github/workflows/release-docker.yml

Resolve URL discrepancies in binary upgrade system and CI

The binary upgrade system and CI don't currently use the exact same URLs. The binary upgrade system expects URLs of the form {ver_hos_c}/{pace}/{version_number}/vere-v{version_number}-{os}-{cpu}, but CI uploads binaries to {ver_hos_c}/{pace}/v{version_number}/vere-v{version_number}-{os}-{cpu} (note the leading "v" before the first {version_number}). What's worse, CI uses a shortened commit SHA in place of the version number for the edge train. To fix this, the binary upgrade system should add a leading "v" before the first {version_number} in its URL, and CI should upload using an actual version number to the edge train, not a commit SHA.

Use `git` tags exclusively for setting the release version

Describe the bug
Currently we use a combination of git tags and the pkg/vere/VERSION file to denote release version numbers. To simplify the workflow, developers should only have to tag a new release in order to set the URBIT_VERSION macro and upload binaries and Docker images with the correct version labels.

Additional context
See:

• pkg/vere/VERSION
• shared.yml

Running the binary against a non-existent pier name creates a new comet instead of complaining

Describe the bug
Running urbit thispierdoesntexist starts a new comet, instead of complaining and expecting the -c flag.

To Reproduce

1. run urbit thispierdoesntexist (substitute a different pier name if you happen to have a pier with that name :P )
2. observe that the runtime happily boots a new comet

Expected behaviour
If the pier isn't found and -c (create a new pier) isn't specified, the runtime should throw an error (possibly suggesting the use of -c to the user).

System:

• OS: linux64 but also observed on Mac
• Vere and Urbit OS versions: occurs on all recent vere (from 1.0 to 1.5)

Additional context
I've seen a few users run into confusion when they are in the wrong directory, or have moved their pier, and run an older command in an attempt to start up a ship they think should be there. A particular example (which prompted this report) was from a user who migrated their ship into Port:

• normally, run ./urbit sampel-palnet to start up
• import ship into Port, encounter errors and give up
  • unbeknownst to the user, Port defaults to moving their pier into its own directory to avoid double-booting
• try to go back to using the regular command - but ./urbit sampel-palnet now boots a comet in the same location because the folder had been removed from its previous spot

Similar things have happened in the absence of Port when e.g. the user installs the binaries and their pier into ~/urbit and doesn't realise they need to be in that same directory to start back up again later - suddenly ./urbit sampel-palnet complains about urbit being a directory. They do some digging and find that the binary they want is inside that directory, so they run ./urbit/urbit sampel-palnet, but because they're a directory up from where the pier is, they get a new comet at ~/sampel-palnet instead of booting ~/urbit/sampel-palnet as they were intending. (I don't think it's sensible to go as far as searching for pier-shaped directories and asking "did you mean to use <this other path to a pier>?" but throwing an error would give the user a clear indication that something was amiss with what they were asking the binary to do).

As it stands these sorts of cases appear to the user like "my planet suddenly turned into a comet and I don't know why".

Spin up UF-owned `macos-aarch64` and `linux-aarch64` runners for GitHub Actions

We currently use @mcevoypeter's macos-aarch64 and linux-aarch64 boxes as self-hosted runners for builds, but we should use our own. We can use MacStadium for the former and BuildJet for the latter platforms.

Temporarily disable fake-ship tests

The fake ship tests take ~20 minutes to run, which interferes with the ability to rapidly iterate on CI-related changes that don't affect the functionality of the runtime.

Reduce CI log noise/verbosity on `build static binary` and `run tests` jobs

See this recent job for reference.

Fix musl libc toolchain cache access

We install the musl libc toolchains (for linux-x86_64 and linux-arm64) in /usr/local, which causes an issue with the GitHub cache action because writing to /usr/local requires sudo. To get around this, we should install the musl libc toolchains in ~.

Build with `bazel` on `linux-arm64`

Build the urbit binary on linux-arm64. This will build off the work done in #2. The steps required:

• Tweak third-party dependency bazel build rules in bazel/third_party/ to build on linux-arm64 platforms.
• Tweak the bazel build rules for all packages in pkg/ to build on linux-arm64 platforms.

Deleting memory.bin with each checkpoint can cause write amplification

On flash storage filesystems that don’t have aggressive defaults regarding TRIM (i.e. specifying the discard mount option in /etc/fstab) the frequency with which Urbit writes checkpoints can pretty quickly fill up the device with undiscarded unused blocks, causing write amplification, reduced write speed, and increased wear.

Memory.bin is created, then deleted, every two minutes. The size of memory.bin varies, but it doesn’t have to be particularly large: if each memory.bin averages 100MB then that's 144 GB in 24 hours. The most common behavior by Linux distributions seems to be running fstrim on a weekly basis, so any device less than a terabyte in size risks being filled. And then there’s something like a Raspberry Pi, which typically has much less storage and by default doesn’t run TRIM at all…

Decreasing checkpoint frequency would help a lot, but I think the ideal fix would be to avoid deleting memory.bin and overwriting the file in place instead.

@joemfb

Build with `bazel` on `darwin-x86_64`

Build the urbit binary on darwin-x86_64 (i.e. x86_64 machines running macOS). This will build off the work done in #2. The steps required:

• Tweak the third-party dependency bazel build rules in bazel/third_party/ to build on darwin-x86_64 platforms.
• Tweak the bazel build rules for all packages in pkg/ to build on darwin-x86_64 platforms.

Upload the latest binary version number to GCP

The latest binary version number needs to be uploaded to {url_base}/{pace}/v{version_number}/last.

Fakeship tests fail on macOS builds

https://github.com/urbit/vere/actions/runs/3715632920/jobs/6301027661

Same errors on macos-x86_64 and macos-arm64.

Set up `bazel`-based continuous integration flow

Create a new GitHub action .github/workflows/bazel-build.yml that builds all targets via bazel build ... and runs all test targets via bazel test ... for all supported platforms. For the list of platforms we support, see #1. The existing nix-based continuous integration flow in .github/workflows/build.yml may serve as a useful reference point.

This feature requires that #1 is complete.

Run file system IO driver in a separate process

The file system IO driver should be rewritten (in Rust) to run in a separate process to improve security by sandboxing one of Urbit's interfaces to the outside world and to enable future scalability. The steps required:

1. Re-implement the following functions (in pkg/vere/io/unix.c) that release file system effects in Rust in the IO drivers, rendering the existing driver implementation redundant:
   a. u3_unix_ef_dirk(): commit a new mount point.
   b. u3_unix_ef_ergo(): update the file system
   c. u3_unix_ef_ogre(): delete an existing mount point.
   d. u3_unix_ef_hill(): scan for new mount points.
2. Replace the now-redundant file system driver C implementation with an implementation that asynchronously write a request to the new file system process's stdin and reads any subsequent response asynchronously from the file system proccess's stdout. Note that this implementation is nearly identical to the same implementation required for the HTTP client driver in #11; the two implementations should be unified into a single implementation.

Both Scaling Urbit's Runtime and Scalable IO provide additional relevant context for this work.

Select proper system include paths on macOS

The fix for #23 introduced in #24 does not work on a macOS install that has the Xcode application installed because Bazel detects a dependency on /Applications/Xcode.app/Contents/Developer instead of on /Library/Developer/CommandLineTools. The proper sys_includes paths for macOS really depend on the output of xcode-select --print-path, but Bazel makes it difficult to consume version information from arbitrary binaries like xcode-select. A possible work-around is to force Bazel to ignore /Applications/Xcode.app using the --sandbox_block_path flag and therefore force it to use /Library/Developer/CommandLineTools instead.

Correct faulty default pill path

The runtime attempts to download the default pill from https://bootstrap.urbit.org/props/{version}/brass.pill, but the default pill actually lives at https://bootstrap.urbit.org/urbit-v{version}.pill.

Self-host M1 macOS GitHub Actions runner

To test the macos-arm64 build in CI, we need an M1 macOS machine. Unfortunately, GitHub Actions doesn't provide them by default, so we need to set up a self-hosted option instead.

Replace Xcode-specific `sys_includes` path in `clang-macos-arm64` toolchain

The current include paths for the clang-macos-arm64 toolchain require the Xcode is installed:

sys_includes = [
    "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include",
    "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/{compiler_version}/include",

     ],

These paths should instead be changed to paths that only require the developer command line tools to be installed:

sys_includes = [
    "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include",
    "/Library/Developer/CommandLineTools/usr/lib/clang/{compiler_version}/include",
]

Build with `bazel` on `openbsd-x86_64`

Build the urbit binary on openbsd-x86_64. This will build off the work done in #2. The steps required:

• Tweak third-party dependency bazel build rules in bazel/third_party/ to build on openbsd-x86_64 platforms.
• Move the platform-specific third-party dependency patches in pkg/urbit/compat/openbsd/ to the appropriate dependency directory in bazel/third_party/; verify the patches are still correct in the context of the dependency versions we're using; and apply the patches to the dependency builds in WORKSPACE.bazel using select().
• Migrate the platform-specific files in pkg/urbit/compat/openbsd/ to pkg/noun/platform/openbsd/x86_64/ or pkg/vere/platform/openbsd/x86_64/ (whichever is more appropriate) and incorporate them in the appropriate build rule(s).
• Tweak the bazel build rules for all packages in pkg/ to build on openbsd-x86_64 platforms.

Test release of 1.16

We should perform a test release of 1.16 to ensure the release flow works as expected. Of chief importance is ensuring that a fake ship running 1.15 can use the next subcommand to get the 1.16.

Use `bazel` caching to improve CI performance

Use bazel caching to improve automated build and test performance.

Build for `macos-x86_64` without sandboxing

Due to a dependency resolution error¹ when building for macos-x86_64 on macos-12 GitHub Actions hosted runner machines, we use --spawn_strategy=sandboxed and --sandbox_block_path=/usr/local/include to build successfully in our CI workflow. Unfortunately, using Bazel's sandboxing feature effectively disables caching, resulting in macos-x86_64 build/test times of usually 15+ minutes. Comparatively, our builds for linux-arm64 and linux-x86_64 (which both utilize caching) usually complete in under 3 minutes.

We need to determine why the dependency resolution error occurs and implement a solution which doesn't use sandboxing. Once done, our macos-x86_64 builds will automatically begin to benefit from caching in the same way the other two target's builds do.

Footnotes

1. https://github.com/urbit/vere/actions/runs/3586593418/jobs/6035968777 ↩

Use `bazel`'s remote caching feature

Describe the bug
https://bazel.build/remote/caching

Switch from tag-based release flow to push-based release flow

To simplify the release flow, we should switch from a tag-based flow to a push-based release flow. In the new scheme, we'll have a release branch, which has all work that is tested, reviewed, and ready to go out in the next release. All feature work will target the release branch. Any time a commit is pushed to a feature branch--any branch that is not master or release--all unit and integration tests will run via the feature flow. Once a feature is ready to be staged for release, the feature's PR will merge into release, at which point the pre-release flow will run, triggering binary uploads for the soon train to bootstrap.urbit.org. Once a release is ready to go out, the version number in pkg/vere/VERSION will be bumped and release will be rebased on top of master, triggering the release flow to run, which will re-run all unit and integration tests and--assuming the tests pass--trigger binary uploads for the live train to bootstrap.urbit.org.

Build with `bazel` on `darwin-arm64`

Build the urbit binary on darwin-arm64 (i.e. Apple Silicon machines running macOS). This will build off the work done in #2. The steps required:

• Tweak third-party dependency bazel build rules in bazel/third_party/ to build on darwin-arm64 platforms.
• Move the platform-specific third-party dependency patches in pkg/urbit/compat/m1brew/ to the appropriate dependency directory in bazel/third_party/; verify the patches are still correct in the context of the dependency versions we're using; and apply the patches to the dependency buidls in WORKSPACE.bazel using select().
• Incorporate the compilation flags in pkg/urbit/compat/m1brew/compat.mk into the appropriate build rule(s) in pkg/.
• Tweak the bazel build rules for all packages in pkg/ to build on darwin-arm64 platforms.

Remove support for `openbsd` and `windows` platforms

Uncomment definition of U3_OS_ARCH on macos-aarch64

The definition of U3_OS_ARCH on macos-aarch64 is commented out in pkg/c3/portable.h. Now that we have a native macos-aarch63 build, this should be uncommented.

Run HTTP client IO driver in a separate process

The HTTP client IO driver should be rewritten (in Rust) to run in a separate process to improve security by sandboxing one of Urbit's interfaces to the outside world and to enable future scalability. The steps required:

1. Re-implement _cttp_ef_http_client() (the HTTP client driver function defined in pkg/vere/io/cttp.c that releases effects) in Rust in the IO drivers repo, rendering the existing driver implementation redundant.
2. Replace the now-redundant HTTP client driver C implementation with an implementation that asynchronously writes a request to the new HTTP client process's stdin and reads any subsequent response asynchronously from the HTTP client proccess's stdout.

Both Scaling Urbit's Runtime and Scalable IO provide additional relevant context for this work.

Review and update continuous deploy flow

Upon completion of the new bazel-based CI flow (see #8), we need to review and likely update our continuous deploy flow in .github/workflows/release.yml for releasing new versions of the urbit binary.

This requires that #8 is complete.

`cc_test` targets fail to build for `linux-arm64`

Description

All cc_test targets fail to build when targeting linux-arm64 because of a failure to detect a suitable toolchain. Toolchain resolution for tests apparently differs from library and binary targets, presumably because tests are built for the execution platform rather than the target platform by default.

Reproduction Steps

To reproduce, run the following on an x86_64 machine with Linux and version 12.2.0 of aarch64-linux-gnu-gcc installed:

% bazel build --aarch64_linux_gnu_gcc_version="12.2.0" --host_platform=//:aarch64_linux_gnu_gcc-linux-x86_64 --platforms=//:linux-arm64 --toolchain_resolution_debug='.*' ...
INFO: Build option --toolchain_resolution_debug has changed, discarding analysis cache.
INFO: ToolchainResolution: Target platform //:linux-arm64: Selected execution platform //:aarch64_linux_gnu_gcc-linux-x86_64,
INFO: ToolchainResolution: Target platform //:linux-arm64: Selected execution platform //:aarch64_linux_gnu_gcc-linux-x86_64,
INFO: ToolchainResolution:   Type @bazel_tools//tools/cpp:toolchain_type: target platform //:linux-arm64: execution //:aarch64_linux_gnu_gcc-linux-x86_64: Selected toolchain //bazel/toolchain:aarch64_linux_gnu_gcc-linux-x86_64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform //:linux-arm64: Rejected toolchain //bazel/toolchain:clang-linux-x86_64; mismatching values: x86_64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform //:linux-arm64: Rejected toolchain //bazel/toolchain:gcc-linux-x86_64; mismatching values: x86_64
INFO: ToolchainResolution:   Type @bazel_tools//tools/cpp:toolchain_type: target platform //:linux-arm64: execution //:aarch64_linux_gnu_gcc-linux-x86_64: Selected toolchain //bazel/toolchain:aarch64_linux_gnu_gcc-linux-x86_64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform //:linux-arm64: Rejected toolchain //bazel/toolchain:clang-linux-x86_64; mismatching values: x86_64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform //:linux-arm64: Rejected toolchain //bazel/toolchain:gcc-linux-x86_64; mismatching values: x86_64
INFO: ToolchainResolution: Target platform //:linux-arm64: Selected execution platform //:aarch64_linux_gnu_gcc-linux-x86_64, type @bazel_tools//tools/cpp:toolchain_type -> toolchain //bazel/toolchain:aarch64_linux_gnu_gcc-linux-x86_64
INFO: ToolchainResolution: Target platform //:linux-arm64: Selected execution platform //:aarch64_linux_gnu_gcc-linux-x86_64, type @bazel_tools//tools/cpp:toolchain_type -> toolchain //bazel/toolchain:aarch64_linux_gnu_gcc-linux-x86_64
INFO: ToolchainResolution: Target platform //:aarch64_linux_gnu_gcc-linux-x86_64: Selected execution platform //:aarch64_linux_gnu_gcc-linux-x86_64,
INFO: ToolchainResolution: Target platform //:linux-arm64: Selected execution platform //:aarch64_linux_gnu_gcc-linux-x86_64,
INFO: ToolchainResolution:   Type @rules_foreign_cc//toolchains:m4_toolchain: target platform //:linux-arm64: execution //:aarch64_linux_gnu_gcc-linux-x86_64: Selected toolchain @rules_foreign_cc//toolchains:preinstalled_m4
INFO: ToolchainResolution:   Type @rules_foreign_cc//toolchains:automake_toolchain: target platform //:linux-arm64: execution //:aarch64_linux_gnu_gcc-linux-x86_64: Selected toolchain @rules_foreign_cc//toolchains:preinstalled_automake
INFO: ToolchainResolution: Target platform //:aarch64_linux_gnu_gcc-linux-x86_64: Selected execution platform //:aarch64_linux_gnu_gcc-linux-x86_64,
INFO: ToolchainResolution:   Type @rules_foreign_cc//toolchains:pkgconfig_toolchain: target platform //:linux-arm64: execution //:aarch64_linux_gnu_gcc-linux-x86_64: Selected toolchain @rules_foreign_cc//toolchains:preinstalled_pkgconfig
INFO: ToolchainResolution:   Type @rules_foreign_cc//toolchains:make_toolchain: target platform //:linux-arm64: execution //:aarch64_linux_gnu_gcc-linux-x86_64: Selected toolchain @rules_foreign_cc//toolchains:preinstalled_make
INFO: ToolchainResolution:   Type @rules_foreign_cc//toolchains:autoconf_toolchain: target platform //:linux-arm64: execution //:aarch64_linux_gnu_gcc-linux-x86_64: Selected toolchain @rules_foreign_cc//toolchains:preinstalled_autoconf
INFO: ToolchainResolution:   Type @rules_foreign_cc//foreign_cc/private/framework:shell_toolchain: target platform //:linux-arm64: execution //:aarch64_linux_gnu_gcc-linux-x86_64: Selected toolchain @rules_foreign_cc_framework_toolchain_linux//:commands
INFO: ToolchainResolution: Target platform //:linux-arm64: Selected execution platform //:aarch64_linux_gnu_gcc-linux-x86_64, type @rules_foreign_cc//toolchains:autoconf_toolchain -> toolchain @rules_foreign_cc//toolchains:preinstalled_autoconf, type @rules_foreign_cc//toolchains:m4_toolchain -> toolchain @rules_foreign_cc//toolchains:preinstalled_m4, type @rules_foreign_cc//foreign_cc/private/framework:shell_toolchain -> toolchain @rules_foreign_cc_framework_toolchain_linux//:commands, type @bazel_tools//tools/cpp:toolchain_type -> toolchain //bazel/toolchain:aarch64_linux_gnu_gcc-linux-x86_64, type @rules_foreign_cc//toolchains:pkgconfig_toolchain -> toolchain @rules_foreign_cc//toolchains:preinstalled_pkgconfig, type @rules_foreign_cc//toolchains:automake_toolchain -> toolchain @rules_foreign_cc//toolchains:preinstalled_automake, type @rules_foreign_cc//toolchains:make_toolchain -> toolchain @rules_foreign_cc//toolchains:preinstalled_make
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform //:aarch64_linux_gnu_gcc-linux-x86_64: Rejected toolchain //bazel/toolchain:aarch64_linux_gnu_gcc-linux-x86_64; mismatching values: arm64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: execution platform //:aarch64_linux_gnu_gcc-linux-x86_64: Rejected toolchain //bazel/toolchain:clang-linux-x86_64; mismatching values: clang
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: execution platform //:aarch64_linux_gnu_gcc-linux-x86_64: Rejected toolchain //bazel/toolchain:gcc-linux-x86_64; mismatching values: gcc
INFO: ToolchainResolution:   Type @bazel_tools//tools/cpp:toolchain_type: target platform //:aarch64_linux_gnu_gcc-linux-x86_64: No toolchains found.
ERROR: /home/tlon/.cache/bazel/_bazel_tlon/d85bfd16708db53fe097086a278bf2d0/external/remote_coverage_tools/BUILD:9:12: While resolving toolchains for target @remote_coverage_tools//:Main: No matching toolchains found for types @bazel_tools//tools/cpp:toolchain_type. Maybe --incompatible_use_cc_configure_from_rules_cc has been flipped and there is no default C++ toolchain added in the WORKSPACE file? See https://github.com/bazelbuild/bazel/issues/10134 for details and migration instructions.
ERROR: Analysis of target '//pkg/noun:nock_tests' failed; build aborted:
INFO: Elapsed time: 0.271s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (1 packages loaded, 1225 targets configu\
red)

Related

• https://bazel.build/extending/exec-groups#execution-groups-and-platform-execution-properties
• bazelbuild/bazel#13626
• bazelbuild/bazel#12719

Migrate from Urbit monorepo

Pre-migration checklist

• linux-x86_64: clean build
• linux-x86_64: boot fake ship
• linux-x86_64: boot comet
• macos-arm64: clean build
• macos-arm64: boot fake ship
• macos-arm64: boot comet
• macos-x86_64: clean build
• macos-x86_64: boot fake ship
• macos-x86_64: boot comet
• Review and merge #21.
• Perform a dummy release and ensure the resulting binaries work on all four supported platforms.
  • linux-x86_64
  • macos-arm64
  • macos-x86_64
• Grab any additional commits to next/vere in urbit/urbit that have landed since 71d06e544f7fe641ac0a9dad1fe07576761227ec.
• Review and update documentation to remove references to non-runtime work.
• Prepare updates to binary links throughout urbit.org
• Complete final review of old build system to ensure all features have been ported over.

Post-migration checklist

• Remove all runtime-related files and references in urbit/urbit.
• Clearly communicate to runtime developers that they need to file an issue against this repo for anything that appears broken in the new repo.
• Collect list of open runtime-related PRs in urbit/urbit and re-open them in this repo.

Replace `arm64` with `aarch64` as canonical identifier for 64-bit ARM architecture

We've been using arm64 to identify the 64-bit ARM architecture in the build system, but they are the same, and aarch64 appears to be more widely used.

Build with `bazel` on `linux-x86_64`

Build the urbit binary on linux-x86_64. The steps required:

• Build all third-party dependencies from source using bazel.
• Build the urcrypt package using bazel.
• Split the urbit package into three separate packages--c3, noun, and vere--with clearly defined, directed dependency relationships between the three.

Point binary upgrade system back to https://bootstrap.urbit.org/vere

#58 (via #60 ) pointed the binary upgrade system to a test endpoint--https://bootstrap.urbit.org/vere-test. Once we complete testing the new release pipeline, we should point the binary upgrade system back to https://bootstrap.urbit.org/vere.

CI action logs are too noisy

Describe the bug
For example: https://github.com/urbit/vere/actions/runs/3847814390/jobs/6554760836

75%+ of the logs are relatively unhelpful tar lines printed during musl installation.

Build with `bazel` on `mingw-x86_64`

Build the urbit binary on mingw-x86_64. This will build off the work done in #2. The steps required:

• Tweak third-party dependency bazel build rules in bazel/third_party/ to build on mingw-x86_64 platforms.
• Move the platform-specific third-party dependency patches in pkg/urbit/compat/mingw/ to the appropriate dependency directory in bazel/third_party/; verify the patches are still correct in the context of the dependency versions we're using; and apply the patches to the dependency builds in WORKSPACE.bazel using select().
• Migrate the platform-specific files in pkg/urbit/compat/mingw/ to pkg/noun/platform/mingw/x86_64 or pkg/vere/platform/mingw/x86_64 (whichever is more appropriate) and incorporate them in the appropriate build rule(s).
• Tweak the bazel build rules for all packages in pkg/ to build on mingw-x86_64 platforms.

Automatically create GitHub releases

Use the gh CLI to create a new GitHub release as part of the existing release flow in our CI code. GitHub releases should include:

• a compressed archive for each of our supported platforms,
• release notes,
• and a list of contributions/commits since last release.

Overwrite binaries deployed to `soon` and `edge` trains

The binaries deployed to https://bootstrap.urbit.org/vere/{edge|soon}/v{version_number}/vere-v{version_number}-{platform} should be overwritten as we update the tip of develop (the branch tracked by the edge train) and release (the branch tracked by the soon train) in preparing a release. Binaries deployed to the live train should never be overwritten.

Correctly cast `hun_y` when trimming newlines in `u3_king_next()`

hun_y is a c3_y *, but strrchr() expects a c3_c *. We should cast hun_y to c3_c *.

`argon2`, `gmp`, and `softfloat` have "no symbols" (linker warnings on `macos-x86_64`)

Describe the bug

"has no symbols" linker warnings for these libraries occur when building with bazel build //... on macos-x86_64:

INFO: From Linking external/argon2/libargon2.a:
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: bazel-out/darwin-fastbuild/bin/external/argon2/libargon2.a(thread.o) has no symbols
...
INFO: From Linking external/softfloat/softfloat.a:
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: bazel-out/darwin-fastbuild/bin/external/softfloat/libsoftfloat_x86_64.a(s_countLeadingZeros16.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: bazel-out/darwin-fastbuild/bin/external/softfloat/libsoftfloat_x86_64.a(s_countLeadingZeros32.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: bazel-out/darwin-fastbuild/bin/external/softfloat/libsoftfloat_x86_64.a(s_countLeadingZeros64.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: bazel-out/darwin-fastbuild/bin/external/softfloat/libsoftfloat_x86_64.a(s_mul64ByShifted32To128.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: bazel-out/darwin-fastbuild/bin/external/softfloat/libsoftfloat_x86_64.a(s_mul64To128.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: bazel-out/darwin-fastbuild/bin/external/softfloat/libsoftfloat_x86_64.a(s_mul128By32.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: bazel-out/darwin-fastbuild/bin/external/softfloat/libsoftfloat_x86_64.a(s_mul128To256M.o) has no symbols

Expected behaviour

The libraries don't generate symbols in these files on linux-x86_64 either, so there's no error. We'd just like to suppress the warnings.

Additional context

I will add more context next week.

Build with musl instead of glibc on Linux

The urbit binary should link against musl, which supports static linking, instead of glibc. When linked against glibc, an urbit binary may fail to run on a different machine than the one it was compiled on, even if both machines share an OS and architecture, because of mismatches in the glibc version. Statically linking against musl addresses this problem. Unfortunately, musl availability is confined to Linux because it's built atop the Linux system call API.¹

Related

musl-cross-make: a tool for compiling musl-targeting compilers

Footnotes

1. See https://musl.libc.org/about.html. ↩

Make release pipeline resilient to newline in `pkg/vere/VERSION`

If pkg/vere/VERSION ends in a newline, the binary upgrade system fails with the following:

% zod/.run next
~
urbit 1.15
boot: home is /home/tlon/code/gh/urbit/new-vere/zod
loom: mapped 2048MB
lite: arvo formula 11a9e7fe
lite: core 38d4ad4d
lite: final state 38d4ad4d
loom: mapped 2048MB
boot: protected loom
live: loaded: MB/54.853.632
boot: installed 351 jets
vere: next (%edge): 1.16

curl: failed https://bootstrap.urbit.org/vere-test/edge/v1.16
/vere-v1.16
-linux-x86_64: Error
unable to save https://bootstrap.urbit.org/vere-test/edge/v1.16
/vere-v1.16
-linux-x86_64 to /home/tlon/code/gh/urbit/new-vere/zod/.bin/edge/vere-v1.16
-linux-x86_64: -1
vere: upgrade failed
vere: prep for upgrade failed

Correct upload of Docker image

This CI job turned up issues with //:upload_docker. The tag attribute of container_push takes a string, not a label. Instead of the string_flag image_tag, we should instead use a genrule that generates a file with the appropriate container tag from pkg/vere/VERSION and pkg/vere/PACE.

Ensure GitHub workflow caches aren't flushed

For some reason (probably a simple size limit rule), our Bazel and musl caches for our CI jobs are getting flushed, making build times very slow when they don't have to be.

Point binary upgrade system to bootstrap.urbit.org/vere-test

Change the URL prefix the runtime binary upgrade system uses from https://bootstrap.urbit.org/vere to https://bootstrap.urbit.org/vere-test to test the release pipeline system.

Migrate `git` history from Urbit monorepo

Migrate all runtime-related commits from next/vere in the Urbit monorepo. The last commit shared by this repo and the Urbit monorepo is df9a0e7c1d317cf556dc9752e348009d90d3ce99, which means we need all commits in next/vere committed after df9a0e7c1d317cf556dc9752e348009d90d3ce99.

Document build system's dependency on `automake` and `libtool` on Linux

When building on Linux, the automake and libtool suite of tools need to be installed, just as they do on macOS. INSTALL.md should call attention to this.

Build with `bazel`

Build the runtime using bazel for each of the following supported platforms:

• #3
• #2
• #4
• #5
• #6
• #7

macos: `bazel build //pkg/vere:boot-fake-ship` produces `libsigsegv (stackoverflow_deinstall_handler): Cannot allocate memory`

Describe the bug
On both macos-x86_64 and macos-arm64, booting a ship works without error when executing the urbit binary from the console. However, when trying to boot a fake ship from within bazel (with bazel build //pkg/vere:boot-fake-ship), a libsigsegv error prints during the boot phase:

matt@mbp14 vere % bazel build //pkg/vere:boot-fake-ship
...
libsigsegv (stackoverflow_deinstall_handler): Cannot allocate memory
...

The ship finishes booting and seems to behave normally from the dojo, however. Also, when re-booting the ship from the resulting pier, no libsigsegv (or other) errors appear.

To Reproduce
Steps to reproduce the behaviour:

1. bazel build //pkg/vere:boot-fake-ship

Expected behaviour
No error message.

System (please supply the following information, if relevant):

• OS: macOS Ventura 13.0
• Vere 1.13

Additional context

See pkg/noun/manage.c for additional context.

Notify maintainers
@mcevoypeter @joemfb