License: Mozilla Public License 2.0
Earthly's official collection of functions.
Earthly is licensed under the Mozilla Public License Version 2.0. See LICENSE.
When using the commands provided in the Earthfile I get the following error:
Warning: No version specified in github.com/earthly/lib/Earthfile. Implying VERSION 0.5, which is not the latest available. Please note that in the future, the VERSION command will be required for all Earthfiles.
Not sure if there's a reason for that.
As alluded to in issue #51, the --output argument to CARGO doesn't work properly when invoked multiple times with the same target directory.
Take the following example which you can run on a new project created with cargo new --bin foo:
VERSION 0.8
IMPORT github.com/earthly/lib/rust AS rust
install:
FROM rust:latest
RUN apt-get update -qq
RUN apt-get install --no-install-recommends -qq autoconf autotools-dev libtool-bin clang cmake bsdmainutils
DO rust+INIT --keep_fingerprints=true
source:
FROM +install
RUN echo asdf
COPY --keep-ts Cargo.toml Cargo.lock ./
COPY --keep-ts --dir src ./
build:
FROM +source
RUN rm -rf target
DO rust+CARGO --args="build --release --bins" --output="release/[^/\.]+"
RUN ./target/release/foo
test:
FROM +build
DO rust+CARGO --args="build --release --tests" --output="release/deps/foo-[^/\.]+"
RUN ./target/release/deps/foo-*
all:
BUILD +build
BUILD +test
earthly +all yields the following:
...
+build | --> RUN rm -rf target
+build | --> RUN if [ ! -n "$EARTHLY_CACHE_PREFIX" ]; then echo "+INIT has not been called yet in this build environment" ; exit 1; fi;
+build | --> RUN if [ ! -n "$EARTHLY_CACHE_PREFIX" ]; then echo "+INIT has not been called yet in this build environment" ; exit 1; fi;
+build | --> mkdir /run
+build | --> IF [ "$EARTHLY_KEEP_FINGERPRINTS" = "false" ]
+build | --> RUN set -e; cargo $args; cargo sweep -r -t $EARTHLY_SWEEP_DAYS; cargo sweep -r -i;
+build | Finished release [optimized] target(s) in 0.00s
+build | [INFO] Searching recursively for Rust project folders
+build | [INFO] Cleaned 0.00 B from "/target"
+build | [INFO] Searching recursively for Rust project folders
+build | [INFO] Using all installed toolchains: ["1.76.0-x86_64-unknown-linux-gnu"]
+build | [INFO] Cleaned 0.00 B from "/target"
+build | --> mkdir /run
+build | --> IF [ "$output" != "" ]
+all | --> BUILD +test
+build | --> RUN if [ ! -n "$EARTHLY_RUST_TARGET_CACHE" ]; then echo "+SET_CACHE_MOUNTS_ENV has not been called yet in this build environment" ; exit 1; fi;
+build | --> RUN if [ -n "$output" ]; then echo "Copying output files" ; mkdir -p $TMP_FOLDER; cd target; find . -type f -regextype posix-egrep -regex "./$output" -exec cp --parents {} $TMP_FOLDER \; ; cd ..; fi;
+build | Copying output files
+build | --> RUN mkdir -p target; mv $TMP_FOLDER/* target 2>/dev/null || echo "no files found within ./target matching the provided output regexp";
+build | --> RUN ./target/release/foo
+build | Hello, world!
+test | --> RUN if [ ! -n "$EARTHLY_CACHE_PREFIX" ]; then echo "+INIT has not been called yet in this build environment" ; exit 1; fi;
+test | --> RUN if [ ! -n "$EARTHLY_CACHE_PREFIX" ]; then echo "+INIT has not been called yet in this build environment" ; exit 1; fi;
+test | --> mkdir /run
+test | --> IF [ "$EARTHLY_KEEP_FINGERPRINTS" = "false" ]
+test | --> RUN set -e; cargo $args; cargo sweep -r -t $EARTHLY_SWEEP_DAYS; cargo sweep -r -i;
+test | Finished release [optimized] target(s) in 0.00s
+test | [INFO] Searching recursively for Rust project folders
+test | [INFO] Cleaned 0.00 B from "/target"
+test | [INFO] Searching recursively for Rust project folders
+test | [INFO] Using all installed toolchains: ["1.76.0-x86_64-unknown-linux-gnu"]
+test | [INFO] Cleaned 0.00 B from "/target"
+test | --> mkdir /run
+test | --> IF [ "$output" != "" ]
+test | --> RUN if [ ! -n "$EARTHLY_RUST_TARGET_CACHE" ]; then echo "+SET_CACHE_MOUNTS_ENV has not been called yet in this build environment" ; exit 1; fi;
+test | --> RUN if [ -n "$output" ]; then echo "Copying output files" ; mkdir -p $TMP_FOLDER; cd target; find . -type f -regextype posix-egrep -regex "./$output" -exec cp --parents {} $TMP_FOLDER \; ; cd ..; fi;
+test | Copying output files
+test | --> RUN mkdir -p target; mv $TMP_FOLDER/* target 2>/dev/null || echo "no files found within ./target matching the provided output regexp";
+test | no files found within ./target matching the provided output regexp
+test | --> RUN ./target/release/deps/foo-*
+test | /bin/sh: 1: ./target/release/deps/foo-*: not found
+test | ERROR Earthfile line 26:2
+test | The command
+test | RUN ./target/release/deps/foo-*
+test | did not complete successfully. Exit code 127
The main warning I'm looking at is +test | no files found within ./target matching the provided output regexp. This is unexpected since cargo build --release --tests should produce an output that matches that regex.
If I simply add rm -rf target in the test target, it works:
test:
FROM +build
RUN rm -rf target
DO rust+CARGO --args="build --release --tests" --output="release/deps/foo-[^/\.]+"
RUN ./target/release/deps/foo-*
+test | Copying output files
+test | --> RUN mkdir -p target; mv $TMP_FOLDER/* target 2>/dev/null || echo "no files found within ./target matching the provided output regexp";
+test | --> RUN ./target/release/deps/foo-*
+test | running 0 tests
+test | test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
This is because the output copy logic tries a blind mv of the $TMP_FOLDER contents to the target without checking to see if a directory already exists:
RUN mkdir -p target; \
mv $TMP_FOLDER/* target 2>/dev/null || echo "no files found within ./target matching the provided output regexp";
If I simply remove the stderr redirect to /dev/null, the error message shows what's going on:
mv: inter-device move failed: '/tmp/earthly/lib/rust/release' to 'target/release'; unable to remove target: Directory not empty
The rust library currently requires that the version of rust being used is able to build the latest version of cargo-sweep.
Line 170 in 855c98f
The current master branch of cargo-sweep only builds on rust 1.74.0+ which is then a soft-MSRV of this library. Users can work around this by installing cargo-sweep before calling INIT, but this is UX which could be improved.
Because we're installing the latest commit of cargo-sweep and cargo-sweep doesn't seem to do any checks its own MSRV, this could update at any point. It's then practically required to install cargo-sweep separately as otherwise you can be rugged at any point.
Note that we're not even using the lockfile for cargo-sweep so if any of its dependencies update their MSRV improperly then that also rugs users of this library.
Ideally we'd be locking to a particular version of cargo-sweep with a fixed lockfile to try to avoid unnecessary MSRV bumps. e.g. using
cargo install [email protected] --locked
would allow for installing a consistent version of this dependency which makes it easier for users to rely on the library's default behaviour.
Hi, apologies if this has been asked before but ive hit a bit of a snag when trying to use rust+CROSS to build for the aarch64-apple-darwin target. Given that rs-cross does not host this target, i need to have a local instance loaded. Normally id would do this with something like:
WITH DOCKER \
--load ghcr.io/cross-rs/aarch64-apple-darwin:0.2.3
RUN ...
END
Is there a way to psecify local images to load when using rust+CROSS?
Cargo cross requires docker or podman installed.
Earthly doesn't support calling a FUNCTION within a WITH DOCKER statement, so there the rust library cannot be used to cache cargo cross runs as it is.
The rust+CARGO UDC is great. One thing that tripped me up was --output="release/[^\./]+"
If no output is supplied could we just assume everything in /target/ should be the output? At least if doing cargo build, I think I would usually want the output of it somewhere accessible.
Self-hosted satellites now have public documentation. Pricing is currently documented as being based on user count, and go on to say
If you would like to estimate the number of users that a project has according to how we track users, please check out the github.com/earthly/lib/billing+active-users target.
That target currently only works for gitlab and github. It's likely that users/evaluators of self-hosted satellites are doing so due to having self-hosted CI (such as myself), so it would be nice if the target worked with self-hosted git solutions as well.
I've seen cross mentioned in the documentation, so I thought it could be easy to run it. Not that much so far :-)
Here is what I tried to do:
VERSION --global-cache 0.7
IMPORT github.com/earthly/lib/rust AS rust
rust:
FROM rust:alpine
RUN apk add --no-cache musl-dev findutils
DO rust+INIT --keep_fingerprints=true
cross:
FROM +rust
DO rust+CARGO --args="install cross"
RUN apk add docker jq
ARG --required target
DO rust+SET_CACHE_MOUNTS_ENV
COPY --keep-ts . ./
WITH DOCKER
RUN --mount=$EARTHLY_RUST_CARGO_HOME_CACHE \
--mount=$EARTHLY_RUST_TARGET_CACHE \
cross build --target $target --release
END
DO rust+COPY_OUTPUT --output="[^\./]+/release/[^\./]+"but cross fails with this error:
+cross | build=cross target=i686-unknown-linux-musl
+cross | --> WITH DOCKER RUN --privileged cross build --target $target --release
+cross | Starting dockerd with data root /var/earthly/dind/f93b09b85d482e3c2c2a9ba24dac0885b16ada45cadeaaa5667074ec0e54ffda/tmp.HMgOjE
+cross | error: DEPRECATED: future versions of rustup will require --force-non-host to install a non-host toolchain as the default.
+cross | warning: toolchain '1.75.0-x86_64-unknown-linux-gnu' may not be able to run on this system.
+cross | warning: If you meant to build software to target that platform, perhaps try `rustup target add x86_64-unknown-linux-gnu` instead?
+cross | info: syncing channel updates for '1.75.0-x86_64-unknown-linux-gnu'
+cross | info: latest update on 2023-12-28, rust version 1.75.0 (82e1608df 2023-12-21)
+cross | info: downloading component 'cargo'
+cross | info: downloading component 'rust-std'
+cross | info: downloading component 'rustc'
+cross | info: installing component 'cargo'
+cross | info: installing component 'rust-std'
+cross | info: installing component 'rustc'
+cross | 1.75.0-x86_64-unknown-linux-gnu installed - (error reading rustc version)
+cross | error: rustup is not installed at '/tmp/earthly/.cargo'
+cross | Error:
+cross | 0: couldn't install toolchain `1.75.0-x86_64-unknown-linux-gnu`
+cross | 1: `rustup toolchain add 1.75.0-x86_64-unknown-linux-gnu --profile minimal` failed with exit status: 1
+cross | earthly debugger | Command /bin/sh -c 'cross build --target $target --release' failed with exit code 1
$ earthly --version
earthly version v0.8.3 70916968c9b1cbc764c4a4d4d137eb9921e97a1f linux/amd64; Ubuntu 23.10 (Mantic Minotaur)
Is there a example with cross available somewhere?
TLDR: Compose v2 causes problems.
The title is misleading (I wasn't actually trying/wanting to use docker:dind), but it's a true statement, and the root of the actual problem I wanted to solve.
The recommended earthly/dind:alpine image solves 99% of my problems, but I have one project where it's preferrable to utilize the INSTALL_DIND UDC on top of an unrelated image. I was getting annoyed with the VERSION warning, so went to create a PR for it (already tracked in #5). It's a one-line fix, but I still ran the tests, and was surprised to see them fail.
The root cause of the test failures is a combination of two things:
docker:dind image currently bundles a V2 compose (but with a docker-compose symlink, which satisfies the auto-install check)There are many ways to deal with this; if there was a clear "best" solution I would have submitted a PR. This is far from exhaustive (and some can be combined), but the 'easy' potential solutions:
docker:dind from the test suite (it's presence implies it's supported)-p to the compose invocations (which lets the compose file live in the root directory)write_compose_config and use the resulting config (not to conflate multiple issues, but doing this may address a bug-not-yet-reported, and the fact that it writes to a non-root directory is what solves the immediate problem)The non-easily solutions basically amount to "natively support V2" (which could also pave the way for #6). That's further complicated due to "natively supporting V2" can mean supporting the V2 version (what we're dealing here, since there's a compatibility symlink), or supporting the V2 CLI syntax (i.e., docker compose vs. docker-compose).
Another possibility (but an obviously breaking change) would be to drop support for --compose entirely. That sounds crazy... but personally, if there was a --load docker-compose.yml, I doubt I would ever use --compose. In other words, the value (for me) is the loading of images, not in saving me from doing up myself (which is usually a hindrance). I may bring this up as feature request in the main earthly project, but figured I'd throw it out there in case it spurred other thoughts.
Again, this issue (as described in the title) does not actually impact me (though it may impact someone else). But it does impact the ability to add changes to this repo due to the failing tests.
Isolating it to the one failing test:
$ earthly -v
earthly version v0.6.24 20339818277375d1addc9cc20066a9890483761e linux/amd64; Ubuntu 20.04.5 LTS (Focal Fossa)
$ earthly -P --no-cache +test-install-dind-for-image --base_image=docker:dind
1. Init 🚀
————————————————————————————————————————————————————————————————————————————————
buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)
2. Build 🔧
————————————————————————————————————————————————————————————————————————————————
Warning: No version specified in ./Earthfile. Implying VERSION 0.5, which is not the latest available. Please note that in the future, the VERSION command will be required for all Earthfiles.
docker:dind | --> Load metadata linux/amd64
alpine:3.13 | --> Load metadata linux/amd64
+install-dind-script | --> FROM alpine:3.13
context | --> local context .
+test-install-dind-for-image | base_image=docker:dind
+test-install-dind-for-image | --> FROM docker:dind
+install-dind-script | [██████████] 100% resolve docker.io/library/alpine:3.13@sha256:100448e45467d4f3838fc8d95faab2965e22711b6edf67bbd8ec9c07f612b553
+test-install-dind-for-image | [██████████] 100% resolve docker.io/library/docker:dind@sha256:1be6768d332a4aaed59b344e6525e1701c467c4ef8553d5d18bac086cbd82ef1
ongoing | context (5 seconds ago)
context | [ ] 0% transferring .:
context | [██████████] 100% transferring .:
+install-dind-script | --> COPY ./install-dind.sh ./
+install-dind-script | --> SAVE ARTIFACT ./install-dind.sh +install-dind-script/install-dind.sh
+test-install-dind-for-image | --> COPY +install-dind-script/install-dind.sh /tmp/install-dind.sh
+test-install-dind-for-image | --> RUN /tmp/install-dind.sh
+test-install-dind-for-image | jq is missing. Attempting to install automatically.
+test-install-dind-for-image | fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/x86_64/APKINDEX.tar.gz
+test-install-dind-for-image | fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/community/x86_64/APKINDEX.tar.gz
+test-install-dind-for-image | (1/2) Installing oniguruma (6.9.8-r0)
+test-install-dind-for-image | (2/2) Installing jq (1.6-r1)
+test-install-dind-for-image | Executing busybox-1.35.0-r17.trigger
+test-install-dind-for-image | OK: 31 MiB in 57 packages
+test-install-dind-for-image | /usr/local/bin/dockerd
+test-install-dind-for-image | /usr/local/bin/docker-compose
+test-install-dind-for-image | base_image=docker:dind
+test-install-dind-for-image | --> RUN echo "
+test-install-dind-for-image | version: \"3\"
+test-install-dind-for-image | services:
+test-install-dind-for-image | hello:
+test-install-dind-for-image | image: hello-world:latest
+test-install-dind-for-image | " >./docker-compose.yml
+test-install-dind-for-image | base_image=docker:dind
+test-install-dind-for-image | --> WITH DOCKER (install deps)
+test-install-dind-for-image | /usr/bin/jq
+test-install-dind-for-image | /usr/local/bin/dockerd
+test-install-dind-for-image | /usr/local/bin/docker-compose
+test-install-dind-for-image | base_image=docker:dind
+test-install-dind-for-image | --> WITH DOCKER (docker-compose config)
hello-world:latest | --> Load metadata linux/amd64
hello-world:latest | --> DOCKER PULL hello-world:latest
hello-world:latest | [██████████] 100% resolve docker.io/library/hello-world:latest@sha256:62af9efd515a25f84961b70f973a798d2eca956b1b2b026d0a4a63a3b0b6a3f2
ongoing | +test-install-dind-for-image (4 seconds ago)
context | transferred 1 file(s) for context /tmp/earthly-docker-load1692874303 (11 kB, 1 file/dir stats)
+test-install-dind-for-image | base_image=docker:dind
+test-install-dind-for-image | --> WITH DOCKER RUN --privileged true
+test-install-dind-for-image | Starting dockerd with data root /var/earthly/dind/c22eb5c3ac654bc3a3ba123aea987989c7ab25b3dd74017ad3aa36871ac86309/tmp.nMEmBc
+test-install-dind-for-image | Loading images from BuildKit via tar files...
+test-install-dind-for-image | Loaded image: hello-world:latest
+test-install-dind-for-image | ...done
+test-install-dind-for-image | Network _default Creating
+test-install-dind-for-image | Network _default Created
+test-install-dind-for-image | Container -hello-1 Creating
+test-install-dind-for-image | Error response from daemon: Invalid container name (-hello-1), only [a-zA-Z0-9][a-zA-Z0-9_.-] are allowed
+test-install-dind-for-image | ERROR: Command exited with non-zero code: WITH DOCKER RUN --privileged true
============================ ❌ FAILURE [2. Build 🔧] ============================
Repeating the output of the command that caused the failure
+test-install-dind-for-image *failed* | base_image=docker:dind
+test-install-dind-for-image *failed* | --> WITH DOCKER RUN --privileged true
+test-install-dind-for-image *failed* | Starting dockerd with data root /var/earthly/dind/c22eb5c3ac654bc3a3ba123aea987989c7ab25b3dd74017ad3aa36871ac86309/tmp.nMEmBc
+test-install-dind-for-image *failed* | Loading images from BuildKit via tar files...
+test-install-dind-for-image *failed* | Loaded image: hello-world:latest
+test-install-dind-for-image *failed* | ...done
+test-install-dind-for-image *failed* | Network _default Creating
+test-install-dind-for-image *failed* | Network _default Created
+test-install-dind-for-image *failed* | Container -hello-1 Creating
+test-install-dind-for-image *failed* | Error response from daemon: Invalid container name (-hello-1), only [a-zA-Z0-9][a-zA-Z0-9_.-] are allowed
+test-install-dind-for-image *failed* | ERROR: Command exited with non-zero code: WITH DOCKER RUN --privileged true
Error: build target: build main: failed to solve: process "/bin/sh -c EARTHLY_DOCKERD_DATA_ROOT=\"/var/earthly/dind/c22eb5c3ac654bc3a3ba123aea987989c7ab25b3dd74017ad3aa36871ac86309\" EARTHLY_DOCKER_LOAD_FILES=\"/var/earthly/load-0/image.tar\" EARTHLY_IMAGES_WITH_DIGESTS=\"\" EARTHLY_START_COMPOSE=\"true\" EARTHLY_COMPOSE_FILES=\"docker-compose.yml\" EARTHLY_COMPOSE_SERVICES=\"\" DIND_COMMIT=42b1175eda071c0e9121e1d64345928384a93df1 DOCKER_BUILDX_VERSION=0.9.1 DOCKER_COMPOSE_VERSION=2.11.2 DOCKER_TLS_CERTDIR=/certs DOCKER_VERSION=20.10.18 EARTHLY_LOCALLY=false PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin base_image=docker:dind /var/earthly/dockerd-wrapper.sh execute /usr/bin/earth_debugger /bin/sh -c 'true'" did not complete successfully: exit code: 1
(A -p foo, or moving the compose file to /foo/, would both result in a valid container name of foo-hello-1.)
I have a unit test which depends on an external postgres database to run. With GitLab CI, I'm able to run the test like this:
unit-test:
...
variables:
POSTGRES_DB: keylay-test
POSTGRES_USER: keylay
POSTGRES_PASSWORD: keylay
services:
- postgres:latest
script:
- >
KEYLAY_TEST_DATABASE_URL="postgres://keylay:keylay@postgres/keylay-test"
cargo test --release --frozen --offline --features postgresI'm trying to figure out how to express the same test with Earthly (using lib/rust). I understand how to use WITH DOCKER to start up services, but I'm not sure how to run cargo test in that context.
If I were dealing with the cargo commands directly, I could do something like this at the end of my binary build container:
ENTRYPOINT ["cargo", "test", ... "--features", "postgres"]
SAVE IMAGE cargo-test:postgresAnd then run the resultant container in the WITH DOCKER context alongside the postgres container, but I don't see a way to do that. I realize this may be an odd use case, but I'm curious if there's an existing solution. If not, I'd be happy to help work on a solution!
Subsequent builds after the first run on my self-hosted satellite seem to consistently and infinitely block here:
Blocking waiting for file lock on package cache
I'm not sure if some lock file was left over after the previous build failure maybe? Since all the builds share a cargo home, perhaps we need to monitor that? I dunno enough about how cargo package locks work to say for sure though.
I don't know if this is possible in current Earthly.
Both the rust+CARGO command and the rust+RUN_WITH_CACHE command do not support --ssh or --mount=type=secret.
This makes it impossible to authenticate against and use private registries.
Sharing target mount cache across builds with different RUSTFLAGS overwrites cache entries and results in cache misses. The reason is $CARGO_TARGET_DIR cache entries, despite depending on RUSTFLAGS are not keyed by them.
Current version of the lib/rust library uses several mount caches per tuple of {runner, project, os_release}:
$CARGO_HOME, shared across all builds and supporting parallel access$CARGO_TARGET_DIR. One per calling Earthly target. Notice that an Earthly build can trigger multiple (Earthly) targets in parallel, each one of them using their own mount cacheThe problem comes when RUSTFLAGS are changed, some cache entries might be overwritten, or when the user Earthfile has successive +CARGO builds in the same Earthly target using different RUSTFLAGS. These scenarios would result in the loss of the original cache entries.
We could additionally key $CARGO_TARGET_DIR mount caches by the whole cargo command + hashOf(rustflags) to make sure these caches are only reused across builds of the same command, with the same RUSTFLAGS
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.