juliaci / external-buildkite-buildkite-plugin

Download Buildkite configuration files from an external repository.

external-buildkite Buildkite Plugin

Download Buildkite configuration files from an external repository.

Example

To use this plugin, add the following YAML snippets to your pipeline.yml file:

steps:
  - plugins:
    - JuliaCI/external-buildkite#v1.0:
        version: './buildkite.version'
        repo_url: 'https://github.com/JuliaCI/julia-buildkite.git'
        folder: '.buildkite'

Note that version can be a gitsha, a gitref (e.g. master), or the path to a file that contains within it a gitsha or gitref. If you opt to use a file for version, you must specify its path starting with ./ or / to disambiguate it from an arbitrary gitref.

external-buildkite-buildkite-plugin's Issues

Why special-case first job?

I don't quite understand why we special-case the first job?

Add support for caching repositories with `--reference`

We should do what the buildkite agent does with the "git mirrors" experiment (which we generally have turned on).

Broadly, we should do:

REPO_URL="${BUILDKITE_PLUGIN_EXTERNAL_BUILDKITE_EXTERNAL_REPO}"
REPO_URL_HASH="$(shasum -a 256 <<<"${REPO_URL}")"
REFERENCE_REPO_PATH="/cache/external-buildkite/repos/${REPO_URL_HASH}"

# Clone our persistent mirror repository
if [[ ! -d "${REFERENCE_REPO_PATH}" ]]; then
    git clone --mirror "${REPO_URL}" "${REFERENCE_REPO_PATH}"
else
    git fetch -C "${REFERENCE_REPO_PATH}" origin
fi

# Clone a fresh checkout, using our mirror as a reference (this should generally never require network access)
git clone --reference "${REFERENCE_REPO_PATH}" "${REPO_URL}" ...

Allow the user to override the value of the `buildkite.version` file, maybe via an environment variable?

Suppose that I go to Buildkite, select a pipeline (e.g. julia-release-1.6), and click the "New Build" button. The dialog box that pops up includes an "Environment Variables" section in which I can specify custom environment variables as a list of key-value pairs.

Perhaps we can use this as a method for running builds with a specific buildkite.version.

For example, suppose that the release-1.6 branch of the JuliaLang/julia repository has a buildkite.version file with the following contents:

buildkite-1.6

This tells the JuliaCI/external-buildkite plugin that we should use the buildkite-1.6 branch of the JuliaCI/julia-buildkite repository.

Suppose that we instead want to make a build of the release-1.6 branch of Julia, but instead of using the buildkite-1.6 Buildkite configuration, we want to specify a custom commit from the JuliaCI/julia-buildkite repository.

Perhaps we can have an environment variable that the user can use to provide the custom Buildkite commit that they want to use. And if this environment variable is set (and nonempty), we ignore the buildkite.version file and instead use the commit specified in the environment variable.

I'm not sure what the best user interface is here, but some way of achieving this feature would be useful.

Use Buildkite agent metadata?

We can work around this by having the first job set a buildkite metadata value. E.g. we can do something like:

METADATA_KEY="external-buildkite-${REPO_URL_HASH}-${EXTERNAL_BRANCH_NAME}"
CONCRETE_EXTERNAL_GITSHA="$(buildkite-agent meta-data get "${METADATA_KEY}" "")"
if [[ -z "${CONCRETE_EXTERNAL_GITSHA}" ]]; then
    CONCRETE_EXTERNAL_GITSHA="$(do_git_fetch "${REPO_URL}")"
    buildkite-agent meta-data set "${METADATA_KEY}" "${CONCRETE_EXTERNAL_GITSHA}"
else
    do_git_checkout "${REPO_URL}" "${CONCRETE_EXTERNAL_GITSHA}"
fi

Originally posted by @staticfloat in #13 (comment)

Bikeshed: come up with a better name for this plugin?

I wonder if we can come up with a better name for this plugin?

@staticfloat @maleadt

Incorrect behavior when the branch in the external repo (e.g. `JuliaCI/julia-buildkite`) is updated while a build is running

Consider the following hypothetical situation:

JuliaLang/julia repository	JuliaCI/julia-buildkite repository
release-1.6	buildkite-1.6

In other words, suppose that the release-1.6 branch of the JuliaLang/julia repository is tracking the buildkite-1.6 branch of the JuliaCI/julia-buildkite repository. More concretely, this means that the release-1.6 branch contains a file named buildkite.version with the following contents:

buildkite-1.6

Suppose that, currently, the commit history of the buildkite-1.6 branch looks like this:

A
|
B
|
C <--- buildkite-1.6

In other words, the buildkite-1.6 branch currently points to commit C.

Suppose that a build is started on the release-1.6 branch. Suppose that our build matrix only contains two jobs: package_linux64 and tester_linux64. The tester_linux64 job cannot start until after the package_linux64 has successfully finished.

So, the package_linux64 job starts, and it pulls the latest version of the buildkite-1.6 branch, which gives it the Buildkite configuration files corresponding to commit C in the JuliaCI/julia-buildkite repository.

While the package_linux64 job is still running, someone pushes a new commit to the buildkite-1.6 branch. Now, the commit history of the buildkite-1.6 branch looks like this:

A
|
B
|
C
|
D <--- buildkite-1.6

After the package_linux64 job finishes, the tester_linux64 job starts, and it pulls the latest version of the buildkite-1.6 branch, which gives it the Buildkite configuration files corresponding to commit D in the JuliaCI/julia-buildkite repository.

So this has led to a very strange situation, in which two jobs in the same build have used different Buildkite configuration files:

Job	JuliaCI/julia-buildkite repository
package_linux64	Commit C
tester_linux64	Commit D

I think that this is a bug. I think we want to enforce the same Buildkite configuration across all jobs in a single build.

Specifically, I think that, in this example, we want to enforce that all jobs in this build use commit C, because that was the commit to which the buildkite-1.6 branch was pointing when the build first started.