brndnmtthws / labhub

GitHub bot for using GitLab CI in OSS projects

🤖 LabHub

A GitHub bot written in Rust for using GitLab CI in OSS projects.

Features

• Listens for webhooks from GitHub
• Pushes branches to GitLab from external (forked) PRs
• Accepts commands by way of PR comments
• Possibly more coming soon 👻

Commands

Commands can be executed by commenting on a PR with your CI user's login.

• @labhub retry: retry a pipeline that has failed

The Problem

GitLab has a great CI system, however it's not suitable for open source projects 😧 (at the time of writing) because it won't build external PRs by default. There are security concerns about the risk of exposing secrets in external builds, and GitLab errs on the side of caution by not building external PRs by default.

For more details on the issue, please take a look at this GitLab discussion.

✨ The Solution

If you're not concerned with leaking secrets, then LabHub may be for you! LabHub listens for webhooks from GitHub to notify for new pull requests. If the PR is from a forked repo, it will push a branch to GitLab (for the corresponding PR) to test the build.

🏃‍♀️ In Action

Using LabHub? Open a PR to add your project here! 😀

LabHub is currently being used by the following projects:

• Conky
• DOMjudge

Compiling

LabHub requires Rust nightly. To compile using rustup:

$ rustup toolchain install nightly
$ rustup default nightly
$ cargo build

Be sure to switch back to stable with rustup default stable if that's your preferred toolchain.

🎛 Configuration

LabHub is configured using LabHub.toml. For details, see src/config.rs. You can specify the path to LabHub.toml by setting the LABHUB_TOML environment variable.

🚀 Deployment

Setup Webhooks

You'll need to set up webhooks for any repo you wish to enable LabHub for. Currently, only GitHub webhooks are required. To get started, go to github.com/<org>/<repo>/settings/hooks and add a new webhook.

Configure the webhook to send PR and push events.

• Set the payload URL path to /github/events, which is the path LabHub is expecting for GitHub events.
• Create a secret (ex: cat /dev/urandom | LC_CTYPE=C tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1) and set the same value in the webhook config as in LabHub.
• Make sure the payload type is application/json.
• Here's how your webhook should look

Create SSH keys

You'll need a CI user with SSH keys for both GitHub and GitLab. Create an account on both sites (if you don't already have a CI user), and create an SSH key for LabHub:

$ ssh-keygen -f labhub-key.ecdsa -t ecdsa -b 521

Keep labhub-key.ecdsa safe, and upload labhub-key.ecdsa.pub to both GitHub and GitLab for the CI user.

Create Personal Access Tokens

Create personal access tokens for your CI user on both GitHub, and GitLab. Supply these tokens by setting the api_token parameter in LabHub.toml for both GitHub and GitLab.

Personal Access Token for GitHub

• Go to https://github.com/settings/tokens
• Click "Generate new token"
• Give the token a name, and enable the repo scope, like this.
• Save that token to your LabHub.toml

Personal Access Token for GitLab

• Go to https://gitlab.com/profile/personal_access_tokens
• Give the token a name, and enable the api scope, like this.
• Save that token to your LabHub.toml

Deploy to Kubernetes with Helm

There's a Helm chart included in this repo, which is the preferred method of deployment. To use you, you must first create the SSH key secrets with kubectl. Assuming your SSH private key is labhub-key.ecdsa:

$ kubectl create secret generic labhub-ssh-keys --from-file=github=labhub-key.ecdsa --from-file=gitlab=labhub-key.ecdsa

You may use separate keys for GitHub and GitLab if you choose, respectively.

Once you have the secrets, install the helm chart from helm/labhub/:

$ cd helm/labhub/
$ cp values.yaml myvalues.yaml
### Edit myvalues.yaml to your liking ###
$ helm upgrade --install labhub . -f myvalues.yaml

Not implemented:

• No periodic reconciling of GitLab branches with open PRs: if a webhook is missed for any reason, the GitLab pipeline may not correctly reflect the PR state

labhub's Issues

Not able to deploy with docker

I updated the LabHub file with my credentials and then did a docker build. It works fine but the moment i do docker run it gives me the following error:
Warning: environment is 'production', but no secret_key is configured
thread 'main' panicked at 'Unable to open the file: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/libcore/result.rs:1084:5
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace.

Mirroring PRs fails due to SSH error

I've already opened an issue with git2 as I guess it's an issue with either that library or the libssh2 that it's using but maybe this is some known issue and can be worked around somehow or I just did something wrong… (Although googling for that error it seemed that it is caused by a bug in libssh2)

When I trigger a PR mirror it fails with the following error:

labhub[929]: Error: Caught error handling PR: GitError { message: "Git error: \"Failed to authenticate SSH session: Unable to send userauth-publickey request\" …

Build errors

Hi! I was hoping to use this tool for one of my repositories, but after installing rustup and running the 3 commands from the Readme:

rustup toolchain install nightly
rustup default nightly
cargo build

I get the following build errors:

error[E0277]: the `?` operator can only be applied to values that implement `Try`
  --> src/api/github_client.rs:39:31
   |
39 |       let res: github::RepoPr = client
   |  _______________________________^
40 | |         .get(&format!("{}/pulls/{}", make_repo_url(org, repo), number))
41 | |         .headers(headers(&config::CONFIG.github.api_token))
42 | |         .send()?
   | |________________^ the `?` operator cannot be applied to type `impl futures::Future`
   |
   = help: the trait `Try` is not implemented for `impl futures::Future`
   = note: required by `into_result`

error[E0277]: the `?` operator can only be applied to values that implement `Try`
  --> src/api/github_client.rs:54:19
   |
54 |       let mut res = client
   |  ___________________^
55 | |         .post(&format!(
56 | |             "{}/issues/{}/comments",
57 | |             make_repo_url(org, repo),
...  |
61 | |         .body(serde_json::json!({"body":body.to_string()}).to_string())
62 | |         .send()?;
   | |________________^ the `?` operator cannot be applied to type `impl futures::Future`
   |
   = help: the trait `Try` is not implemented for `impl futures::Future`
   = note: required by `into_result`

error[E0277]: the `?` operator can only be applied to values that implement `Try`
  --> src/api/gitlab_client.rs:51:38
   |
51 |       let res: Vec<gitlab::Pipeline> = client
   |  ______________________________________^
52 | |         .get(&format!(
53 | |             "{}/pipelines?page={}&per_page={}",
54 | |             make_api_url(project),
...  |
58 | |         .headers(headers(&config::CONFIG.gitlab.api_token))
59 | |         .send()?
   | |________________^ the `?` operator cannot be applied to type `impl futures::Future`
   |
   = help: the trait `Try` is not implemented for `impl futures::Future`
   = note: required by `into_result`

error[E0277]: the `?` operator can only be applied to values that implement `Try`
  --> src/api/gitlab_client.rs:69:15
   |
69 |       let res = client
   |  _______________^
70 | |         .post(&format!(
71 | |             "{}/pipelines/{}/retry",
72 | |             make_api_url(project),
...  |
75 | |         .headers(headers(&config::CONFIG.gitlab.api_token))
76 | |         .send()?;
   | |________________^ the `?` operator cannot be applied to type `impl futures::Future`
   |
   = help: the trait `Try` is not implemented for `impl futures::Future`
   = note: required by `into_result`

error: aborting due to 4 previous errors

For more information about this error, try `rustc --explain E0277`.
error: could not compile `labhub`

I never did anything with Rust before so I'm a bit at a loss on how to fix this. Would appreciate any help ;)

I'm on Arch linux, but since this is using Rustup, I don't think that should matter?

Report job status in Github PR

It would be great if the labhub bot would send back the job status of the Gitlab build on the Github PR using the Github status API.

Add support for /ok-to-test for triggering CI job to protect secrets

Taking kubernetes PR procedure as example, it would be cool if CI jobs could be triggered only on /ok-to-test command, rather than on PR open/commit push.

This way, maintainer needs to manually inspect the PR before allowing it in CI.

Since support for commands is already implemented, this should be very easy to add. It could actually use the same functionality as retry command.

LabHub breaks when setting a milestone on a PR

When you set a milestone on a PR, LabHub breaks with the following message:

{"error":"Error(\"invalid type: map, expected a string\", line: 1, column: 1919)"}

I think this is because the code assumes milestone is a Option<String> but it is either null or an object. See

labhub/src/api/models/github.rs

Line 418 in 9e759f6

pub milestone: Option<String>,

Sometimes fails to sync pull request branch after force push

On some occasions when I force push to a branch for a pull request it doesn't trigger a pipeline. On the LabHub console I see:

Error: Caught error handling PR: GitError { message: "Git error: \"cannot push non-fastforwardable reference\"" }

Does it need to force push branches from GitHub to GitLab? I am puzzled why this doesn't happen every time though.

pipeline is to often triggered

First, thanks for this nice bot.

I realized that our gitlab CI pipeline is triggered each time I change e.g. the title of a pull request or assign a developer to the request.

This behavior is coming from the grouped github events for Pull request. It is in github not possible to set events only for Pull request opened" and Pull updated`(hash changed)

github configuration for webhooks:

Is it possible to update the bot that a pipe is only triggered when the pull request is opened or updated (e.g. forcepush or new commits)?

CC-ing: @SimeonEhrig