rust-lang / rust-central-station

Old home of some infra pieces

Archived repository!

This repository was the home of some of the most important services hosted by the Rust Infrastructure Team, but over time those services were moved to new homes:

• The release process is now developed and deployed in rust-lang/promote-release.
• Bors is now developed, configured and deployed in rust-lang/homu.
• Team synchronization is now developed and deployed in rust-lang/sync-team.
• Cancelbot is not in use anymore.

Rust Central Station

Or otherwise just another name for the old buildmaster.

This repo is hooked up to an automated docker build

• https://hub.docker.com/r/alexcrichton/rust-central-station/

On the destination machine you can run it as:

./run-prod.sh

Services currently provided are:

• cancelbot for rust-lang/rust
• cancelbot for rust-lang/cargo
• homu
• nginx in front of homu
• ssl via letsencrypt

Future services

• signing Rust releases

Architecture

This is intended to be run as a container on the destination server, so the container here specifies everything about what's being run.

• Secrets are stored in secrets.toml next to secrets.toml.example and are shared with the container.
• Programs are provided in the container (tq and rbars) which will read the TOML configuration for use in shell scripts.
• Everything pipes output to logger to collect output
• Services are just run as simple daemons, not a lot of management.

rust-central-station's Issues

Promotion should use a consistent date and time

The sign-and-hash invocation on x.py calls date, but so does RCS on startup. If these straddle a new day, bad things happen.

One big hammer is to use faketime to fix the time at the top level and make every date command return the same value.

Test generated manifests against rustup

Refiling rust-lang/rust#41767 here.

Several times, PRs have changed the format of the manifests in such a way that they can no longer be installed by rustup. This results in the corresponding nightlies being uninstallable, and doesn't give much time to fix it before the next nightly: rust-lang/rustup#1092 (comment)

We should add tests to catch these problems before they occur.

bors/homu should set S- tags

when bors acknowledges a commit, set to S-waiting-on-bors, if a merge fails, set to S-waiting-on-author. There would still be some manual work, but significantly less.

Add ability to roll back a nightly rust release

We should have the ability to "roll back" the latest nightly release in case of breakage such as rustup is incompatible with the manifest and/or artifacts or when the artifacts themselves are just busted. This rollback should be a documented command that's probably a subset of the promote-release binary. This command would just take the artifacts from the previous night and redeploy them to the non-archive locations.

Migrate rust-lang-ci bucket to us-west-1

To reasons to do this:

• us-east-1, the current location of this bucket, tends to be the most frequent region to experience service outages.
• rust-central-station is located in us-west-1, so right now we're publishing artifacts from across the country.

AFAIK this isn't just a switch we can flip, so I suspect the migration path will look like:

• Create a new bucket. Probably rust-lang-ci2. Make sure it's in us-west-1
• Turn on replication of rust-lang-ci2, set it to replicate to rust-lang-ci.
• Update Travis/AppVeyor to publish to rust-lang-ci2
  • This should transitively publish to rust-lang-ci based on replication
• Update rust-central-station to pull from rust-lang-ci2
• Migrate all existing data in rust-lang-ci to rust-lang-ci2 -- not sure how to do this, I've heard there's a "COPY operation"
• Delete the rust-lang-ci bucket
• Help fix clients that are now broken.
  • Preliminary list: Rustbuild, Servo, bisect-rust.

Directory listing generation should be faster

Today this takes ~15 minutes which is quite long and with the current approach will only grow.

There are two incremental steps here:

• We should only be generating an index for the "current upload"; not for the whole bucket. This is probably relatively easy and would be enough for quite a while.
• Stop generating indices at upload and generate them client-side via a custom 404 page.

run-prod.sh is wrong

Since all the data seems to be in /opt, the following is correct:

docker run --name rcs --volume `pwd`/data:/data --volume `pwd`/data/letsencrypt:/etc/letsencrypt --volume `pwd`/logs:/var/log --publish 80:80 --publish 443:443 --rm alexcrichton/rust-central-station

The data under ~ should be nuked as well.

robots.txt is not being updated

doc.rust-lang.org/robots.txt isn't getting updated, though /stable/robots.txt (and other URLs) are; those are however useless to my knowledge because crawlers only look in the root.

This is a follow up on rust-lang/rust#68677

cc @steveklabnik

Tracking issue: Rustbuild submodule management can break nightlies

We had a brief outage of nightlies, which is Not Supposed to be Possible.

The problem:

Rustbuild itself checks out submodules, but on CI that's not what we do. On CI a script manages submodules and then the build happens in a container where rustbuild takes over.

It's possible to check in a change that breaks rustbuild submodule management, which means ./configure + make is not guaranteed to work.

The change that broke in this instance was that a Cargo.toml that was a member of the rustbuild workspace was a member of a submodule, so for cargo to build rustbuild we needed to update submodules. To update submodules, though, we needed rustbuild. 🐍 🐍

A fix has now been implemented to move submodule management to the x.py python script (src/bootstrap/bootstrap.py), so this hopefully won't happen again, but this issue is to track if it ever does so that @alexcrichton can eat his hat 🎩 and we can invest time in making rust-central-station even more like CI.

Letsencrypt is broken in a long-running RCS container, and can only be fixed by chance

https://mozilla.logbot.info/rust-infra/20180620#c14911644-c14911887

a) it seems to not renew unless the whole container is restarted
b) it only seems to renew on container restart because the certificates revert back to ones from 2017 somehow, despite the whole letsencrypt directory being mounted into the container. This seems obviously wrong, but on the plus side does provide a workaround for a

Make nagbot filter on S- tags

Now that PRs have solid status information, we should use it for nagbot to filter the list!

Logs on central station are still inconsistent

the syslog files appear empty and new entries seem to be going into syslog.1 which feels wrong.

cc @aidanhs

Gate rustc on both Azure and (GitHub Actions - macOS)

Gate rustc CI on both Azure and GitHub Actions with the exception of macOS
builders to quickly detect builds that will inevitably fail on Azure, while
avoiding spurious failures described in rust-lang/rust#71988.

Time to failure (TTF) on Azure and GitHub Actions (GHA) from a few different
rollups (measurements for GHA include non-spurious failures from macOS builders
under simplifying assumption that they would also fail elsewhere around the
same time):

Rollup	Azure TTF	GHA TTF
73271	1h 48m 1s	22m 4s
73240	1h 7m 30s	23m 52s
73231	2h 22m 45s	25m 48s
73168	1h 50m 34s	23m 41s
73064	2h 19m 42s	42m 13s
73062	1h 58m 42s	41m 24s
73053	2h 5m 57s	37m 32s

Potential drawbacks: increased likelihood of spurious failures.

Option to rollback to the previous release

It would be nice to have a tool to rollback to the previous release in a channel. This would be used, for example, when a broken nightly is released, to minimise the ecosystem breakage.

An implementation idea we had during the infra meeting was copying the previous manifest with a .old suffix during the release process, so rolling back is just copying it back.

Mailing list synchronization broken

Our script to synchronize Mailgun mailing lists is currently broken, and stops working when trying to update all@. This is probably caused by the huge amount of people subscribed to all@.

Apr 15 16:00:02 localhost sync-mailgun: error: failed to sync ^all(?:\+.+)?@rust-lang\.org$
Apr 15 16:00:02 localhost sync-mailgun:   cause: failed to send request to /routes/5c5b47a662957400010358ba
Apr 15 16:00:02 localhost sync-mailgun:   cause: failed to get a 200 code, got 400
Apr 15 16:00:02 localhost sync-mailgun: 
Apr 15 16:00:02 localhost sync-mailgun: {
Apr 15 16:00:02 localhost sync-mailgun:   "message": "total stored \"actions\" cannot exceed 4k bytes"
Apr 15 16:00:02 localhost sync-mailgun: }

Dependabot can't resolve your Rust dependency files

Dependabot can't resolve your Rust dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

error: failed to parse lock file at: /home/dependabot/dependabot-updater/dependabot_tmp_dir/Cargo.lock

Caused by:
  package `error-chain` is specified twice in the lockfile

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

Add license to rust-central-station

rust-central-station doesn't currently have any Open Source licensing explicitly listed. We'd like to relicense rust-central-station to Rust's standard MIT OR Apache-2.0 license, so that we can freely copy code between it and other Rust projects. In order to safely do that, we would like explicit permission from the following people:

• @Mark-Simulacrum
• @RalfJung
• @SimonSapin
• @aidanhs
• @alexcrichton
• @aturon
• @davidtwco
• @frewsxcv
• @kennytm
• @oli-obk
• @phansch
• @pietroalbini
• @thibaultdelor

If you're OK with us using your contributions under Rust's standard Open Source license, please post a comment copy-pasting the following:

I agree to license my contributions to rust-central-station under Rust's standard MIT OR Apache-2.0 dual-license.

We'll track approvals using the checkboxes above, and once we have all the boxes checked, we'll relicense the repository.

Thank you!