rust-lang-deprecated / rust-buildbot Goto Github PK

This seems like an internal tool, but it might be a good idea to change from 'master/slave' to something less problematic.

I'm not familiar with this tool, but primary/replica might work. (or leader/follower)

@steveklabnik @brson, thoughts?

This should probably use the linux builder with the old build system as we would want to run a full make check like we do with the x86_64-bit musl support.

cc #78
cc @japaric

Currently, Tiers-2 platforms (bitrig-64-opt, linux-musl-64-opt, freebsd10_32-1, freebsd10_64-1, dragonflybsd-64-opt, and openbsd-64-opt) have buildbot configuration but are ignored for any failure by @bors.

I wonder if it could be revised to be ignored only for failing in test, and make a failure on compile be blocking too for merging ?

Eventually having 3 levels for platforms:

• Tiers-1 platforms: any failure is blocking
• Tiers-2 platforms: any failure on compile is blocking (but failure on test are not)
• Tiers-3 platforms: not blocking

I don't known enough buildbot for proposing PR. But I can try if this proposition have any interest.

This issue is to track my desire to have a nightly musl build.

We can't use a release of llvm because 3.7 isn't out yet (https://llvm.org/bugs/show_bug.cgi?id=23364), so I'd say there's nothing to do at the moment.

The packaging builders on nightly have an "upload" step where they upload their artifacts to the buildmaster, but on the macs this can take hours where the Linux/Windows machine take seconds. The buildmaster is on EC2 and so are the Linux/Windows machines, but we're pushing at most 1GB of data from Macstadium to EC2 which shouldn't take an hour.

I've watched locally and it looks like stunnel is uploading at ~100KB/s. So far I've tried (none have worked)

• Restarting stunnel
• Adding TCP_NODELAY to our stunnel config on the client
• Updating stunnel to 5.35

I also was able to shove about 10MB/s of zeros via nc to the buildmaster, so I don't think the network is at fault here.

cc @brson this is probably causing the invalidation script to get fired before we actually upload artifacts
cc @edunham, @larsbergstrom -- infra issue, y'all seen this over at servo?

A regression recently popped up which apparently would have been caught if the buildbots actually had LLVM assertions enabled rust-lang/rust#35991 (comment)

Relevant discussion: rust-lang/rust#20629

curl-sys isn't compiling because it can't find the OpenSSL libraries:

configure: error: OpenSSL libs and/or directories were not found where specified!
thread '<main>' panicked at 'assertion failed: t!(cmd . status (  )).success()', /home/rustbuild/src/rust-buildbot/slave/cargo-linux-64/.cargo/registry/src/github.com-1ecc6299db9ec823/curl-sys-0.1.21/build.rs:128

Full output: http://buildbot.rust-lang.org/builders/nightly-dist-cargo-linux-64/builds/135/steps/compile/logs/stdio

It's been failing since May 5th. I apologize if this isn't the right place to report this issue.

/cc @brson?

As Firefox looks to adding Rust support on all of its supported platforms, we'd like i686-linux-android (tier 3) to be given the same status as arm-linux-androideabi (tier 2). Part of that is having buildbot understand how to build binaries.

It's been broken twice in two weeks, so let's use that processing power to make sure we don't break it again. Perhaps also upload the docs somewhere so we have another location should @Manishearth 's docs get outdated/broken.

cc @retep998

Example.

Reason is multiple files with the same name inside the different source directories; we need to filter them somehow. Particularly the rust-src tarballs and maybe docs? A ls -R should be helpful.

Would be good to at least have as an option. Unfortunately other cross builds of Cargo happen in the linux-cross image and this'll need to happen in the linux image. We may as well run the full test suite of Cargo though as that'll help prevent even more regressions.

Soon wasm32-experimental-emscripten will land in stable Rust.

Could you guys please add the new target?

Would of been great to have this sooner, before stable, for testing purposes, but I guess this is ok too.

Ping @alexcrichton @brson

We recently added a privileged user for building 'dist' builds. After that patch though non-privileged users can no longer cancel non-dist builds.

cc @bheesham @alexcrichton

So I've wanted to take a look on the AArch64 build testing due to rust-lang/rust#33737, but apparently the bot isn't public? I don't see aarch64 on the builders page, and in https://github.com/rust-lang/rust-buildbot/blob/master/master/master.cfg#L253, it's marked as NOAUTO. What does that mean?

On an unrelated build: https://buildbot.rust-lang.org/builders/auto-win-msvc-64-opt-rustbuild/builds/3217
The """freshconfig""" key is set to the value """0 /usr/bin/find: paths must precede expression: ^ Usage: /usr/bin/find [-H] [-L] [-P] [-Olevel] [-D help|tree|search|stat|rates|opt|exec] [path...] [expression]"""

Relevant source: https://github.com/rust-lang/rust-buildbot/blob/master/master/master.cfg#LC976

The find command is valid on my macOS machine here.

It caused the should_wipe function to return the wrong value, making the WORK_DIR directory to not be removed before git clean -d -f -f in the next step. No clue if it contributed to the build failure.

Until MIR is turned on by default for everyone, we want to make sure we don't regress.

In rust-lang/rust#32080 I've tagged relevant functions in tests that involve features not yet in MIR (such as debuginfo or overflow checks) with #[rustc_no_mir], and we can keep adding those as long as we track them, so I believe the new MIR builders should gate all builds.

We're going to need at least one UNIX builder and one MSVC because of the unwinding complexities found in the latter (I don't even know if MSVC can bootstrap w/ MIR right now).

Blocked on the snapshots being added in rust-lang/rust#32345. cc @alexcrichton

https://buildbot.rust-lang.org/builders/nightly-dist-rustc-musl-linux/builds/322/steps/configure/logs/stdio (from https://buildbot.rust-lang.org/builders/nightly-dist-rustc-musl-linux/builds/322) has things like

configure: CFG_GCC              := /usr/bin/gcc (5.4.0-6ubuntu1)

which indicate that perhaps the dist builder is no longer centos. In fact, the dist centos should be version 4.7.4 - https://github.com/rust-lang/rust-buildbot/blob/master/slaves/dist/build_gcc.sh

Is https://github.com/rust-lang/rust-buildbot/blob/master/slaves/dist/Dockerfile (https://hub.docker.com/r/alexcrichton/rust-slave-dist/) no longer in use?

It is currently not in a condition that allows people to easily set up their own instances.

To resolve, file issues documenting all reasons it's difficult for a clean instance to be set up. These will probably be along the lines of missing documentation and missing sample configurations.

I would like instructions that would allow a random member of the public to set up an instance, because I would like to know exactly how it works, because it will soon be my problem when it breaks.

Firefox compiles its Android port with version 9 of the Android platform (not sure if I have the terminology right here). The buildbot configs use version 18, but at least on x86 Android, compiling with version 9 works just fine, so I assume that's the case for the ARM Android port as well. If ARM Android works with version 9, I'd like to request that the automated builds use version 9 to make life more consistent for Firefox's sake.

I believe this is blocking rust-lang/rustup#797.

This is tested with alexcrichton/rust-slave-linux-cross:2016-11-11 on a local dev server:

rustbuild@13a39093d77d:/buildslave$ echo 'int main(){return 0;}' > test.c    
rustbuild@13a39093d77d:/buildslave$ powerpc-linux-gnu-gcc -m64 ./test.c  
/usr/lib/gcc-cross/powerpc-linux-gnu/5/../../../../powerpc-linux-gnu/bin/ld: skipping incompatible /usr/lib/gcc-cross/powerpc-linux-gnu/5/libgcc.a when searching for -lgcc
/usr/lib/gcc-cross/powerpc-linux-gnu/5/../../../../powerpc-linux-gnu/bin/ld: cannot find -lgcc
/usr/lib/gcc-cross/powerpc-linux-gnu/5/../../../../powerpc-linux-gnu/bin/ld: skipping incompatible /usr/lib/gcc-cross/powerpc-linux-gnu/5/libgcc_s.so.1 when searching for libgcc_s.so.1
/usr/lib/gcc-cross/powerpc-linux-gnu/5/../../../../powerpc-linux-gnu/bin/ld: skipping incompatible /usr/lib/gcc-cross/powerpc-linux-gnu/5/../../../../powerpc-linux-gnu/lib/libgcc_s.so.1 when searching for libgcc_s.so.1
/usr/lib/gcc-cross/powerpc-linux-gnu/5/../../../../powerpc-linux-gnu/bin/ld: skipping incompatible //usr/powerpc-linux-gnu/lib/libgcc_s.so.1 when searching for libgcc_s.so.1
/usr/lib/gcc-cross/powerpc-linux-gnu/5/../../../../powerpc-linux-gnu/bin/ld: cannot find libgcc_s.so.1
/usr/lib/gcc-cross/powerpc-linux-gnu/5/../../../../powerpc-linux-gnu/bin/ld: skipping incompatible /usr/lib/gcc-cross/powerpc-linux-gnu/5/libgcc.a when searching for -lgcc
/usr/lib/gcc-cross/powerpc-linux-gnu/5/../../../../powerpc-linux-gnu/bin/ld: cannot find -lgcc
collect2: error: ld returned 1 exit status
rustbuild@13a39093d77d:/buildslave$ powerpc64-linux-gnu-gcc-5 -m64 ./test.c
rustbuild@13a39093d77d:/buildslave$ file ./a.out 
./a.out: ELF 64-bit MSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), dynamically linked, interpreter /lib64/ld64.so.1, for GNU/Linux 3.2.0, BuildID[sha1]=aad3d80e94dfdbbb80763dbabeb351287ec3a02e, not stripped

We're publishing it publicly on Github, which implies an intent to share it with the world, but we don't tell people what they're actually allowed to do with it. Since it's all configuration files, I don't know whether CC-BY or the standard Rust code license would be more appropriate.

We should either add a LICENSE/COPYING file or a note in the README discussing how people are allowed to use the contents of this repo.

• Pull the glibc version in the cross image backwards, probably the same way as arm-unknown-linux-gnueabi
• Add auto builder cross-compiling to aarch64 (346e9a2)
• Add nightly builder cross compiling to aarch64 for the compiler (346e9a2)
• Add cargo build for aarch64 (rust-lang/cargo#2538)
• Add rustup build for aarch64 (rust-lang/rustup#257)

cc @japaric
cc @brandonedens

Rust now supports these additional targets:

arm-unknown-linux-musleabi
arm-unknown-linux-musleabihf
armv7-unknown-linux-musleabihf

Could we get builds for these targets?

Just like we had no-mir bot, shouldn't we have no-rustbuild bot?

Just a reminder of the extra binaries that can be build in the linux-cross image after its next rebuild.

• libstd for mips(el)-unknown-linux-musl (2 targets)
  • Image: linux-cross
  • Uses: the old build system
  • Probably just tack thxs on top of the 'nightly-dist-rustc-cross-linux' builder. Add both targets
    to the --target argument of the configure script.
• rustc for arm(v7)-unknown-linux-gnueabi(hf) (3 hosts)
  • Image: linux-cross
  • Uses: rustbuild
  • Blocked on: rust-lang/rust#32237
  • This wants a new builder.
  • Since this job will also produce libstd tarballs, perhaps we can remove these 3 targets from the 'nightly-dist-rustc-cross-linux' builder and have this builder upload 3 rustc tarballs and 3 libstd tarballs.
• libstd for i686-unknown-linux-musl
  • Image: linux
  • Uses: the old build system
  • Blocked on #79
  • This should go in its own builder because AFAICT one can't build libstd for two static musl targets. And we can't do that because we would need to pass one --musl-root to configure for each target but that flag only accepts one argument.

cc @alexcrichton

Current site buildbot not working :(.

Although the rustc packages appear to keep their revisions in sync now, the combined rust packages do not. Probably I picked the wrong unique id to synchronize on that wasn't actually unique.

It appears that there's some issue building LLVM:

Latest failure: http://buildbot.rust-lang.org/builders/nightly-dist-rustc-linux/builds/570/steps/compile/logs/stdio
Linux nightly: http://buildbot.rust-lang.org/builders/nightly-dist-rustc-linux

See rust-lang/rustup#708 where cargo didn't exist.

The most likely reason for this to happen is that the network request in live_package_url failed. It should probably have a retry loop and not be allowed to fail for any package in the host_list.

After this dance with the old GCC 4.4.5 MIPS cross-compiler used in the linux-cross buildbot, I've been trying to update it.

Prebuilt MIPS cross compilers are unfortunately rather rare.

1. Ubuntu does not contain cross-compilers for MIPS in its repositories, even though Debian has them.
2. Inserting Debian's repositories into any version of Ubuntu results in spectacular dependency failures.
3. Debian Jessie (the current stable) does not have cross-compilers in its repositories yet.
4. Debian Stretch has MIPS cross-compilers. However ... they are built with soft-float disabled. Rust's C flags for MIPS specify soft-float, so we cannot use them.

There are a few options.

1. Modify the Ubuntu packages to build soft-float MIPS compilers, and continue using Ubuntu.

2. Modify the Debian packages to build soft-float MIPS compilers, and use Debian.

   These two solutions likely mean that someone has to recompile the entire GCC suite and glibc due to the way the Debian/Ubuntu package is set up. Ubuntu Launchpad lets us use build farms to build any source packages, so this can work. I'm particularly averse of dealing with Debian(/Ubuntu) packages though, because their packaging system is arcane and a mess.

3. Just build the compilers separately, don't care about packaging them.

   It may be difficult to update the compiler again in future.

4. Use a different distribution (with a saner packaging system).

   Not sure about the costs of doing so. I'm thinking of Arch as well, which is a rolling-release distribution, which cause Docker's Arch images to be difficult to update sometimes.

5. Just leave it. We only use GCC to build jemalloc, compiler-rt, etc., after all.

   It's an easy solution, but.. it'll probably mean we run into issues next time something is updated.