grahamc / r13y.com

NixOS Reproducibility Checker

What is this?

This repo is the tooling and automation to generate https://r13y.com, a website which tracks how reproducible the NixOS package builds are.

See https://reproducible-builds.org/ for information about why reproducible builds matter, other projects involved in the effort, and also a collection of tools and other information about reproducible builds.

How can I run this?

Right now, this repository builds every date at https://buildkite.com/grahamc/r13y-dot-com/

If you want to run it yourself, check out ./check.sh. It will need minor modifications (the rsync line) to complete successfully.

What might be next for the project?

Check out #4 for some discussion about what might be next.

r13y.com's Issues

website mentions freenode

There is support for an automatic diff-hook in Nix 2, but it is much more complicated to set up. If you would like to work on this, or need help setting it up, contact gchristensen on Freenode. We can work together to write docs on how to use it.

just FYI :)

Add issue links to Nixpkgs github for each unreproducible path

To keep track of status, and reduce googling.

Why is guile-2.0.13 in the report?

nix why-depends /nix/store/rmx9bqzfgi5mlr0sb9x9zgi047lhy4x3-nixos-minimal-21.03pre56789.gfedcba-x86_64-linux.iso /nix/store/jrnf3bpskbqx3652cw6ir1dq24vmy27x-guile-2.0.13.drv does not find a link.

nix-store --query --requisites /nix/store/rmx9bqzfgi5mlr0sb9x9zgi047lhy4x3-nixos-minimal-21.03pre56789.gfedcba-x86_64-linux.iso does not seem to list any guile path either.

Buildkite has not succeeded in quite a while

Between 1 Oct 2022 and 2 Oct 2022, the buildkite pipeline stopped being run:

This is because the agent tags could not be satisfied (

r13y.com/buildkite.yml

Lines 22 to 23 in 0e676b0

	- "r13y=true"
	- "packet-spot=true"

) -- there are no agents tagged with r13y=true and/or packet-spot=true (but there is one with packet-spot-buildkite-agent=true).

cc @grahamc

I want to check stable

I have a workstation with 8 core i9-9900K, 32 GB RAM and 512 GB SSD. Currently it does nothing, so i would like it to do some useful work.

I would like to check graphical stable for reproducibility. Just once, so we know where we are.

@grahamc do you have any idea how long that could take to build and how much space is needed? A week would be OK!

Can i publish my results on r13y.com? We could have a menu at the top for stable/unstable minimal/graphical.

When i built graphical, minimal should be very fast.

I will also use this issue to document my journey and plan to update the documentation.

Please add a README

• What is this?
• Why is the topic important?
• Who else is doing it?
• How can i run this?
• What are ideas to extend it?

A good guide including a template is https://www.makeareadme.com/!

Extend scope

This project is amazing! It shows that we are very far in the goal to make nixos reproducible.

At least for nixos-unstable's iso_minimal job for x86_64-linux.

It would be nice to have this metric for the stable releases. We might get some headlines for nixos and reproducibility.

It would also be nice to extend the scope of this "experiment" and integrate it into nixos. I'm not sure how this works because there is no documentation (#5), but you probably compare hashes? So, hydra should calculate this hashes and save them in the metadata of all builds. Then we need some distributed computing infrastructure where the community can run builds on their machines and submit their hashes. Then a central instance can calculate how reproducible each package is that has this data available. We can use that data on our website and package search.

I know BOINC for distributed computing tasks, but we might find a less complex solution.

I would like to just execute one command to build packages from a job set or channel i like and submit hashes.

Preserve hashes

If i understand correctly, the package hashes are lost after comparison.

Until we have some infrastructure to submit them, i suggest a pragmatic solution and just save them in a CSV file or no-sql db.

What datastructure do we need?

I spontaneously think of:

• timestamp of hash calculation
• nix store path of the package
• maybe package name for better human readability
• the hash of the built package
• name of submitter

How could we be sure that it's actually built by the submitter? Sign it with PGP like git?

When we have infrastructure to submit the data (that could be a very simple webapp with nosql backend), we can just upload the data that we preserve now.

Hash collection infrastructure

I wish that this effort is more distributed across more people (see also #4). To make that possible we need infrastructure to submit package hashes. With that, one can just build one or more packages and submit their hash. In best case, we have multiple hashes for every package.

(It could be an option in Nix to submit the hash of every done build automatically.)

I wish to have the hash most people got for a package (maybe call it consensual hash) visible on https://nixos.org/nixos/packages.html including a single command to build the package yourself and compare the hash (optionally submit)!

I want to discuss the architecture of such an infrastructure in this issue.

First, we need a datastructure to organize the data in. See #12 for that.

We need some datastore for structured data. I have never developed with a nosql db, but that's probably the right tool for this task.

Then we can have an API to submit and get data. A simple webinterface would be also nice with some stats and a way to nicely view and search the data.

This might be similar to PGP keyservers, like

https://keys.openpgp.org/
https://gitlab.com/hagrid-keyserver/hagrid/

A basic implementation seems very simple, but a good design seems very hard.

Should a submitter be able to add their name, so they get some appreciation for their work or should it be anonymous? If there is a name field, how do we prevent someone to use my name? Maybe it can be solved by using the e-mail address as handle, sign and verify it with PGP? How can someone trust the server to have correct data?

What are attack vectors? Someone could just submit random hashes or submit the hash of a package compiled with malware multiple times, so that hash is more often in the DB than ours. Is web-of-trust a solution for such a problem? Are there other approaches? Here it get's really hard and academic!

But that's a problem other reproducible builds efforts also have, so we can collaborate to find a solution.

Add a LICENSE

and prevent trouble like we had here NixOS/nixos-homepage#260

buildkite status

Right now, the README states:

Right now, this repository builds every date at https://buildkite.com/grahamc/r13y-dot-com/

That's not true. buildkite is broken and build fail

https://buildkite.com/grahamc/r13y-dot-com/

@grahamc what's the plan with buildkite? do you see a chance this could work in the future or should we just remove it?

Add URL with results to repo

https://r13y.com/

Clarify checks

After the last change (#22), there are now 2 pages which have the same heading:

Tracking: nixos-unstable's iso_minimal job for x86_64-linux.

https://r13y.com/iso_minimal/index.html
https://r13y.com/trivial/index.html

but they have different results.

Please clarify what is checked there.

update used diffoscope version

In this line:

r13y.com/check.sh

Line 2 in 5b3be0e

#!nix-shell -i bash ./default.nix -I nixpkgs=channel:nixos-19.09

I see that nixos-19.09 is used.

I've seen reports mentioning that diffoscope 110 is used, while diffoscope is at version 178 now. Is that due to that line or is that just a red herring?

I can imagine that a newer diffoscope version might give a better overview of where the issue lies with some packages, although it might not matter that much.

Anyhow, I think it would be good to use a somewhat newer version of diffoscope regardless.

Website not being updated automatically

This is a wonderful project. Thank you very much for it.

Currently it seems that the website is not being updated automatically. The footer still says "Generated at 2020-05-10 04:51:34.795219659 UTC", even though https://buildkite.com/grahamc/r13y-dot-com/ reports daily builds.

I'd like to offer my help, but don't really know where to start.

Design

The latest change (#22) added new pages, but no links from the front page.

@grahamc do you have a vision for a design where all information can easily be reached or would you like contributions?

An obvious idea would be a menu, another a single page design with different sections.

Also graphs with historic data would be nice.

GNOME check fails with: "No space left on device"

See i.e. https://buildkite.com/grahamc/r13y-dot-com/builds/790#d8af535c-aaf6-4b46-8def-36b7a647cc14