rust-lang / rustc-perf

Website for graphing performance of rustc

Rust Compiler Performance Monitoring & Benchmarking

This repository contains two primary crates:

• collector: gathers data for each bors commit
• site: displays the data and provides a GitHub bot for on-demand benchmarking

Additional documentation on running and setting up the frontend and backend can be found in the README files in the collector and site directories.

Additional documentation on the benchmark programs can be found in the README file in the collector/compile-benchmarks and collector/runtime-benchmarks directories.

rustc-perf's Issues

Further minor-ish refactoring

See comments on #41

Push branches for testing

Be able to push a local branch to the repo to test its performance.

Display absolute times, not just percentages, on the front page

Percentages are useful but also kind of ... surprising. For example, a 1000% regression is "undone" by a 90% improvement. It'd be nice to see absolute times as well (also -- a percentage can look huge, but actually represent a small amount of time). It might be that we want to display this data only on hover or something.

some indication on the web page when crates don't build

There seem to be a number of issues about error-handling, but none of them with much description. Right now the set of benchmarks that appear on perf.rust-lang.org is much smaller than the full set that appears in https://github.com/rust-lang-nursery/rustc-benchmarks. I think the reason is that some of those benchmarks fail to build -- but there is no indication of this! Very easy to lose sight of a benchmark this way.

Investigate cause of empty files

These are the files I found to be empty:

regex.0.1.30--2016-06-21-04-21-09.json
issue-32062-equality-relations-complexity--2016-06-21-04-21-09.json
helloworld--2016-06-21-04-21-09.json
piston-image-0.3.11--2016-06-21-04-21-09.json
regex-macros.0.1.30--2016-06-21-04-21-09.json
inflate-0.1.0--2016-06-21-04-21-09.json
rust-encoding.0.2.32--2016-06-21-04-21-09.json
hyper.0.5.0--2016-06-21-04-21-09.json
jld-day15-parser--2016-06-21-04-21-09.json
issue-32278-big-array-of-strings--2016-06-21-04-21-09.json
rustc--2016-06-21-04-21-09.json
html5ever-2015-05-15--2016-06-21-04-21-09.json

Rewrite to split out the total meta-phase from the codebase

Not necessarily required, and may not even be a good idea, but a thought.

Rewrite the make_times function to return a Timings struct which will keep the total phase separate, since it is a meta-phase. This is a huge project since this affects most of the codebase; as well as causing problems with the iteration over phases code: it probably expects the total phase to exist.

Add proper logging

liblog has been brought up by @nrc.

decent error handling

Use a database instead of always loading from directory

From IRC:

@nrc:
Long-term, I think that we should have a permanent DB with the processed data in, so startup is just booting up the DB and we only process any dataset once
That means we need to see the diff in a commit, rather than read everything in
but that shouldn't be too hard

Loading from a diff/commit is also useful for the GitHub integration (since we could then fairly quickly just restart the server). cc #29.

Add units to graphs

Both http://perf.rust-lang.org/graphs.html and http://perf.rust-lang.org/mem.html lack units for the graphs.

Write a tool that auto-updates all of the integration tests given a set of changes

This is not required, but helps greatly when working on backend changes.

track -Z no-trans as well as full compile time

Since that is what is important for IDEs etc.

Data not collecting

The site only has data until about 5/17.

Display sub-pass timings

I think this has two fixes:

• (short term) to link to the raw logs in some manner for easy inspection
• (long term) modify the processed data to include the sub-passes

From IRC:

21:09 simulacrum: cc nrc: It looks like the new sub-passes added by nagisa recently aren't available for inspection on perf.rlo, is that because the scripts erase them? I suspect that's the case, but I'm not sure...
21:13 nrc: yeah, it ignores sub-passes, only handles top level passes
21:18 simulacrum: Hmm, some way to show those would be good
21:18 simulacrum: Since right now they aren't doing much good unless someone wants to drill down
21:18 simulacrum: but I can't think of an easy way to do it
21:18 simulacrum: I'll file an issue

comparison page doesn't produce pastable urls

When I submit the comparison page I stay on the http://perf.rust-lang.org/compare.html URL. I'd rather have the form data encoded in the URL so I can paste it into issues.

Update the html5ever benchmark

Which no longer seems to build

Automatic update upon Github webhook receival

• Add code to pull from GitHub
• Add a /on_push path to server.rs (unknown if GET or POST) which will update the state.
  • For now, we can reinitialize the entire state. Eventually, we'll want incremental updates.

Ability to run offline

I really want the ability to run these benchmarks (or some subset) offline. This would complement #1 -- basically I could check a local build, save its results, and compare them against master (again running locally).

Show when last update of data occurred

Include a link to the data repository as a [manual] way to check if there has been updates since when the page was loaded/the server believes the latest update happened.

runtime benchmarks

We're using this issue to track work on making benchmark suite for tracking efficiency of generated code (as opposed to efficiency of compilation time itself).

TODO: It might be good to copy over the list of items from description on rust-lang/rust#31265 . Or figure out some other way to encode them/curate them as a todo list for investigation points.

(original description follows)

---

It'd be great if there were an option to add a make bench target to makefiles that would run runtime benchmarks. We could scrape the output from cargo bench, for example, and include those as data points.

I could then merge the entries from https://github.com/nikomatsakis/rust-runtime-benchmarks and use those.

/get endpoint should use start_date and end_date instead of date array

I think the /get endpoint always passes the equivalent of the common start_date and end_date fields in an array. Requires changes to client side code.

Hyper Error

Log contained ERROR:hyper::server: request error = Version. Nothing more than that is known.

Expose the commit hash

some how.

Perhaps in the tool tip

Factor out title and summary HTML into one place

The overall decision is that moving towards generating shared HTML in shared.js and inserting it into the relevant page is the best approach.

---

Currently, the menu and the settings HTML is mostly duplicated from page to page. This makes editing it difficult; I propose that we move to generating the HTML from a template. This can either be done at-runtime in Rust, at-runtime in JS, or statically.

If we decide on at-runtime, I think doing it in JS makes more sense. Something like React or Vue.js may be worth looking into, though they may be overly complicated.

@dikaiosune: Can you provide any recommendations based on your work with the dashboard?

cc @nrc

Add rustfmt to CI

See http://johannh.me/blog/rustfmt-ci.html for details.

@nrc: Do we want this? If so, I can update the Travis config.

Documentation

The UI is not very intuitive, needs documenting

Move integration tests' expected outputs out into files

Edit: Integration and unit tests are now added for at least some of the functionality, main priority now is to move the integration tests to their own files. (This will ease checking that what we think changed changed).

---

Extracted conversation; primary point being that we'd like unit tests, but for them to be possible, modularization of the code is necessary.

---

Mark-Simulacrum:

I'd like to add tests, but I've been struggling to come up with good isolated portions of the code which can be tested. We can definitely do some form of overall diff-like testing though I think (create a InputData with predetermined state and call the server functions to generate output). That shouldn't be too difficult as the server functions don't depend on a request being passed, just the input data and [optionally] a body.

nrc:

Yeah, how we test this is a big question for me. I think in the medium term we should refactor to make portions testable, one piece at a time, and add unit tests - test coverage is a big goal for me for the rewrite.
In the short-term, that is probably too much to do. Some diff testing with real data from the timing repo would be good to do, if it is not too complex.

Isolate the root URL for fetches

So we can change between the Rust and JS backends by changing one string rather than 7 in 7 different files.

Summary page inconsistency between Rust and JS versions

#31 still doesn't bring exact compatibility with the JS in terms of the results of the summary page.

I'm not sure why this is (I've tried changing the percent logic to what I thought it was before, and a few more things), but nothing seems to bring us back even close to the old JS numbers.

Also, comparing the JS results and the Rust results makes me think a few more of these bugs are hiding from us: while the graph of compile times has dropped significantly, the summary page from the Rust version insists that compile times have gone up by ~13% last I checked while the JS page displays a ~55% drop (which feels more realistic to me when looking at the graphs).

Off by one errors with dates

When you submit a date, you end up with data for the next date (in most views, I think).

breakdown of regression data by phase

for the summary page, show the regression data for each phase, not just a total

Make it easy to go from front page to detailed info

When you see a regression, it'd be great to be able to click on the test name and see more detailed data for just that test -- for example, the graph for total time on that benchmark and also the graphs for all individual compiler passes on that benchmark. This would probably be a simple step that would address #56 in an easier way. That is, if we could just click on the regression, we'd easily see if one particular compiler pass seems to have changed dramatically.

Remove group by selector

The group by selector can be replaced with some JS logic that groups by phase if any phases other than total/maximum are selected, otherwise grouping by crate.

Meaningful URLs

So you can copy and paste a link to a specific graph

Fix CI

Currently can't run Rustfmt, which means all tests are failing.

Stop merging multiple runs with the same datetime into one

When loading runs from disk, we can find more than one run has occurred on a given date. When this happens, we ideally want to keep all of the data. I propose that we continue to merge data, but instead of doing so by date, we should do so by the associated commit. The commit is what gives each run "uniqueness"; the date is associated metadata. We should still sort both data_ vectors by the date, in order to ease display/manipulation and allow binary search functionality.

See #50 and #28 for previous discussion.

Align table cells with numbers to the right

Requested on IRC, and generally a good idea.

lqd: simulacrum nrc: numbers in table are generally aligned right rather than centered, to allow for easier vertical scanning and comparisons between rows :)

Error handling on the backend

Potentially should be merged with #6.

Currently, we use error-chain, but we don't use it's full potential. Do we need error-chain?

Show which line is which phase/crate in the graph views

Perhaps by sorting the "current" one to the top and bolding it in the tooltip.

Unknown intended purpose of overwriting, data loss

I'm not sure what the original intended purpose of the code involving c_benchmarks_add in load.rs was. However, I noticed that this means we overwrite existing data in these cases. None of these are recent, so it may not be too big a deal, but wanted to make sure we were aware of this, since it might lead to data loss in the future as well.

hyper.0.5.0 in hyper.0.5.0--2015-05-25-14-31-35.json
helloworld in helloworld--2015-05-25-14-31-35.json
html5ever-2015-05-15 in html5ever-2015-05-15--2015-05-25-14-31-35.json
regex.0.1.30 in regex.0.1.30--2015-05-25-14-31-35.json
rust-encoding.0.2.32 in rust-encoding.0.2.32--2015-05-25-15-45-57.json
regex-macros.0.1.30 in regex-macros.0.1.30--2015-05-25-14-31-35.json

@nrc did not recall either:

Hmm, I don't recall either. I wonder if it is meant to coalesce data rather than overwrite it?
Add a line note

Potentially incorrect summary page median functionality

I believe the intent with the summarise_data function in the backend was to compute median times in the three weeks around the last week in the pulled data.

However, the current code doesn't do this, instead using only one week (probably the previous week from the last week being examined, if I'm reading the code correctly).

Is this intended?

(I noticed this while trying to understand what is happening in the current code while working on my Rust version of the code).

Use structs for output data rather than ObjectBuilder

More static checking ftw

Start date computed 30, not 60 days before end date

Comment here states that a missing start date is computed as 60 days before the end date, but the code subtracts 30, not 60.

Auto-deploy website

• Build and copy the built binary from Travis to the server
  • After copying, restart the server. Momentary (<1 minute) downtime is probably fine.
• git pull on the server when the repository is updated to keep the JS/HTML/CSS up-to-date.

Investigate switching to a Mozilla graphing library

This https://github.com/mozilla/metrics-graphics library by mozilla looks well maintained and relatively good looking.

Marking as low priority though since we're mostly happy with HighCharts (though @nrc did mention that it looks worse in his opinion).

Track down a good way of parsing/serializing dates passed from/to the frontend

Chrono struggles with parsing semi-arbitrary data.

Extracted comment:

Also, I've been looking at this existing URL pointing at perf.rust-lang.org, and upon testing it with the current code, things break. I don't know of a good way to "best effort parse this date we're throwing at you" in Rust (and even in JS, truth be told). Without that, we'll either need to try and list all the cases we can think of (attempting parsing with each, failing, and moving on to the next one), or think of something else. Perhaps input sanitization? Regex-based search and recombine the input into something more readily parseable?

Let me know if you think we shouldn't care about backwards compatibility with URLs (I think we should). I'm also not 100% certain how to generate the URL in that issue, my attempts don't seem to produce the HH:MM:SS GMT+0000 (UTC) part.

Error(Msg("while parsing Mon Jun 13 2016 19:51:42 GMT+0000 (UTC)"), (Some(ParseError(TooLong)), stack backtrace:
...

Expose SHA1 hashes

Currently, if you hover on a data point, you get the date and time, but not the SHA1 hash of the commit. The SHA1 hash would be much more precise, and make it easy to determine if (e.g.) a given change is also on beta. For bonus points, it could be linked to the "diff" on GH versus the last bullet point, so one can easily browse the set of changes that took place in between.

Graph display is misleading (8% regression is seen as much larger)

From IRC: https://botbot.me/mozilla/rustc/2016-08-18/?msg=71561193&page=3

compare: error if a benchmark stops building

If a benchmark builds in build a, but not build b then we get a fatal JS error.

Change to calculation of compile time percent increase/decrease

Currently, ((previous - current) / current) * 100.0 is the calculation used to derive the percent change between the previous median time and the current median time for the summary page. This page on Wikipedia implies that the formula should instead be ((current - previous) / previous) * 100.0.

Wanted to open this for discussion since this change the results in my current branch from 11.4% increase in compile times to a 9.4% decrease.