rust-lang / highfive Goto Github PK

PRs without S-waiting-on-review can easily fall out of reviewer's task list, and PRs marked as S-waiting-on-author is currently relabeled by hand.

We should add this functionality to highfive: when r? is used, unlabel S-waiting-on-author and relabel S-waiting-on-review.

We currently look for r? @foo in messages, but people often write r? foo. We should check for this and assign a reviewer for it (ignoring r? without a reviewer which is also common).

This will get some false positives, e.g., r? anyone, but I think that is OK, the attempt to set the assignee on GitHub will silently fail. In that case, we should then set a random reviewer (we should close this loophole in any case, I think).

cc @huonw

I did the first part of this - making choosing a reviewer generic over the repo.

Still todo:

• make rust-highfive an admin or owner or whatever on the repos we want to cover
• add the GitHub hooks to these repos
• change the link to contributing.md in the welcome message (maybe, perhaps the rust one is fine for all repos)
• create repo.json files for each repo with the list of people to use as reviewers
• test that it actually works!

E.g. everyone who commented here is cc'd here.

This should also be done for all issue that have already been moved.

It would be awesome if highfive, after choosing a reviewer from a team, assigned the T-team label of that team. This will allow scripts to easily see the assigned team, for example to build graphs.

I think the best way to do this is to remove the dirs field from the json files where we don't want to look at dirs. Then check for dirs for whether to inspect directories or not. Have to make sure newpr actually copes with this.

Currently the bot says hello and tells you that someone will be on the way:

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @huonw (or someone else) soon.

I think this is great, but it would be even better if it could elaborate on our standard process, and give more detailed advice for how to handle the process.

In particular, I've been burned a lot by people knowing that they should rebase changes for merge, but doing it before someone has given the final sign-off. This makes it really hard to tell if they actually have addressed all your concerns, and how.

I suggest it state the following, or something similar:

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @huonw (or someone else) soon.

If the reviewer has any concerns, you should address them by either making a change, or explaining why you think it's the right thing to do. (Or enter into a more detailed design discussion as necessary.)

If any changes are deemed necessary, please add them as extra commits to the current pull-request. This ensures that the reviewer can see what has changed since they last reviewed the code. The way Github handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Once the reviewer believes that all of their concerns have been addressed, they may ask you to rebase your commits to produce a more useful history for the changelog (for simple changes, this should just be a single commit). Do not do this until you are asked. Doing this too soon will make the reviewer's job much more difficult. Note that you may also be asked to rebase your work if new commits that land during the review process conflict with your changes.

When the change is approved and the history is where it should be, the reviewer will make a comment like r+ or r=huonw on the last commit to signal to our automated system that this commit is approved for inclusion in the project. Your PR will then be inserted into the Bors queue for testing. This queue is based on age; older PRs get priority. Once it reaches the top, your PR will be fully tested on top of the current master. If everything passes, it will be merged! If there is an error, your change will be rejected by the system. At this point you should investigate the logs the bot will post on the PR for what went wrong. It's entirely possible that the failure was spurious. The build might be unstable, or the process might have been interrupted by a system failure. If this is the case, a reviewer can prompt the system to retry.

Currently the people on core are eligible for all kinds of PRs. But really everybody has their specialty, so it'd be best to use core just as a fallback for PR assignment, and instead also add people to the teams where they have the most knowledge.

And don't pick one at random

As well as pinging a person with a mention, highfive should set the assignee of the PR to that person. This will require the highfive bot (rust-highfive) to have some extra permissions on the Rust repo.

Highfive could probably detect changes that mostly change lines that start with /// and set @steveklabnik as the reviewer.

and assign the reviewer

This project was originally a fork of jdm/highfive, used for Servo. I removed a few things, changed a few things, and since then we've added a lot. It would be nice if we could push some of the additions back to Servo land. Perhaps the easiest was to do that is to expand this version to cover the stuff removed or changed and then persuade them to switch (or just push everything back to jdm's original fork). I fear taking our changes and adding them to the original would be a tricky job.

cc @jdm

highfive should ping reviewers who forget about a review, probably by leaving a GitHub comment and by an irc ping.

I'm not sure exactly how this should work. We could run a cron job, say every hour and do an API request to get all open PRs with assignees, then find any with no comment from the assignee in the last 24 hours. We would also have to make an API request for comments on each commit of the PR to check for comments there. We would have to check for an r+ at any time, not just in the last 24 hours in order to avoid the situation where a patch is reviewed and we're waiting for bors. We should also not ping in the case where an older PR is r+'ed and a newer one has r+ or r=foo comment from someone other than the assignee. But we should ping if there is no comment at all.

Have I missed any requirements there?

I'm not actually sure I've ever gotten anything meaningful from this, and it's mostly just another email to have to sift through, perhaps we could just remove this for now?

If I use the GH feature to submit a review with Changes Requested, it would be great if that set the status from S-waiting-on-review to S-waiting-on-author.

We currently maintain a list of contributors per repo which supplements the list we pull from GitHub. We could be more efficient (albeit more stateful) by updating this list with contributors as we go along. We could either do this manually, or use a DB instead. See some discussion in #38

See rust-lang/rust#12463

We could do this ourselves fairly easily, though at the expense of adding some state. Or we could bounce off a dedicated irc bot.

The infra team is interested in helping maintain this piece of infrastructure, starting with introducing some debugging to figure out why some PRs are continuing to miss assignments. @aidanhs, in particular, has volunteered to help.

Are you open to moving this repo to the nursery?

We should explore the relationship to Servo's fork before making any move.

there are downsides to having a live fork also be a fork within GitHub because, for instance, it affects your ability to use GitHub search on the code:

https://github.com/rust-lang-nursery/highfive/search?q=reviewer&type=Code

however, it's possible to request that the repo be made a standalone repository:

To detach the fork and turn it into a standalone repository on GitHub, contact GitHub Support. If the fork has forks of its own, let support know if the forks should move with your repository into a new network or remain in the current network. For more information, see "About forks."

(from https://help.github.com/articles/why-are-my-contributions-not-showing-up-on-my-profile/#commit-was-made-in-a-fork)

I was able to follow this process to detach a fork with the following message:

Hi GitHub Support!

As detailed in https://help.github.com/articles/why-are-my-contributions-not-showing-up-on-my-profile/#commit-was-made-in-a-fork, please could you detach https://github.com/sbt/sbt-launcher-package from https://github.com/paulp/sbt-extras, turning it into a standalone repository. Note I co-maintain both, but they're distinct entities at this point.

Also, any fork of sbt/sbt-launcher-package should move with sbt/sbt-launcher-package into a new network, rather than remain in the current network.

Thank you,
Dale

rust-lang/rust#20761 is populated with checkboxes like below:

• Is this bug closed #44:

Highfive could check the boxes anytime the issues referenced are closed.

Highfive currently just responds to PRs from new contributors (it checks for unsafe code in all PRs, but that is separate to this issue). I propose that if a PR has r? @some_user, highfive change the assignee to some_user (note this will silently fail if some_user is not a valid assignee). If there is no proposed reviewer, Highfive will select one, ping them, and set the assignee. If the PR is from a new contributor, Highfive will use the welcome message.

E.g., don't match "...longer?"

Use the rustaceans.org db to find the irc nick, and then ping them

It would be nice if highfive could automatically attach labels to issues that clearly mention what they’re related to in their title. So an issue titled something like ‘rustdoc: foobars create broken links’ would automatically add an ‘A-rustdoc’ label to the issue.

I’m not sure if this is within the scope of highfive, but it’s still a nice thing that could be automated.

highfive is not applying labels to rust-lang/rust anymore, since a few days ago.

I am interested in investigating instances of unexpected Highfive behavior. For example, sometimes, but not consistently, Highfive sends the welcome message to existing contributors. Another example: in rust-lang/rust#49849, Highfive did not take any action at all. It's difficult to know what is going on there without access to the logs. Looking at Highfive's code I can speculate that maybe the diff request failed, but there's several other possibilities.

The first question is: are the organization members comfortable granting access to the logs to a non-member?

If the answer is yes, we should find a reasonable, low-cost-to-@nrc way to accomplish it. A lot of options come to mind, but they generally branch on whether we want to provide access to the host Highfive runs on or whether we want to ship the logs to somewhere else.

1. Provide access to host running Highfive.
   1. Continue running Highfive where it is now?
   2. Move Highfive to another location?
2. Ship the logs elsewhere.
   1. Add application logic to handle this?
   2. Use external process to ship logs to somewhere?

I'm happy to talk more about the possibilities, if the answer to the first question is yes.

• https://github.com/rust-lang/rust.vim
• https://github.com/rust-lang/rust-mode
• https://github.com/rust-lang/gedit-config
• https://github.com/rust-lang/kate-config
• https://github.com/rust-lang/nano-config
• https://github.com/rust-lang/zsh-config

(Filing an issue just so its not forgotten, since I don't have time to do it right now.)

E.g., if I tag an issue with T-compiler, then highfive should notice and add a comment cc'ing @rust-lang/compiler (as long as they are not already cc'ed).

Is this correct behavior? It seems incorrect, since landing code in any branch of a repo implies knowing how to contribute.

Example:

rust-www was formerly on GitHub pages, so its default branch is currently gh-pages. The graphs show contributions only to gh-pages. I got my first PR merged into master, and highfive welcomed me correctly. However, I was welcomed again on my second PR despite the fact that the first one landed.

from reading #35 and servo/highfive#13 I worked out that that both this fork and servo/highfive are "alive" and that this one is the what drives @rust-highfive.

I think this should be clarified somewhere reasonably prominently, like the README, maybe even the repo description.

cc @ben0x539

politely suggest something more descriptive than "typo"

rust-lang-nursery/highfive does not have a license. servo/highfive does: MPL, which was added a couple years after rust-lang-nursery/highfive forked from servo/highfive. Rust projects seem to like licensing with Apache v2 and MIT.

I disabled them since they seem to be generating a lot of false positives, e.g., rust-lang/rust#17727. Nice to have if we can make them work though

They don't work, not sure why.

Currently, if you r? a user who doesn't have the right to r+, rust-highfive will silently ignore it. This can be a little confusing: it'd be nicer if there was some indication of what went wrong.

A workaround would be to use per_page=10000 instead.

In rust-lang/rust we will not automatically merge diff from Cargo.lock (see rust-lang/rust#46539 for rationale). This means all PRs modifying Cargo.lock are mutually exclusive with each other, and when one such PR is merged, all other PRs will need to be rebased.

This may bother reviewers and authors when the PR needs to be rebased and r+'ed several times. It is also pretty confusing to new contributors because a rebase often completes without any conflicts locally.

Thus, I think it is the best for rust-highfive to foreshadow such potential extra work to both reviewers and authors.

⚠️ Warning ⚠️

This PR modified Cargo.lock. We will not automatically merge updates to Cargo.lock, since it often leaves the file in a broken state. One unfortunate result is that this often leads to merge conflicts.

You may need to rebase this PR several times after it has been approved. We are sorry for any inconvenience which may arise.

(Note to reviewers: Please also prepare to r+ several times. Consider giving it a p=1 to minimize queuing time and reduce chance of conflict.)

this repo is lacking a concrete description of what highfive does. I got here from http://huonw.github.io/blog/2015/03/rust-infrastructure-can-be-your-infrastructure/ so i have an idea, but i noticed that the repo does not stand on its own. (could even just link to that from readme for now)

Currently highfive will only respond to r? if it is in a comment by the author of the PR. We should expand this so that more people can do it. At least the currently assigned reviewer, perhaps also the 'all' group of reviewers, some union of reviewer groups, all possible GitHub reviewers (we'll probably want to store this somewhere rather than sending an API request every time), or an explicit group of people.

PR #136 seems to have broken Highfive on the server. The only relevant info in the logs is:

End of script output before headers: newpr.py

The web server responds with a 500 error

cc @davidalber

If I run the tests following the instructions in README.md, I get errors. First I get an indentation error at line 76 in test_cr.py. I tried fixing that by removing the initial spaces on that line. Then I get the error:

  File "/home/ncameron/highfive/test_cr.py", line 71, in <module>
    df = open('diff.diff')
IOError: [Errno 2] No such file or directory: 'diff.diff'

I don't see diff.diff anywhere, so I'm not sure how to fix this.

cc @flaper87

I would like to be able to aggressively set the status of PRs to S-waiting-on-author and then filter those out of my display. For me to feel comfortable doing that, though, I need to be sure that I won't miss updates from them -- therefore, I would like highfive to reset the status back to S-waiting-on-review as soon as a new commit or comment is made. (Possibly just a new comment by the PR author? But probably any comment is fine -- or any comment not by bors.)

For most repos, there's one branch that most PRs are expected to be filed against.

17:11:54 <@brson> this patch was submitted against 'stable' https://github.com/rust-lang/rust/pull/25739 but thankfully bors merged it into master
17:12:22 <@acrichto> brson: whoa
17:12:45 <@acrichto> brson: I wonder if rust-highfive could give a warning if it's not targeting master

The API contains information on which branch a PR was submitted against, in the form of "base"["label"]. For instance, grep rust-lang:stable in the result of curl -i https://api.github.com/repos/rust-lang/rust/pulls/25739.

I believe the architecture for this would be to add an expected_branch field in each repo's config, then add the check to highfive and default to assuming the expected branch is <reponame>:master if none is specified?

If this is something you'd like to get added, I can implement it.

e.g. rust-lang/rust#17508 got a comment but the author has previously submitted rust-lang/rust#17286

cc @huonw

My guess would be a mismatch between the Git and GitHub user info, but who knows?

Currently we just use a random member of the core team to ping. I would like to have a directory based system: look at the PR and find the directory with the most changes, keep a file mapping directory to people, select a person at random from the most changed directory's pool.

Since the mapping from directories -> people is tied to the Rust repo, it might be better to keep it in-tree. It is easier to have it here, for now.