ezyang / ghstack

Submit stacked diffs to GitHub on the command line

ghstack

Conveniently submit stacks of diffs to GitHub as separate pull requests.

pip3 install ghstack

ghstack is tested with several different Python versions. It requires at least Python 3.8.1.

How to setup

Go to github.com Settings→Developer Settings→Personal Access Tokens and generate a token with public_repo access only. Create a ~/.ghstackrc as shown below:

λ cat ~/.ghstackrc
[ghstack]
github_url = github.com
github_oauth = [your_own_token]
github_username = [your_username]
remote_name = upstream [if remote is called upstream and not origin]

How to use

Make sure you have write permission to the repo you're opening PR with.

Prepare a series of commits on top of master, then run ghstack. This tool will push and create pull requests for each commit on the stack.

How do I stack another PR on top of an existing one? Assuming you've checked out the latest commit from the existing PR, just git commit a new commit on top, and then run ghstack.

How do I modify a PR? Just edit the commit in question, and then run ghstack again. If the commit is at the top of your stack, you can edit it with git commit --amend; otherwise, you'll have to use git rebase -i to edit the commit directly.

How do I rebase? The obvious way: git rebase origin/master. Don't do a git merge; ghstack will throw a hissy fit if you do that. (There's also a more fundamental reason why this won't work: since each commit is a separate PR, you have to resolve conflicts in each PR, not just for the entire stack.)

How do I start a new feature? Just checkout master on a new branch, and start working on a fresh branch.

WARNING. You will NOT be able to merge these commits using the normal GitHub UI, as their branch bases won't be master. Use ghstack land $PR_URL to land a ghstack'ed pull request.

Structure of submitted pull requests

Every commit in your local commit stack gets submitted into a separate pull request and pushes commits onto three branches:

• gh/username/1/base - think of this like "master": it's the base branch that your commit was based upon. It is never force pushed; whenever you rebase your local stack, we add merge commits on top of base from the true upstream master.

• gh/username/1/head - this branch is your change, on top of the base branch. Like base, it is never force pushed. We open a pull request on this branch, requesting to merge into base.

• gh/username/1/orig - this is the actual commit as per your local copy. GitHub pull requests never sees this commit, but if you want to get a "clean" commit all by itself, for example, because you want to work on the commits from another machine, this is the best way to get it.

Developer notes

This project uses Poetry, so after you've installed Poetry itself, run this command in your clone of this repo to install all the dependencies you need for working on ghstack:

poetry install

Note that this installs the dependencies (and ghstack itself) in an isolated Python virtual environment rather than globally. If your cwd is in your clone of this repo then you can run your locally-built ghstack using poetry run ghstack $ARGS, but if you want to run it from somewhere else, you probably want poetry shell instead:

poetry shell
cd $SOMEWHERE
ghstack $ARGS

Testing

We have tests, using a mock GitHub GraphQL server! How cool is that?

poetry run python test_ghstack.py

That runs most of the tests; you can run all tests (including lints) like this:

poetry run python run_tests.py

Publishing

You can also use Poetry to publish to a package repository. For instance, if you've configured your Poetry repositories like this:

poetry config repositories.testpypi https://test.pypi.org/legacy/

Then you can publish to TestPyPI like this:

poetry publish --build --repository testpypi

To publish to PyPI itself, just omit the --repository argument.

Design constraints

There are some weird aspects about GitHub's design which lead to unusual design decisions on this tool.

1. When you create a PR on GitHub, it is ALWAYS created on the repository that the base branch exists on. Thus, we MUST push branches to the upstream repository that you want PRs to be created on. This can result in a lot of stale branches hanging around; you'll need to setup some other mechanism for pruning these branches.

2. Branch name does not correspond to pull request number. While this would be excellent, we have no way of reserving a pull request number, so we have no idea what it's going to be until we open the pull request, but we can't open the pull request without a branch.

Ripley Cupboard

Channeling Conor McBride, this section documents mistakes worth mentioning.

Non-stack mode. ghstack processes your entire stack when it uploads updates, but it doesn't have to be that way; you could imagine that you could ask ghstack to only process the topmost commit and leave the rest alone. An easy and attractive looking way of doing this is to edit the stack selection algorithm to look a single commit, rather than all the commits from merge-base to head.

This sounds OK but you try it and you realize two things:

1. This is wrong, if you exclude the commits before your commit you'll end up with a base commit based on the "literal" commit in your Git repository. But this has no relationship with the base commit that was previously uploaded, which was synthetically constructed.

2. You also have do extra work to pull out an up to date stack to write into the pull request body.

So, this is not impossible to do, but it will need some work. You have to work out what the real base commit is, whether or not you need to advance it, and also rewrite the stack rendering code.

ghstack's Issues

`ghstack cherry-pick PR_URL`

this is pretty easy to do by hand (just cherry-pick the orig commit) but would be nice to have some sugar

Feature request: stack on an already submitted PR

Sometimes I realize I should use stacked diff only after I have already submitted one. It would be good if I can still do a stacked diff, but the base is on a submitted PR.

GPG signed commits

Not sure if these is a local config I can enable for this. I can't find any thing that enables signing on commit-tree automatically.

If there isn't a git config, I see two ways of adding this feature:

1. add a config line for commit signing and optionally allow keyid to be passed in from that config line. i.e.

None   => don't sign
""     => sign with default key
keyid  => sign with key id

2. check git config --list and use the key and behavior in there.

Update PR description when rewording a commit on the stack

Thanks for writing ghstack! After using Phabricator at $OLDJOB, it's interesting to try out a Phabricator-like workflow in GitHub.

I tried out ghstack with this workflow:

• Prepare a stack of commits C1, C2, C3
• Run ghstack, open PRs for each commit
• Get a review on C2, make a commit C2_fixup, which changes C2 enough that I have to reword C2. C1 and C3 are approved.
• Rerun ghstack

At this point, the generated PR corresponding to C2 has an outdated description that doesn't match the wording of C2. I wonder if it might be useful to at least reword the PR title to correspond to the updated shortlog(?) message.

I can work towards a PR for this, filing the issue mainly to get thoughts if any. It's possible I'm missing some reason why the PR shouldn't or can't be re-titled.

Remove explicit references to 'master' in the codebase

Fail if multiple commits have the same `ghstack-source-id`

Description of the problem

I made a mistake on rebasing. Originally it was:

[my commit 3, ghstack-source-id: <3>]
[my commit 2, ghstack-source-id: <2>]
[my commit 1, ghstack-source-id: <1>]
[.... commits from origin/master ...]

After rebase, I get

[my commit 4, ghstack-source-id: <1>]
[my commit 3, ghstack-source-id: <3>]
[my commit 2, ghstack-source-id: <2>]
[my commit 1, ghstack-source-id: <1>]
[.... commits from origin/master ...]

where commit 4 is just a modification of commit 1 but it has the same ghstack-source-id and Pull Request resolved in the commit message as commit 1. When I run ghstack, I find that the stack info is modified like:

And the PR is merged. I fixed the problem to modify my local repo as

[my commit 3, ghstack-source-id: <3>]
[my commit 2, ghstack-source-id: <2>]
[my commit 4, ghstack-source-id: <1>]
[.... commits from origin/master ...]

and run ghstack again, I get errors. and after running ghstack --force all PRs are merged.

See:
pytorch/pytorch#28343

Expected behavior

ghstack should check whether ghstack-source-id is unique before taking any action.

What does this stack trace mean? How do I fix?

Support `ghstack REFSPEC`

Something I find myself doing frequently is I have some slush commit with some local changes that are not ready for committing, and then a stack under it with the real stuff. To submit ghstack, I keep having to git rebase -i edit the commit I want to send, and then run ghstack there. Supporting ghstack REFSPEC meaning "submit the ghstack starting at this commit" would be helpful.

Support non-default base branch

I was reading the docs and was curious as to whether creating stacks off of non-default base branches was possible. I couldn't find an answer there, so I looked at the code and it seems like not. Would it be possible to make the base branch configurable?

As some potential inspiration, the GitHub CLI supports this in the following way:

gh pr create -B my_base_branch

Add support for submitting partial stacks

Dear @ezyang,

one of the beautiful things of ghstack is that I don't need to handle so many branches for a stacked PR workflow anymore.
However, I usually have part of the stack I'm working on ready for review and another part (the tail) that is still WIP, where I don't want to open PRs yet, because I don't want to commit (no pun intended) to a final look of my chain of commits that are yet to come and delete them without poluting github with many PRs.

As far as I know ghstack submits/updates from the HEAD up.
I now see that submitting a partial stack is as easy as checking out the respective most-downstream commit and running ghstack from this detached HEAD state. This is however not perfect because next I need to rebase my working branch on top of the detached HEAD (which has a new commit ID now!) etc.

I came up with a porcelain bash function ghpartial <commit sha> that takes care of what would ideally happen if ghstack <commit sha> would be run (in my humble opinion :) )

ghpartial () {
	BRANCH=$(git rev-parse --abbrev-ref HEAD)
	SUBMIT_BRANCH=$BRANCH-submit
	SUBMIT_COMMIT=$1
	DIRTY=false
	if [[ ! -z $(git status -s) ]]
	then
		echo "stashing uncommited changes..."
		DIRTY=true
		git stash -u
	fi
	if ! git show-ref --quiet refs/heads/$SUBMIT_BRANCH
	then
		echo "creating submit branch"
		git checkout -b $SUBMIT_BRANCH
	fi
	git checkout $SUBMIT_BRANCH
	git reset --hard $SUBMIT_COMMIT
	ghstack
	git checkout $BRANCH
	git rebase $SUBMIT_BRANCH
	if [ "$DIRTY" = true ]
	then
		git stash pop
	fi
	echo "All done. continue chipping away..."
}

(This keeps a branch around that shows how much of the work was submitted already, which is nice for my personal needs because I rebase the current working branch a lot and then it can be handy to have this marker in a prettyfied git log --graph --full-history view)

Would you like a PR for adding such an argument to ghstack natively? Maybe you have ideas how you would like it to be implemented?

Anyway, this tool is just a blaze to use and I convinced half of the people on my team to switch over after presenting it to them recently! Incredibly useful work ❤️

Best,
Zacharias

Make locally-built ghstack easier to use

After the migration to Poetry (#32), it's a bit more difficult to use ghstack (e.g. for experimentation, to check or validate behaviors) while locally developing it. The solution given in the README is to use poetry shell, but that is a bit unergonomic because it requires one to cd into their clone of ghstack, run poetry shell, then cd out again.

One alternative could be to put a shell script like this on your PATH (assuming your clone of ghstack is in ~/github/ezyang/ghstack):

#!/usr/bin/env bash
$(cd ~/github/ezyang/ghstack && poetry run which ghstack) "$@"

The downside of this is of course that it has a hardcoded path. Maybe it would be worth it to add a Makefile to this repo such that make install automatically puts such a file on the PATH, using the actual path to the repo clone for the cd subshell command.

Hitting github rate limits

Recently I've been unable to use ghstack effectively because I keep hitting github rate limits.

I ran ghstack a couple times this morning and eventually ran into:

RuntimeError: {
 "message": "API rate limit exceeded for user ID xxx.",
 "documentation_url": "https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"
}

So I used the rate limit API to check when it would reset and then tried again and ran into the same issue after a single ghstack.

I haven't looked into the source code but it does seem like this tool is hitting the API a lot and maybe there's some optimization that could be done to either limit the number of API queries or change the authentication strategy to increase the rate limit

PR unexpectedly merged when commits are reordered

I had a stack of 4 PRs, like this:

1. PR A
2. PR B
3. PR C
4. PR D

PR A was taking a while to review, so I rebased my stack to put it at the end:

1. PR B
2. PR C
3. PR D
4. PR A

But for some reason, after I ran ghstack, PR A got merged (!)

At first I thought I misclicked, but this is the second time it's happened to me.

Log

$ git remote get-url origin
$ git fetch origin
$ git merge-base origin/dev HEAD
$ git rev-list --header ^2f730d04bea150a5ac4cdd00a9903b7206d1a523 HEAD
$ git rev-list --header ^2f730d04bea150a5ac4cdd00a9903b7206d1a523^@ 2f730d04bea150a5ac4cdd00a9903b7206d1a523
$ git rev-list --max-count=1 --header origin/gh/uycire/23/orig
Pushing to #15173
$ git rev-parse origin/gh/uycire/23/base
$ git merge-base --is-ancestor origin/gh/uycire/23/base 2f730d04bea150a5ac4cdd00a9903b7206d1a523
$ git commit-tree f3870cc051da430fbb658597890e5f5af73d2aee -p origin/gh/uycire/23/base -p 2f730d04bea150a5ac4cdd00a9903b7206d1a523
$ git rev-parse c1614293dbd668e61ac7bb15bc34ded9187d1f94~^{tree}
$ git commit-tree dbfadf306f49558ed6eb704ec9bdd285a5971d55 -p origin/gh/uycire/23/head -p 58aba48019cbf29bce99550d02324eefe6a6be1d
Restacking commit on 2f730d04bea150a5ac4cdd00a9903b7206d1a523
$ git commit-tree dbfadf306f49558ed6eb704ec9bdd285a5971d55 -p 2f730d04bea150a5ac4cdd00a9903b7206d1a523
$ git rev-list --max-count=1 --header origin/gh/uycire/24/orig
Pushing to #15174
$ git rev-parse origin/gh/uycire/24/base
$ git merge-base --is-ancestor origin/gh/uycire/24/base 1179b46a16042a70e8ebf841278e395acf5ef622
$ git rev-parse b6e9506a067ba95070a4fd373299c959c68cdf14~^{tree}
$ git commit-tree fcd7930234022d4f51d98b5e2ae2df96d9e0786f -p origin/gh/uycire/24/head -p 1179b46a16042a70e8ebf841278e395acf5ef622
Restacking commit on ee28d0e1ab0ae29c708224a48bde2ed40d639684
$ git commit-tree fcd7930234022d4f51d98b5e2ae2df96d9e0786f -p ee28d0e1ab0ae29c708224a48bde2ed40d639684
$ git rev-list --max-count=1 --header origin/gh/uycire/34/orig
Pushing to #15214
$ git rev-parse origin/gh/uycire/34/base
$ git merge-base --is-ancestor origin/gh/uycire/34/base bed81715a6983c25786741de018a111e36a5e901
$ git rev-parse e0858fea3333550cd06a601ce42ace508d25daa2~^{tree}
$ git commit-tree dafe580de6588db31ac4b393531ad5f4a4b6550b -p origin/gh/uycire/34/head -p bed81715a6983c25786741de018a111e36a5e901
Restacking commit on 462d714e2b4e358db54715c1101b9d466ae19f5d
$ git commit-tree dafe580de6588db31ac4b393531ad5f4a4b6550b -p 462d714e2b4e358db54715c1101b9d466ae19f5d
$ git rev-list --max-count=1 --header origin/gh/uycire/22/orig
Pushing to #15167
$ git rev-parse origin/gh/uycire/22/base
$ git merge-base --is-ancestor origin/gh/uycire/22/base 93ea4d8d70e5310ded9411bab98213ed798377a4
$ git rev-parse e0f74da1612150f476eb54d9cc0417ab14dfd421~^{tree}
$ git commit-tree 2bab0f9f6aebf106b6ffa73f847f447e38bb8220 -p origin/gh/uycire/22/head -p 93ea4d8d70e5310ded9411bab98213ed798377a4
Restacking commit on cb0471ac2de373b973f13becd25affcbf0d6eb03
$ git commit-tree 2bab0f9f6aebf106b6ffa73f847f447e38bb8220 -p cb0471ac2de373b973f13becd25affcbf0d6eb03
$ git reset --soft 3ee46301fe403da8c7258003fcd18ae61d8c8155
# Updating https://github.com/benchling/aurelia/pull/15173
# Updating https://github.com/benchling/aurelia/pull/15174
# Updating https://github.com/benchling/aurelia/pull/15214
# Updating https://github.com/benchling/aurelia/pull/15167
$ git push origin 58aba48019cbf29bce99550d02324eefe6a6be1d:refs/heads/gh/uycire/23/base 1179b46a16042a70e8ebf841278e395acf5ef622:refs/heads/gh/uycir
e/24/base bed81715a6983c25786741de018a111e36a5e901:refs/heads/gh/uycire/34/base 93ea4d8d70e5310ded9411bab98213ed798377a4:refs/heads/gh/uycire/22/bas
e
To github.com:benchling/aurelia.git
   6482da57fb..93ea4d8d70  93ea4d8d70e5310ded9411bab98213ed798377a4 -> gh/uycire/22/base
   f2a04320da..58aba48019  58aba48019cbf29bce99550d02324eefe6a6be1d -> gh/uycire/23/base
   616f8d5ef0..1179b46a16  1179b46a16042a70e8ebf841278e395acf5ef622 -> gh/uycire/24/base
   ddcc776f27..bed81715a6  bed81715a6983c25786741de018a111e36a5e901 -> gh/uycire/34/base
$ git push origin 1179b46a16042a70e8ebf841278e395acf5ef622:refs/heads/gh/uycire/23/head bed81715a6983c25786741de018a111e36a5e901:refs/heads/gh/uycir
e/24/head 93ea4d8d70e5310ded9411bab98213ed798377a4:refs/heads/gh/uycire/34/head eff85617349d7019a1755f2dbcc38ada0a410f72:refs/heads/gh/uycire/22/hea
d
To github.com:benchling/aurelia.git
 ! [remote rejected]       eff85617349d7019a1755f2dbcc38ada0a410f72 -> gh/uycire/22/head (cannot lock ref 'refs/heads/gh/uycire/22/head': unable to
resolve reference 'refs/heads/gh/uycire/22/head')
 ! [remote rejected]       1179b46a16042a70e8ebf841278e395acf5ef622 -> gh/uycire/23/head (failed)
 ! [remote rejected]       bed81715a6983c25786741de018a111e36a5e901 -> gh/uycire/24/head (failed)
 ! [remote rejected]       93ea4d8d70e5310ded9411bab98213ed798377a4 -> gh/uycire/34/head (failed)
error: failed to push some refs to '[email protected]:benchling/aurelia.git'
ERROR: Fatal exception
Traceback (most recent call last):
  File "/home/eric/.pyenv/versions/3.8.1/lib/python3.8/site-packages/ghstack/logging.py", line 108, in manager
    yield
  File "/home/eric/.pyenv/versions/3.8.1/lib/python3.8/site-packages/ghstack/__main__.py", line 107, in main
    ghstack.submit.main(
  File "/home/eric/.pyenv/versions/3.8.1/lib/python3.8/site-packages/ghstack/submit.py", line 213, in main
    submitter.push_updates()
  File "/home/eric/.pyenv/versions/3.8.1/lib/python3.8/site-packages/ghstack/submit.py", line 985, in push_updates
    self.sh.git("push", self.remote_name, *push_branches)
  File "/home/eric/.pyenv/versions/3.8.1/lib/python3.8/site-packages/ghstack/shell.py", line 272, in git
    return self._maybe_rstrip(self.sh(*(("git",) + args), **kwargs))
  File "/home/eric/.pyenv/versions/3.8.1/lib/python3.8/site-packages/ghstack/shell.py", line 211, in sh
    raise RuntimeError(
RuntimeError: git push origin 1179b46a16042a70e8ebf841278e395acf5ef622:refs/heads/gh/uycire/23/head bed81715a6983c25786741de018a111e36a5e901:refs/h$ads/gh/uycire/24/head 93ea4d8d70e5310ded9411bab98213ed798377a4:refs/heads/gh/uycire/34/head eff85617349d7019a1755f2dbcc38ada0a410f72:refs/heads/gh/$ycire/22/head failed with exit code 1

merge commits using the normal GitHub UI or provide a CLI

From the README:

You will NOT be able to merge these commits using the normal GitHub UI, as their branch bases won't be master. For the PyTorch repository, we have a special mechanism for landing diffs; if you need a way to land these commits on a regular GitHub repository, give a holler on issues and we'll add this functionality.

Stacking diffs in Phabricator is awesome, would be great to have this feature in open source repos as well.

ghstack land (no arguments) should land the stack at HEAD

Right now PR URL is mandatory but it should work if you're checked out to an appropriate stack.

If a PR in your stack was merged and its orig branch was deleted, do something better

Right now, you get an error like this:

$ git rev-list --max-count=1 --header origin/gh/ezyang/1084/orig        
fatal: ambiguous argument 'origin/gh/ezyang/1084/orig': unknown revision or path not in the working tree.

which sux

ghstack produced weird PR description when the top PR in the stack starts with a bulleted list

It's really confusing here that the bulleted list transitions from a list of PRs in the stack to the list that was in the original PR description:
pytorch/pytorch#60249

Some formatting that makes it clear where the original PR description begins would be helpful in this csae.

Make ghstack land work with protected master branch

ghstack land directly pushes to master, which means that if master is protected, you can't use it. It would be nice to figure out some way to work around this.

One silly way to do it is to open a NEW pr that is based on master, and then use GitHub API to merge it. However, if you have mandatory checks on a project, you have to wait for all the checks to finish before you can merge. Another possibility is to get a bot with push to protected rights and teach ghstack to ask that bot to do the merge.

cc @fmassa

Make commit messages more descriptive

Currently ghstack is wont to produce long strings of fairly unhelpful identical commit messages, usually just "Update on ". Example:

• https://github.com/pytorch/pytorch/pull/54924/commits

Since the goal is for these sorts of PRs to eventually be squashed and merged into a single commit on the default branch, it would be nice (if possible) for ghstack to give more descriptive commit messages for these updates, such as "rebased onto commit x".

creating a new commit and making it the first commit in the stack has bugs

A, B, C, D are commits

• master branch -> A -> B -> C
• ghstack invoked, and PRs created for A, B, C
• commit D created, and rebased so that master branch -> D -> A -> B -> C
• ghstack invoked

in this case, the base branch of A is not moved to be D.

version ghstack v0.2.1

Tried updating to ghstack v0.3.0 and running ghstack again, but it didn't do anything

Feature Request: Provide an interface to access different iterations of the code in a ghstack pull request

I would like to have an interface to access different iterations of the code in a ghstack pull request.

ghstack said Fatal: unable to access 'https://github.com/pytorch/pytorch.git/'

I want to use ghstack to pull a series of commits to PyTorch as the following step:

1. git clone https://github.com/pytorch/pytorch.git
2. prepare my commits
3. run ghstack

but I get the following error:

From https://github.com/pytorch/pytorch
   79d47c1..7e55494  master     -> origin/master
 + 272f3a2...b91ecc0 true_division -> origin/true_division  (forced update)
$ git merge-base origin/master HEAD
$ git rev-list --header ^79d47c1c5ff5306bdd275196b7171c04bebbdcca HEAD
$ git rev-list --header ^79d47c1c5ff5306bdd275196b7171c04bebbdcca^@ 79d47c1c5ff5306bdd275196b7171c04bebbdcca
$ git for-each-ref refs/remotes/origin/gh/xiaobingsuper --format=%(refname)
$ git rev-parse f7bc54125a83ac499570e7dcf7863b4ba613d544~^{tree}
$ git commit-tree 44e36f68d40a59b55c43ea9bba7048ca6749dd8e -p 79d47c1c5ff5306bdd275196b7171c04bebbdcca
$ git push origin b8904886ab258623abe263b9bd57c1168270e0c6:refs/heads/gh/xiaobingsuper/1/head 79d47c1c5ff5306bdd275196b7171c04bebbdcca:refs/heads/gh/xiaobingsuper/1/base
Username for 'https://github.com': xiaobingsuper
Password for 'https://[email protected]': 
remote: Permission to pytorch/pytorch.git denied to XiaobingSuper.
fatal: unable to access 'https://github.com/pytorch/pytorch.git/': The requested URL returned error: 403
ERROR: Fatal exception
Traceback (most recent call last):
  File "/home/xiaobing/anaconda3/envs/pytorch-test/lib/python3.7/site-packages/ghstack/logging.py", line 108, in manager
    yield
  File "/home/xiaobing/anaconda3/envs/pytorch-test/lib/python3.7/site-packages/ghstack/__main__.py", line 113, in main
    github_url=conf.github_url,
  File "/home/xiaobing/anaconda3/envs/pytorch-test/lib/python3.7/site-packages/ghstack/submit.py", line 206, in main
    submitter.prepare_updates()
  File "/home/xiaobing/anaconda3/envs/pytorch-test/lib/python3.7/site-packages/ghstack/submit.py", line 896, in prepare_updates
    self.process_new_commit(s)
  File "/home/xiaobing/anaconda3/envs/pytorch-test/lib/python3.7/site-packages/ghstack/submit.py", line 584, in process_new_commit
    *new_branches,
  File "/home/xiaobing/anaconda3/envs/pytorch-test/lib/python3.7/site-packages/ghstack/shell.py", line 271, in git
    return self._maybe_rstrip(self.sh(*(("git",) + args), **kwargs))
  File "/home/xiaobing/anaconda3/envs/pytorch-test/lib/python3.7/site-packages/ghstack/shell.py", line 212, in sh
    .format(' '.join(args), returncode)
RuntimeError: git push origin b8904886ab258623abe263b9bd57c1168270e0c6:refs/heads/gh/xiaobingsuper/1/head 79d47c1c5ff5306bdd275196b7171c04bebbdcca:refs/heads/gh/xiaobingsuper/1/base failed with exit code 128

it seems that I need apply a permission of PyTorch repro, perhaps I miss something. Thanks!

ghstack possibly modifying commit author

Repro steps

• Did a ghstack checkout for PR from another contributor
• Added some changes and amended the commit (which should not change the author by default). git log still showed original author at this point.
• Ran ghstack, after this, git log showed me as the author. The commit author also changed to me.
  Example where this happened

Landing all (appended) commits in PRs

If a reviewer appends commits to PRs (say, via the Github UI), is there a way to have ghstack land (and ghstack checkout) to not drop these? Thanks!

Permission denied to create PR via ghstack

I'm trying to use ghstack to stack my PRs. But I encountered the "permission denied" error. The details is as follows. I checked with Meta internally and got the response that I'm on the external contributor list and should have the ghstack privilege. Hence, I raise this issue here and check what the next step is. I notice that Make sure you have write permission to the repo you're opening PR with. in the README of ghstack repo. So does that mean I do not have the write permission?

version: 0.6.0
command: /home/eikan/dev/anaconda3/envs/pyt-te/bin/ghstack
status: 3f6b1ab9d "Test ghstack"

config_path = /home/eikan/.ghstackrc
conf = Config(proxy=None, github_oauth='<GITHUB_OAUTH>', github_username='EikanWang', circle_token=None, fbsource_path='/home/eikan/local/fbsource', github_path='/home/eikan/local/ghstack-pytorch', default_project_dir='fbcode/caffe2', github_url='github.com', remote_name='origin')
$ git remote get-url origin
Using selector: EpollSelector
https://github.com/pytorch/pytorch.git

# POST https://api.github.com/graphql
Request GraphQL query:

        query ($owner: String!, $name: String!) {
            repository(name: $name, owner: $owner) {
                id
                isFork
                defaultBranchRef {
                    name
                }
            }
        }
Request GraphQL variables:
{
 "owner": "pytorch",
 "name": "pytorch"
}
Starting new HTTPS connection (1): api.github.com:443
https://api.github.com:443 "POST /graphql HTTP/1.1" 200 None
Response status: 200
Response JSON:
{
 "data": {
  "repository": {
   "id": "MDEwOlJlcG9zaXRvcnk2NTYwMDk3NQ==",
   "isFork": false,
   "defaultBranchRef": {
    "name": "master"
   }
  }
 }
}
$ git fetch --prune origin
# stderr:
From https://github.com/pytorch/pytorch
 + c7e42f55f4...983664876a functionality/fsdp_commhook_interface -> origin/functionality/fsdp_commhook_interface  (forced update)
   ec4be38ba9..61305cd638  viable/strict -> origin/viable/strict

$ git merge-base origin/master HEAD
2148e6b4a4c51f84ffc406089876af3e5472e7fc

$ git rev-list --header '^2148e6b4a4c51f84ffc406089876af3e5472e7fc' HEAD
3f6b1ab9d252a52c5567d9692889c0b2b4637f65
tree b7927314af86400ee2ab263a70d1f592d6494cc1
parent 2148e6b4a4c51f84ffc406089876af3e5472e7fc
author Wang, Eikan <[email protected]> 1655873368 +0000
committer Wang, Eikan <[email protected]> 1655873368 +0000

    Test ghstack
\0
$ git rev-list --header '^2148e6b4a4c51f84ffc406089876af3e5472e7fc^@' 2148e6b4a4c51f84ffc406089876af3e5472e7fc
2148e6b4a4c51f84ffc406089876af3e5472e7fc
tree cfceb81d57d8c98b7931d7668459e8af9a3fd048
parent b99a1653ed0ffb01b924d99ae72178f08ad23f48
author PyTorch MergeBot <[email protected]> 1655869300 +0000
committer PyTorch MergeBot <[email protected]> 1655869302 +0000

    [torchdynamo hash update] update the pinned torchdynamo hash (#79149)
    
    This PR is auto-generated nightly by [this action](https://github.com/pytorch/pytorch/blob/master/.github/workflows/_update-commit-hash.yml).
    Update the pinned torchdynamo hash.
    Pull Request resolved: https://github.com/pytorch/pytorch/pull/79149
    Approved by: https://github.com/clee2000, https://github.com/malfet
\0
$ git rev-parse --show-toplevel
/home/eikan/dev/pytorch

$ git config --default /home/eikan/dev/pytorch/.git/hooks --get core.hooksPath
/home/eikan/dev/pytorch/.git/hooks

$ git for-each-ref refs/remotes/origin/gh/EikanWang '--format=%(refname)'
$ git rev-parse '3f6b1ab9d252a52c5567d9692889c0b2b4637f65~^{tree}'
cfceb81d57d8c98b7931d7668459e8af9a3fd048

$ git config --get commit.gpgsign
$ git commit-tree -p 2148e6b4a4c51f84ffc406089876af3e5472e7fc b7927314af86400ee2ab263a70d1f592d6494cc1
6ba845eb50d83158e14702d23d6e28ec21901f4b

$ git push origin 6ba845eb50d83158e14702d23d6e28ec21901f4b:refs/heads/gh/EikanWang/1/head 2148e6b4a4c51f84ffc406089876af3e5472e7fc:refs/heads/gh/EikanWang/1/base
# stderr:
remote: Permission to pytorch/pytorch.git denied to EikanWang.
fatal: unable to access 'https://github.com/pytorch/pytorch.git/': The requested URL returned error: 403

ERROR: Fatal exception
Traceback (most recent call last):
  File "/home/eikan/dev/anaconda3/envs/pyt-te/lib/python3.8/site-packages/ghstack/logs.py", line 107, in manager
    yield
  File "/home/eikan/dev/anaconda3/envs/pyt-te/lib/python3.8/site-packages/ghstack/__main__.py", line 101, in main
    ghstack.submit.main(
  File "/home/eikan/dev/anaconda3/envs/pyt-te/lib/python3.8/site-packages/ghstack/submit.py", line 203, in main
    submitter.prepare_updates()
  File "/home/eikan/dev/anaconda3/envs/pyt-te/lib/python3.8/site-packages/ghstack/submit.py", line 930, in prepare_updates
    self.process_new_commit(s)
  File "/home/eikan/dev/anaconda3/envs/pyt-te/lib/python3.8/site-packages/ghstack/submit.py", line 611, in process_new_commit
    self.sh.git(
  File "/home/eikan/dev/anaconda3/envs/pyt-te/lib/python3.8/site-packages/ghstack/shell.py", line 273, in git
    return self._maybe_rstrip(self.sh(*(("git",) + args), **kwargs))
  File "/home/eikan/dev/anaconda3/envs/pyt-te/lib/python3.8/site-packages/ghstack/shell.py", line 210, in sh
    raise RuntimeError(
RuntimeError: git push origin 6ba845eb50d83158e14702d23d6e28ec21901f4b:refs/heads/gh/EikanWang/1/head 2148e6b4a4c51f84ffc406089876af3e5472e7fc:refs/heads/gh/EikanWang/1/base failed with exit code 128

ghstack fails to update if tree matches exactly, even if update is needed

Hey @ezyang! Thanks for the awesome tool.

Is landing partial stacks supposed to work? Experimentally I couldn't get it to work, but wanted to see if I was missing something!

Add `sync` subcommand

The goal would be to update local commit messages with info from GitHub (essentially the opposite of --update-fields), similar to jf sync.

ghstack command won't work if some PR has been closed

I am trying to modify pytorch/pytorch#18650 (comment) by checking out the corresponding commits in master branch, modify the file, and git commit --amend and then run ghstack, but I get an error:

# Updating https://github.com/pytorch/pytorch/pull/18648
ERROR: Fatal exception
Traceback (most recent call last):
  File "/home/gaoxiang/.virtualenvs/pt/lib/python3.7/site-packages/ghstack/__main__.py", line 153, in main
    update_fields=args.update_fields
  File "/home/gaoxiang/.virtualenvs/pt/lib/python3.7/site-packages/ghstack/submit.py", line 153, in main
    submitter.post_process()
  File "/home/gaoxiang/.virtualenvs/pt/lib/python3.7/site-packages/ghstack/submit.py", line 594, in post_process
    base=s.base)
  File "/home/gaoxiang/.virtualenvs/pt/lib/python3.7/site-packages/ghstack/github.py", line 50, in patch
    return self.rest('patch', path, **kwargs)
  File "/home/gaoxiang/.virtualenvs/pt/lib/python3.7/site-packages/ghstack/github_real.py", line 98, in rest
    r.raise_for_status()
  File "/home/gaoxiang/.virtualenvs/pt/lib/python3.7/site-packages/requests/models.py", line 939, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: https://api.github.com/repos/pytorch/pytorch/pulls/18648

I think this is because the corresponding PR (pytorch/pytorch#18648) has been landed and closed.

In such case, instead of failing with error, should we just skip modifying that PR and continue?

Amending commit message would not update PR description

If I run

git commit --amend

and modify the commit message, running ghstack would change the commit message back to the previous version and not update the PR description.

Don't unconditionally allow pushing to other people's stacks

Probably should require you to set some flag first before we are willing to push over it

Add command line option to specify labels

Consider adding a command line option (e.g. --labels=...) that would allow users to specify labels that should be applied to the PR when it is created. Without this, users have to open the PR and add the label(s) manually or use the GitHub command line tool after running ghstack.

I'm 90% sure the GitHub API supports adding labels to PRs (either at creation time or after), so I think it's matter of adding the option and plumbing the value provided by the user through to the API call.

Omitting `--no-skip` reverts local commit message

I ran into this bug when trying to submit a change to the commit message without using --no-skip, the commit message on my local repo got reverted to the one from GitHub

[mac] ~/D/pytorch > git log --pretty='%s' | head -n 1
[Skip CI] Document '[Skip CI]'

[mac] ~/D/pytorch > git commit --amend --allow-empty
[master 9860f92d62] A different title
 Author: Your Name <[email protected]>
 Date: Tue Apr 6 16:08:01 2021 -0700
 1 file changed, 2 insertions(+)

[mac] ~/D/pytorch > git log --pretty='%s' | head -n 1
A different title

[mac] ~/D/pytorch > ghstack submit  
...
# Summary of changes (ghstack 0.4.1)

 - Skipped https://github.com/pytorch/pytorch/pull/55418
...

[mac] ~/D/pytorch > git log --pretty='%s' | head -n 1
[Skip CI] Document '[Skip CI]'

RuntimeError: git config --default failed with exit code 129

Bug Report

When submitting commits ghstack was trying to assign option --default to git config but my git does not support it.

Git version: 2.17.1

Error and tracestack:

$ ghstack submit
NB: configuration saved to /data/home/langong/.ghstackrc
$ git remote get-url origin
$ git fetch --prune origin
$ git merge-base origin/main HEAD
$ git rev-list --header '^89c57c311b9361407da3b8fe82a53ed2682e0906' HEAD
$ git rev-list --header '^89c57c311b9361407da3b8fe82a53ed2682e0906^@' 89c57c311b9361407da3b8fe82a53ed2682e0906
$ git rev-parse --show-toplevel
$ git config --default /fsx/users/langong/work/multimodal/.git/hooks --get core.hooksPath
error: unknown option `default'
usage: git config [<options>]

Config file location
    --global              use global config file
    --system              use system config file
    --local               use repository config file
    -f, --file <file>     use given config file
    --blob <blob-id>      read config from given blob object

Action
    --get                 get value: name [value-regex]
    --get-all             get all values: key [value-regex]
    --get-regexp          get values for regexp: name-regex [value-regex]
    --get-urlmatch        get value specific for the URL: section[.var] URL
    --replace-all         replace all matching variables: name value [value_regex]
    --add                 add a new variable: name value
    --unset               remove a variable: name [value-regex]
    --unset-all           remove all matches: name [value-regex]
    --rename-section      rename section: old-name new-name
    --remove-section      remove a section: name
    -l, --list            list all
    -e, --edit            open an editor
    --get-color           find the color configured: slot [default]
    --get-colorbool       find the color setting: slot [stdout-is-tty]

Type
    --bool                value is "true" or "false"
    --int                 value is decimal number
    --bool-or-int         value is --bool or --int
    --path                value is a path (file or directory name)
    --expiry-date         value is an expiry date

Other
    -z, --null            terminate values with NUL byte
    --name-only           show variable names only
    --includes            respect include directives on lookup
    --show-origin         show origin of config (file, standard input, blob, command line)

ERROR: Fatal exception
Traceback (most recent call last):
  File "/fsx/users/langong/miniconda3/lib/python3.9/site-packages/ghstack/logs.py", line 107, in manager
    yield
  File "/fsx/users/langong/miniconda3/lib/python3.9/site-packages/ghstack/__main__.py", line 101, in main
    ghstack.submit.main(
  File "/fsx/users/langong/miniconda3/lib/python3.9/site-packages/ghstack/submit.py", line 179, in main
    run_pre_ghstack_hook(sh, base, stack[0].oid)
  File "/fsx/users/langong/miniconda3/lib/python3.9/site-packages/ghstack/submit.py", line 1062, in run_pre_ghstack_hook
    hooks_path = sh.git("config", "--default", default_hooks_path, "--get", "core.hooksPath")
  File "/fsx/users/langong/miniconda3/lib/python3.9/site-packages/ghstack/shell.py", line 273, in git
    return self._maybe_rstrip(self.sh(*(("git",) + args), **kwargs))
  File "/fsx/users/langong/miniconda3/lib/python3.9/site-packages/ghstack/shell.py", line 210, in sh
    raise RuntimeError(
RuntimeError: git config --default /fsx/users/langong/work/multimodal/.git/hooks --get core.hooksPath failed with exit code 129

Fixed this by commenting out this line here:

ghstack/ghstack/submit.py

Line 179 in 58404ad

run_pre_ghstack_hook(sh, base, stack[0].oid)

Merge to master from GitHub UI or with other checks passing?

We have Jenkins and CodeCov jobs that are running on our PRs. In the GitHub UI, this blocks merging into master until all of those pass. It's unclear how/if we could use ghstack and maintain all those other checks and making PRs for in the GitHub UI. Does this work?

Make python setup.py clean work

Currently python setup.py clean doesn't actually clean anything :P
It should at least remove build/ folder, might also clean up a few additional files generated in the build process.
This can be done by overwriting clean command in setup.py, see some examples in https://github.com/taichi-dev/taichi/blob/master/setup.py#L213.

Name pull requests to group them by the stack their in

I'm evaluating this library and just made a test repo, pushed some stacked diffs on master, then created a feature branch, pushed that stack. The problem I saw right away is that as a code reviewer, I can't look at my PRs tab and understand what's going on, the PRs simply have the names of the commits. I'd expect them to be prepended by some information to groups them in the way you already do with the branch names and whatnot.

I'm assuming y'all have a way of working around this, maybe you don't use the "Pull requests" tab on github? Would appreciate understanding how you approach this.

PyPA homepage URL

On https://pypi.org/project/ghstack/ , the homepage link seems to be broken, it points to https://github.com/pypa/ghexport

Usernames with capital letters are rejected

My GitHub username contains capital letters and was initially rejected by ghstack as an invalid GitHub username. It accepted once I entered it in all lowercase letters. I think it would a better user experience if the tool accepted usernames as they are displayed on GitHub and in commit messages (in my case, with capital letters), and converted to lowercase in the code if needed.

Put ghstack message below commit message body / update local message with remote

Just another observation from using it - enjoying ghstack more every day. :)

I was wondering if there is a (technical) reason that the ghstack info is inserted above the commit message? Maybe it could be a config to either inject the stack info above or below the message.

And I was curious if amending the local commit message with remote changes has come up in the past? The use case here is that I/we often add screenshots to a commit on github.com directly. When I update the commit by running ghstack again, the remote edits are overwritten.

That's all - one of these days I have to take an afternoon to work on these improvements. ;)

Add 'ghstack pull'

Right now, if someone updated a ghstack PR, and then you try to push to it without updating first, ghstack will just error out. If we supported ghstack pull that would result in a Git merge that would be better. We can just do the merge on the synthetic head branches, and then recreate a correct orig commit after it.

Automatically delete head branches option makes ghstack land fail

When you have the Automatically delete head branches option enabled for PR merges, ghstack errors out when landing due to the head branch already being deleted:

error: failed to push some refs to '[email protected]:foo/foo.git'
$ git checkout main
Previous HEAD position was a12eb99 foo commit message
Switched to branch 'main' 
ERROR: Fatal exception
Traceback (most recent call last): 
File "/home/xiphirx/.local/lib/python3.8/site-packages/ghstack/logs.py", line 107, in manager
yield
File "/home/xiphirx/.local/lib/python3.8/site-packages/ghstack/__main__.py", line 123, in main
ghstack.land.main(
File "/home/xiphirx/.local/lib/python3.8/site-packages/ghstack/land.py", line 130, in main
sh.git("push", remote_name, "--delete", orig_ref, base_ref, head_ref)
File "/home/xiphirx/.local/lib/python3.8/site-packages/ghstack/shell.py", line 271, in git
return self._maybe_rstrip(self.sh(*(("git",) + args), **kwargs))
File "/home/xiphirx/.local/lib/python3.8/site-packages/ghstack/shell.py", line 210, in sh
raise RuntimeError(
RuntimeError: git push origin --delete gh/xiphirx/2/orig gh/xiphirx/2/base gh/xiphirx/2/head failed with exit code 1

Turning the option off makes ghstack work as expected, but I think it would be cool if ghstack would recognize that the branch is already gone + skip deleting it + delete the others and continue on like normal.

Block / Abort Land If Not All PRs In Stack Are Approved

Started using ghstack with our project this week. The workflow feels already so much better - or really just how I'm used to working with Phabricator - small diffs / change requests and always on top of the base branch.

I looked through the issues but didn't see any mention of being able to block landing based on GH PR approval status. Has it just not come up before? Am I thinking about this wrong such as one should just use GH branch protection rules?

Would it be possible to turn this into a gh extension?

make it possible/easy to recover ghstack metadata when the original local branch is deleted

so, i haven't looked much into this, just noticed that ghstack strips the metadata after pushing to remote (still available locally tho). now, what would happen if a user accidentally deletes their branch/checkout but still wants to push more commits as part of the original stack?

i know that ghstack uses the tree hash, but i haven't checked how difficult (or possible at all) it would be to recover in a situation like this.

thoughts?

ps: why is metadata stripped in the first place? is it to protect others from messing with your stack?

Introduce concept of env to make gpg signing easier to test

#78 is a reasonable implementation given the current code structure but it is not great for testing because it (1) makes use of the ambient Git configuration (which is implemented as a one off function that probes git config), (2) requires us to be able to interpose on Git configuration in testing. The style of testing in ghstack today is that environmental settings are all turned into concrete arguments to the function at main() and that means that calls from testing can override the behavior there; refactor ghstack so that this occurs here as well.

Stop replicating PR titles in the stack description

GitHub has changed their rendering to display the entire PR name when a PR shows up in a bullet list, as so:

So we should stop tacking on the title now

0.4 breaks CI jobs that merge with master

A side effect of #44 is that we no longer maintain accurate merge-base tracking on head branch. Normally this is fine but if someone checks out head branch and merges with master, you will end up with spurious merge conflicts. This is affecting pytorch/pytorch CI, which does precisely this.

Stacking a commit on top of someone else's stacked PR is failing

Not sure if this is even possible or not?

ghstack checkout https://github.com/pytorch/pytorch/pull/38201
git switch -c mypy-stackme
git cherry-pick some-commit-id
ghstack

gives:

$ git remote get-url origin
$ git fetch origin
Enter passphrase for key '/home/rgommers/.ssh/id_rsa': 
$ git merge-base origin/master HEAD
$ git rev-list --header ^b579433bf7fc2d4cd333bfdacee6f08f5c7f30b1 HEAD
$ git rev-list --header ^b579433bf7fc2d4cd333bfdacee6f08f5c7f30b1^@ b579433bf7fc2d4cd333bfdacee6f08f5c7f30b1
$ git rev-list --max-count=1 --header origin/gh/rgommers/751/orig
fatal: ambiguous argument 'origin/gh/rgommers/751/orig': unknown revision or path not in the working tree.

Clear that that branch doesn't exist, it replaces the username of the original PR submitter with my username.

git lg output looks fine:

* 33b6c27182 - (HEAD -> mypy-stackme) Add type annotations for iinfo/finfo, run mypy on some test files (8 minutes ago)
* 6fde698cba - (origin/gh/ezyang/755/orig) Move torch/autograd/grad_mode.pyi stubs inline (17 hours ago)
* 90a2d0e4b6 - (origin/gh/ezyang/752/orig) Device and torch._C function cleanup (17 hours ago)
* df2cd3de05 - (origin/gh/ezyang/751/orig) Delete torch/__init__.pyi, deferring to direct extension stubs (17 hours ago)

Submitting reordered commits results in close of certain PRs

Let's say I submit three commits:

$ git commit -am Howdy
$ git commit -am Wally
$ git commit -am Indie
$ ghstack # results in PRs 15, 16, 17

Then I reorder Wally and Indie with git rebase -i, so that the order is now: Howdy, Indie, Wally.

When I re-run ghstack, I see that Wally (now at the end) has its PR closed:

yang merged commit 6c6635b into gh/yang/15/base 4 minutes ago

Is there a way to prevent that close from happening? Thanks!

ghstack land merges PR even if master push fails (e.g., because it's protected)

This is because of the order of operations: we have to merge the PR first otherwise the pushed commit will close it. But that means that if the push actually fails then you have a merged PR but not actually getting to master.