glandium / git-cinnabar Goto Github PK

git remote helper to interact with mercurial repositories

License: GNU General Public License v2.0

Makefile 1.95% C 5.44% Python 10.95% Shell 1.78% Perl 13.94% Rust 65.94%

git-cinnabar's Introduction

git-cinnabar 0.7

cinnabar is the common natural form in which mercury can be found on Earth. It contains mercury sulfide and its powder is used to make the vermillion pigment.

git-cinnabar is a git remote helper to interact with mercurial repositories. Contrary to other such helpers ([1] [2] [3] [4] [5] [6]), it doesn't use a local mercurial clone under the hood.

The main focus at the moment is to make it work with mozilla-central and related mercurial repositories and support Mozilla workflows (try server, etc.).

Repositories last used with versions lower than 0.5.0 are not supported. Please run git cinnabar upgrade with version 0.5.0 first.

License:

The git-cinnabar source code is distributed under the terms of the Mozilla Public License version 2.0 (see the MPL-2.0 file), with parts (the git-core subdirectory) distributed under the terms of the GNU General Public License version 2.0 (see the git-core/COPYING file).

As a consequence, git-cinnabar binary executables are distributed under the terms of the GNU General Public License version 2.0.

Requirements:

Git (any version should work ; cinnabarclone bundles require 1.4.4).
In order to build from source:
- Rust 1.74.0 or newer.
- A C compiler (GCC or clang).
- make.
- CURL development headers and libraries (except on Windows). Please note that on MacOS they are included in the SDK.

Setup:

Prebuilt binaries

Assuming a prebuilt binary is available for your system, get the download.py script and run it (requires python 3.6 or newer) with:
```
$ ./download.py
```
Add the directory where the download happened to your PATH. If you have another git-remote-hg project in your PATH already, make sure the git-cinnabar path comes before.

Cargo

Run the following:

$ cargo install --locked git-cinnabar
$ git cinnabar setup

Build manually

Run the following:

$ git clone https://github.com/glandium/git-cinnabar
$ cd git-cinnabar
$ make

Add the git-cinnabar directory to your PATH.

Usage:

$ git clone hg::<mercurial repo>

where <mercurial repo> can be a path to a local directory containing a mercurial repository, or a http, https or ssh url.

Essentially, use git like you would for a git repository, but use a hg:: url where you would use a git:// url.

See https://github.com/glandium/git-cinnabar/wiki/Mozilla:-A-git-workflow-for-Gecko-development for an example workflow for Mozilla repositories.

Remote refs styles:

Mercurial has two different ways to handle what git would call branches: branches and bookmarks. Mercurial branches are permanent markers on each changeset that belongs to them, and bookmarks are similar to git branches.

You may choose how to interact with those with the cinnabar.refs configuration. The following values are supported, either individually or combined in a comma-separated list:

bookmarks: in this mode, the mercurial repository's bookmarks are exposed as refs/heads/$bookmark. Practically speaking, this means the mercurial bookmarks appear as the remote git branches.
tips: in this mode, the most recent head of each mercurial branch is exposed as refs/heads/$branch. Any other head of the same branch is not exposed. This mode can be useful when branches have no more than one head.
heads: in this mode, the mercurial repository's heads are exposed as refs/heads/$branch/$head, where $branch is the mercurial branch name and $head is the full changeset sha1 of that head.

When these values are used in combinations, the branch mappings are varied accordingly to make the type of each remote ref explicit and to avoid name collisions.

When combining bookmarks and heads, bookmarks are exposed as refs/heads/bookmarks/$bookmark and branch heads are exposed as refs/heads/branches/$branch/$head (where $head is the full changeset sha1 of the head).
When combining bookmarks and tips, bookmarks are exposed as refs/heads/bookmarks/$bookmark and branch tips are exposed as refs/heads/branches/$branch. Any other heads of the same branch are not exposed.
When combining all of bookmarks, heads, and tips, bookmarks are exposed as refs/heads/bookmarks/$bookmark, branch heads are exposed as refs/heads/branches/$branch/$head (where $head is the full changeset sha1 of the head), except for the branch tips, which are exposed as refs/heads/branches/$branch/tip.

The shorthand all (also the default), is the combination of bookmarks, heads, and tips.

The refs style can also be configured per remote with the remote.$remote.cinnabar-refs configuration. It is also possible to use cinnabar.pushrefs or remote.$remote.cinnabar-pushrefs to use a different scheme for pushes only.

Tags:

You can get/update tags with the following command:

$ git cinnabar fetch --tags

Fetching a specific mercurial changeset:

It can sometimes be useful to fetch a specific mercurial changeset from a remote server, without fetching the entire repository. This can be done with a command line such as:

$ git cinnabar fetch hg::<mercurial repo> <changeset sha1>

Translating git commits to mercurial changesets and vice-versa:

When dealing with a remote repository that doesn't use the same identifiers, things can easily get complicated. Git-cinnabar comes with commands to know the mercurial changeset a git commit represents and the other way around.

The following command will give you the git commit corresponding to the given mercurial changeset sha1:

$ git cinnabar hg2git <changeset>

The following command will give you the mercurial changeset corresponding to the given git commit sha1:

$ git cinnabar git2hg <commit>

Both commands allow abbreviated forms, as long as they are unambiguous (no need for all the 40 hex digits of the sha1).

Avoiding metadata:

In some cases, it is not desirable to have git-cinnabar create metadata for all pushed commits. Notably, for volatile commits such as those used on the Mozilla try repository.

By default, git-cinnabar doesn't store metadata when pushing to non-publishing repositories. It does otherwise.

This behavior can be changed per-remote with a remote.$remote.cinnabar-data preference with one of the following values:

always
never
phase

phase is the default described above. always and never are self-explanatory.

Cinnabar clone:

For large repositories, an initial clone can take a large amount of time. A Mercurial server operator can install the extension provided in mercurial/cinnabarclone.py, and point to a git repository or bundle containing pre-generated git-cinnabar metadata. See details in the extension file.

Users cloning the repository would automatically get the metadata from the git repository or bundle, and then pull the missing changesets from the Mercurial repository.

Limitations:

At the moment, push is limited to non-merge commits.

There is no support for the following mercurial features:

obsolescence markers
phases
namespaces

Checking corruptions:

Git-cinnabar is still in early infancy, and its metadata might get corrupted for some reason.

The following command allows to detect various types of metadata corruption:

git cinnabar fsck

This command will fix the corruptions it can, as well as adjust some of the metadata that contains items that became unnecessary in newer versions.

The --full option may be added for a more thorough validation of the metadata contents. Using this option adds a significant amount of work, and the command can take more than half an hour on repositories the size of mozilla-central.

`hg://` urls:

The msys shell (not msys2) doesn't keep hg::url intact when crossing the msys/native boundary, so when running cinnabar in a msys shell with a native git, the url is munged as hg;;proto;\host\path\, which git doesn't understand and doesn't even start redirecting to git-remote-hg.

To allow such setups to still work, hg:// urls are supported. But since mercurial can be either on many different protocols, we abuse the port in the given url to pass the protocol.

A hg:// url thus looks like:

hg://<host>[:[<port>.]<protocol>]/<path>

The default protocol is https, and the port can be omitted.

hg::https://hg.mozilla.org/mozilla-central becomes hg://hg.mozilla.org/mozilla-central
hg::http://hg.mozilla.org/mozilla-central becomes hg://hg.mozilla.org:http/mozilla-central
hg::ssh://hg.mozilla.org/mozilla-central becomes hg://hg.mozilla.org:ssh/mozilla-central
hg::file:///some/path becomes (awkward) hg://:file/some/path
hg::http://localhost:8080/foo becomes hg://localhost:8080.http/foo
hg::tags: becomes hg://:tags

Compatibility:

As of version 0.7, some corner cases in Mercurial repositories will generate different git commits than with prior versions of git-cinnabar. This means a fresh clone might have different git SHA-1s than existing clones, but this doesn't impact the use of existing clones with newer versions of git-cinnabar.

Most repositories should remain non-affected by the change.

You can set the cinnabar.compat git configuration to 0.6 to keep the previous behavior.

Experimental features:

Git-cinnabar has a set of experimental features that can be enabled independently. You can set the cinnabar.experiments git configuration to a comma-separated list of those features to enable the selected ones.

The available features are:

merge

Git-cinnabar currently doesn’t allow to push merge commits. The main reason for this is that generating the correct mercurial data for those merges is tricky, and needs to be gotten right.

The main caveat with this experimental support for pushing merges is that it currently doesn’t handle the case where a file was moved on one of the branches the same way mercurial would (i.e. the information would be lost to mercurial users).
similarity

Git doesn't track file copies or renames. It however has flags to try to detect them after the fact. On the other hand, Mercurial does track copies and renames, if they're recorded manually in the first place. Git-cinnabar does exact-copy/rename detection when pushing new commits to a Mercurial repository.

The similarity feature allows to configure how (dis)similar files can be to be detected as a rename or copy. similarity=100 is the default, which means only 100% identical files are considered. similarity=90 means 90% identical files, and so on.

This is equivalent to git diff -C -C${similarity}%

git-cinnabar's People

Contributors

Stargazers

Watchers

git-cinnabar's Issues

Grafting did not seem to work

I just started using cinnabar after following the steps in your post, specifically the "Switching to git-cinnabar" header.

I had an existing clone of gecko-dev with various branches I've accumulated. After following the steps indicated, I ran:

git -c cinnabar.graft=true -c cinnabar.graft-refs=refs/remotes/upstream/* remote update

as instructed (replacing origin with upstream, since that's what I called the remote before). This pulled down the Mercurial data, but did not appear to graft.

git status on existing branches shows they are ahead and behind ~100k commits, so I assume that means it failed.

I rebased one branch onto the pull data as a test, and it no longer seems to use gecko-dev SHAs for existing commits.

What can I do to make it graft correctly?

I did have my connection drop out during my first run of the above remote update command, so I had to restart it after it died.

Allow to graft with existing repositories

Say, you have a clone of https://github.com/mozilla/gecko-dev/. It has different sha1s than the one produced by git-remote-hg, first, because its history was grafted to the mozilla-cvs history, and second because hg-git doesn't produce the same exact commits. It would be useful to be able to graft git-remote-hg on top of existing commits from such a repository.

Allow to push merge commits

Allow to pull a specific revision

Mercurial allows this:
hg pull -r 123456789abc

It would be nice to have the same possibility, especially for mozreview and try.

Cannot clone http://hg.mozilla.org/build/tools/

fatal: Empty path component found in input
fast-import: dumping crash report to .git/fast_import_crash_31992

Most Recent Commands Before Crash
---------------------------------
(snip)
  M 160000 :19321 d245d51304cb9f03041efd73b9f6d4def01f0cd2/FIREFOX_10_0_10esr_RELEASE_RUNTIME
  M 160000 :20430 d245d51304cb9f03041efd73b9f6d4def01f0cd2/FIREFOX_26_0b7_RELEASE
  M 160000 :20213 d245d51304cb9f03041efd73b9f6d4def01f0cd2/FIREFOX_24_0b7_BUILD1
* M 160000 0000000000000000000000000000000000000000 d245d51304cb9f03041efd73b9f6d4def01f0cd2/cdmaker/

Allow pushing HEAD to a remote bookmark

It would be nice if it were possible to do something like

git push --set-upstream repo HEAD:some_bookmark

and have it create a bookmark named some_bookmark in repo, with git pull and git push working as if it were a normal remote branch. I'm not quite sure how feasible that is, but some subset of that behaviour would also likely be useful.

Convert cinnabar-helper.c to struct object_id

spinning for more than half and hour on try push

Tried out the latest git-remote-hg today. Pulled from central ok, but pushing to try has been hung using a full cpu core for half and hour now. Did I do something wrong? This is on MacOS X 10.10.3. Homebrew git 2.4.0, hg 3.4.

Let me use my copy of git

...as long as the system copy of git is adequate, that is.

Building and maintaining tools is a job that OS-level tools are generally much better at.

Add a test suite

'git cinnabar' results in IndexError due to empty command list

Just run the following command:

git cinnabar
Traceback (most recent call last):
File "/mozilla/code/git-cinnabar//git-cinnabar", line 424, in
sys.exit(main(sys.argv[1:]))
File "/mozilla/code/git-cinnabar//git-cinnabar", line 380, in main
cmd = args.pop(0)
IndexError: pop from empty list

If args are given we might want to print usage / help output?

Cloning from empty http-based repo fails

$ git clone hg::http://localhost:8000/test-repo gitrepo
Cloning into 'gitrepo'...
Traceback (most recent call last):
File "/Users/gps/src/hgcustom/version-control-tools/venv/git-cinnabar/git-remote-hg", line 433, in
sys.exit(main(sys.argv[1:]))
File "/Users/gps/src/hgcustom/version-control-tools/venv/git-cinnabar/git-remote-hg", line 284, in main
getbundle(repo, store, heads, branchmap)
File "/Users/gps/src/hgcustom/version-control-tools/venv/git-cinnabar/cinnabar/hg.py", line 228, in getbundle
common=[unhexlify(h) for h in common])
File "/Users/gps/src/hgcustom/version-control-tools/venv/lib/python2.7/site-packages/mercurial/wireproto.py", line 327, in getbundle
f = self._callcompressable("getbundle", *_opts)
File "/Users/gps/src/hgcustom/version-control-tools/venv/lib/python2.7/site-packages/mercurial/httppeer.py", line 257, in _callcompressable
stream = self._callstream(cmd, *_args)
File "/Users/gps/src/hgcustom/version-control-tools/venv/lib/python2.7/site-packages/mercurial/httppeer.py", line 137, in _callstream
resp = self.urlopener.open(req)
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 437, in open
response = meth(req, response)
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 550, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 475, in error
return self._call_chain(_args)
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain
result = func(_args)
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 558, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 500: Internal Server Error
fatal: Could not read ref refs/cinnabar/refs/heads/branches/default/tip
[128]

From the Mercurial server logs:

[Sat Jan 16 00:55:54 2016] [error] [client 192.168.99.1] mod_wsgi (pid=303): Exception occurred processing WSGI script '/repo/hg/webroot_wsgi/hgweb.wsgi'.
[Sat Jan 16 00:55:54 2016] [error] [client 192.168.99.1] Traceback (most recent call last):
[Sat Jan 16 00:55:54 2016] [error] [client 192.168.99.1] File "/repo/hg/venv_hgweb/lib/python2.7/site-packages/mercurial/hgweb/hgweb_mod.py", line 281, in run_wsgi
[Sat Jan 16 00:55:54 2016] [error] [client 192.168.99.1] for r in self._runwsgi(req, repo):
[Sat Jan 16 00:55:54 2016] [error] [client 192.168.99.1] File "/repo/hg/version-control-tools/hgext/serverlog/init.py", line 273, in _runwsgi
[Sat Jan 16 00:55:54 2016] [error] [client 192.168.99.1] for what in super(hgwebwrapped, self)._runwsgi(req, repo):
[Sat Jan 16 00:55:54 2016] [error] [client 192.168.99.1] File "/repo/hg/venv_hgweb/lib/python2.7/site-packages/mercurial/hgweb/protocol.py", line 60, in groupchunks
[Sat Jan 16 00:55:54 2016] [error] [client 192.168.99.1] chunk = cg.read(4096)
[Sat Jan 16 00:55:54 2016] [error] [client 192.168.99.1] AttributeError: 'NoneType' object has no attribute 'read'

It's unfortunate Mercurial is raising an exception here. But my guess is it is a garbage in => garbage out scenario.

AttributeError: 'VersionedDict' object has no attribute 'add' when running git cinnabar fsck after upgrading to 0.3.0

I did the following today:

$ git remote update
$ git checkout release

And then when I tried updating my remotes, I was prompted to run fsk. Running that results in:

$ git cinnabar fsck
Traceback (most recent call last):
  File "/Users/ehsan/src/git-remote-hg/git-cinnabar", line 633, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/Users/ehsan/src/git-remote-hg/git-cinnabar", line 596, in main
    return fsck(args)
  File "/Users/ehsan/src/git-remote-hg/git-cinnabar", line 95, in fsck
    store = UpgradeGitHgStore()
  File "/Users/ehsan/src/git-remote-hg/cinnabar/githg.py", line 655, in __init__
    self._open()
  File "/Users/ehsan/src/git-remote-hg/git-cinnabar", line 59, in _open
    self._hgheads.add((branch, hghead))
AttributeError: 'VersionedDict' object has no attribute 'add'

I was running 0.2.2 before attempting to update.

Startup fails because of multi-line value in `git config -l`

I happened to have a value in my git config -l which was a shell script that included \n. This causes the value to print as multiple lines from git config -l. When I run git cinnabar fsck from commit cc65bd5, I get the stack trace:

Traceback (most recent call last):
  File "/Users/jryans/projects/git-cinnabar/git-cinnabar", line 632, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/Users/jryans/projects/git-cinnabar/git-cinnabar", line 574, in main
    init_logging()
  File "/Users/jryans/projects/git-cinnabar/cinnabar/util.py", line 52, in init_logging
    log_conf = Git.config('cinnabar.log')
  File "/Users/jryans/projects/git-cinnabar/cinnabar/git.py", line 472, in config
    for l in self.iter('config', '-l'))
  File "/Users/jryans/projects/git-cinnabar/cinnabar/git.py", line 470, in <dictcomp>
    k: v
ValueError: need more than 1 value to unpack

For now, I have commented out my config value, since using cinnabar is more important.

Handle copies and moves somehow

Right now git-remote-hg is oblivious to copies and moves, resulting in pushes such as https://hg.mozilla.org/integration/mozilla-inbound/rev/594b535e44c8, which make people sad. I can't think of a good way to handle this, but perhaps we should look for copies/moves with the default settings and use that information, and just not care about the cases where we detect invalid renames/copies?

Add a command for git <-> hg changeset resolution

It would be useful to be able to easily find git commits from their corresponding mercurial changeset sha1, and vice versa.

git-cinnabar is a bit rough on windows

Following the instructions on http://glandium.org/blog/?page_id=3438 ends up with really mangled paths in colons becoming semicolons.

The symlinks from git-remote-hg to git-remote-hg.py don't really work.

Add documentation how to setup git-cinnabar with mozreview

Given that we all should use mozreview for submitting patches to Bugzilla, it would be great if there would be some kind of documentation in how to set this up with cinnabar. Maybe it can be linked from the official mozreview setup documentation?

fsck erroneously reports missing git tree for empty manifests

$ git clone hg::https://gem5-gpu.cs.wisc.edu/repo/benchmarks/libopencl
Cloning into 'libopencl'...
Reading 4 changesets
Reading 4 manifests
Reading and importing 8 files
Importing 4 manifests
Importing 4 changesets
Checking connectivity... done.
It is recommended that you set "remote.origin.prune" or "fetch.prune" to "true".
  git config remote.origin.prune true
or
  git config fetch.prune true
$ cd libopencl
$ git cinnabar fsck
Reading 16 mercurial to git mappings
Reading 4 commit to changeset mappings
Reading 4 manifest trees
Missing git tree in manifest commit 450df3fb831f08a7f72d5cfc8fbe3b6f65b178cf
Checking 4 changesets
Your git-cinnabar repository appears to be corrupted. There
are known issues in older revisions that have been fixed.
Please try running the following command to reset:
  git cinnabar reclone

Please note this command may change the commit sha1s. Your
local branches will however stay untouched.
Please report any corruption that fsck would detect after a
reclone.

fatal: empty ident name (for BAD DATA) not allowed

The BAD DATA is formed like "<FIRSTNAME LASTNAME [email protected]>". (I'm happy to share the full address locally, or you can search for it in hg.mozilla.org. Just didn't want to link search engines to it, etc.)

So, that's clearly bad data for git, and questionable for hg; but since hg accepts it, cinnabar needs to do something. Feel free to close or file a new bug elsewhere.

Restricting fetched branches breaks fetching tags

When using setups like on https://github.com/glandium/git-cinnabar/wiki/Mozilla:-A-git-workflow-for-Gecko-development [1] for e.g. mozilla-release, the fact that some branches are never pulled breaks updating tags.

$ git config remote.central.fetch +refs/heads/branches/*/tip:refs/remotes/central/*

Reorganize mercurial metadata

Currently, mercurial metadata looks like this:
refs/notes/cinnabar: contains info mapping git commits to hg changesets
refs/cinnabar/hg2git: contains info mapping hg sha1s to git sha1s
refs/cinnabar/manifest: all manifests commits, stored linearly (which loses their ancestry information, which needs to be reconstructed from changesets history)
refs/cinnabar/tag-cache: self explanatory
refs/cinnabar/branches/$branch/$head: references to all mercurial heads cinnabar knows about

Especially the latter is a lot of refs. The flattened history of manifests make it error-prone to implement issue #26. It was flattened to avoid to essentially double the number of refs used by cinnabar alone.

But it turns out that contrary to what the git fast-import manual implied, there is no limit to the number of parents an octopus merge can have, so we could use cthulhu merges to reduce changeset and manifest refs to one ref each, allowing to store manifests in a more natural non-flat way as well.

While touching manifests, might as well add the mercurial sha1 for each manifest in its corresponding commit message.

Uses a lot of memory when pushing a lot of commits on big repositories

STR:

clone https://hg.mozilla.org/mozilla-central
commit something like 40 dummy changes. In fact, 40 consecutive git commit --allow-empty -m foo should do the trick
push to e.g. https://hg.mozilla.org/try

Memory usage of the git-remote-hg process goes up to close to 4GB with or without the git-cinnabar-helper. That's way too much.

Allow to push to a specific remote branch

It is currently possible to push to a specific (possibly new) bookmark, and to push to the current branch, but it's not possible to push to a different (possibly new) branch.

Tags consolidation doesn't match mercurial's

See http://selenic.com/hg/file/8a544fb645bb/mercurial/tags.py#l163

"No module named mercurial" error after updating

I updated my git-cinnabar clone today from 4d96b8c (0.1.1) to 20de9a0 (0.2.2) and now when I run the git cinnabar command, I get the following error:

$ git cinnabar
Traceback (most recent call last):
  File "/usr/local/libexec/git-core/git-cinnabar", line 8, in <module>
    from cinnabar.githg import (
  File "/usr/local/libexec/git-core/pythonlib/cinnabar/githg.py", line 27, in <module>
    from mercurial import mdiff
ImportError: No module named mercurial

I tried to build and install the native helper like this:

$ git submodule update --init
$ ./configure
$ make
$ sudo make install

I have installed mercurial on my system using brew, and running export PYTHONPATH=/usr/local/Cellar/mercurial/3.4.1/lib/python2.7/site-packages enables me to run git cinnabar successfully.

Some `git cinnabar` subcommands do not work properly from a subdirectory

One for sure is bundle, which skips changes from other directories. The cinnabar-helper is also not working correctly for some things when invoked from a subdirectory.

Error pushing to try after rebase

I have a branch with one commit past inbound, plus a try commit:

$ git st
## breakpad-logging...inbound/branches/default/tip [ahead 2]

Earlier today, I pushed it to try successfully:

$ git push try
remote: waiting for lock on repository /repo/hg/mozilla/try held by 'hgssh1.dmz.scl3.mozilla.com:22878'
remote: got lock after 4 seconds
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 2 changesets with 1 changes to 8 files (+1 heads)
remote: Trying to insert into pushlog.
remote: Inserted into the pushlog db successfully.
remote:
remote: View your changes here:
remote:   https://hg.mozilla.org/try/rev/7cdcc5035e36
remote:   https://hg.mozilla.org/try/rev/acb41f277e14
remote:
remote: Follow the progress of your build on Treeherder:
remote:   https://treeherder.mozilla.org/#/jobs?repo=try&revision=acb41f277e14
To hg::ssh://hg.mozilla.org/try
 * [new branch]      HEAD -> branches/default/tip

The try run blew up because of other breaking changes, so I wanted to rebase and try again:

$ git pull --rebase
Reading 62 changesets
Reading 62 manifests
Reading and importing 233 files
Importing 62 manifests
Importing 62 changesets
From hg::https://hg.mozilla.org/integration/mozilla-inbound
   cce25c60..75e6c48 branches/default/tip -> inbound/branches/default/tip
<...the files...>
First, rewinding head to replay your work on top of it...
Applying: Bug 1166964 - Disable Breakpad INFO logging. r=ted

This removed my empty try commit, so I made another:

$ git commit --allow-empty -m 'try: -b do -p all -u all -t none'
[breakpad-logging b54c868] try: -b do -p all -u all -t none

But now pushing to try fails:

$ git push try
Traceback (most recent call last):
  File "/Users/jryans/projects/git-cinnabar/git-remote-hg", line 442, in <module>
fatal: expected ok/error, helper said '<generator object <genexpr> at 0x12b9160a0>'
    sys.exit(main(sys.argv[1:]))
  File "/Users/jryans/projects/git-cinnabar/git-remote-hg", line 364, in main
    branchmap.names())
  File "/Users/jryans/projects/git-cinnabar/cinnabar/hg.py", line 311, in push
    common = findcommon(repo, store, set(local_bases()))
  File "/Users/jryans/projects/git-cinnabar/cinnabar/hg.py", line 151, in findcommon
    known = repo.known(unhexlify(h) for h in hg_sample)
  File "/usr/local/lib/python2.7/site-packages/mercurial/wireproto.py", line 158, in plain
    encargsorres, encresref = batchable.next()
  File "/usr/local/lib/python2.7/site-packages/mercurial/wireproto.py", line 254, in known
    yield {'nodes': encodelist(nodes)}, f
  File "/usr/local/lib/python2.7/site-packages/mercurial/wireproto.py", line 176, in encodelist
    return sep.join(map(hex, l))
  File "/Users/jryans/projects/git-cinnabar/cinnabar/hg.py", line 151, in <genexpr>
    known = repo.known(unhexlify(h) for h in hg_sample)
TypeError: must be string or buffer, not None

Handle fuzzy copy/renames

Like issue #7, but for copy/renames followed by edits.

Unable to clone from repos which don't support 'getbundle'

$ git clone hg::https://vim.googlecode.com/hg vim
Cloning into 'vim'...
using https://vim.googlecode.com/hg
sending capabilities command
vim.googlecode.com certificate successfully verified
Traceback (most recent call last):
  File "/home/jamessan/src/github.com/git-cinnabar/git-remote-hg", line 650, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/home/jamessan/src/github.com/git-cinnabar/git-remote-hg", line 413, in main
    assert repo.capable('getbundle')
AssertionError

Does this support for win32?

git gc destroys cinnabar metadata

Every time after I run git gc, the cinnabar metadata gets corrupted. For example, when pushing, I get errors like:

Traceback (most recent call last):
  File "/Users/ehsan/src/git-remote-hg/git-remote-hg", line 433, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/Users/ehsan/src/git-remote-hg/git-remote-hg", line 284, in main
    getbundle(repo, store, heads, branchmap)
  File "/Users/ehsan/src/git-remote-hg/cinnabar/hg.py", line 225, in getbundle
    common = findcommon(repo, store, store.heads(branchmap.names()))
  File "/Users/ehsan/src/git-remote-hg/cinnabar/hg.py", line 154, in findcommon
    known = repo.known(unhexlify(h) for h in hg_sample)
  File "/usr/local/lib/python2.7/site-packages/mercurial/peer.py", line 80, in plain
    encargsorres, encresref = batchable.next()
  File "/usr/local/lib/python2.7/site-packages/mercurial/wireproto.py", line 210, in known
    yield {'nodes': encodelist(nodes)}, f
  File "/usr/local/lib/python2.7/site-packages/mercurial/wireproto.py", line 127, in encodelist
    return sep.join(map(hex, l))
  File "/Users/ehsan/src/git-remote-hg/cinnabar/hg.py", line 154, in <genexpr>
    known = repo.known(unhexlify(h) for h in hg_sample)
TypeError: must be string or buffer, not None
fatal: Could not read ref refs/cinnabar/refs/heads/branches/default/tip

$ git cinnabar fsck
Reading 2247270 mercurial to git mappings
Reading 314495 commit to changeset mappings
Reading 312621 manifest trees
Missing manifest commit in manifest branch: 6458c4e176193b1c151e90bc711de0cd8c3baa12
Missing manifest commit in manifest branch: 9b893ce0eee6aed1252bc73afd605ed76c55cfdd
Missing manifest commit in manifest branch: 11bca575472ffa48b2b86aabea24880c7bc61cd5
Missing manifest commit in manifest branch: f28b075c163b3f973e92ca7313481607a79cfd75
Missing manifest commit in manifest branch: c2b0c897b10ea1482a5b2e1b246c564189d07fc8
Missing manifest commit in manifest branch: ee9157759bc68cf42c69ea4c45e22f8ccef2a5f9
Missing manifest commit in manifest branch: 3c32ef6137505427d2d9025b4f5f66b40862ef6c
Missing manifest commit in manifest branch: 3d55b4ae0eee94f170da976ad1293afdfbf32471
Missing manifest commit in manifest branch: 2dfad54e73a77e6c7c6baa07319ecdac9a8b1c74
Missing manifest commit in manifest branch: a28ac59d07c1ea57778d9878f2e4567074534db3
Missing manifest commit in manifest branch: d56c6353ca09839582aa0802d2593afc01e2702d
Missing manifest commit in manifest branch: 77a2f5ec26d34021c923b6b72e2570eb037cf3a7
Missing manifest commit in manifest branch: c9bf6029f5d36819df68b862945cb1387c518f45
Missing manifest commit in manifest branch: 512f8c68fc21a645bcd5c791c3ba6e0e4b441d63
Missing manifest commit in manifest branch: d5d89e8908f5714f91475f0bdbd0728f93fe5cec
Missing manifest commit in manifest branch: e9b448c4e5530bd6e1b4452ba7acb2055dbc683e
Missing manifest commit in manifest branch: 5d612bbf79b1d2129e0b1742edf5a36e1405fd84
Missing manifest commit in manifest branch: a900bff68934d59e9253d5a79965ed45fa30db43
Missing manifest commit in manifest branch: a47eeac2fa7a2d92c514516cbaa6e7035bf167be
Missing manifest commit in manifest branch: aa56af328c8dd3096e1d7b121a3380e6d5c6fafc
Missing manifest commit in manifest branch: c78e09155b0ab13cf519f3357c11017e1543a927
Missing manifest commit in manifest branch: cd94858810e102ab674a4e561a6e47c02c88f160
Missing manifest commit in manifest branch: cd4d9c976b69422b898f73211cab4e4668f244b0
Missing manifest commit in manifest branch: e4f45ac381c6a798770883e01c80755fae902b3a
Missing manifest commit in manifest branch: 872e5503d52a64320322812a5f31ab75a162a8ab
Missing manifest commit in manifest branch: 812e3c13cee31f7a12d0f16daf13d5b03f1475ec
Missing manifest commit in manifest branch: ac6c95ec56b22f5c32ecbd10192cef27ba6e5c27
Missing manifest commit in manifest branch: 211aec3d7d9b4a583f6d0273a506df4a5f3ba6f5
Missing manifest commit in manifest branch: aa6b144be9de9a8a1856a1e9bc76304e47e9a1d7
Missing manifest commit in manifest branch: 9dfdbe07de6ced47dce2d0c212d0d5942bb99374
Missing manifest commit in manifest branch: bf31254b50b1793622c79980869d045c8a978728
Missing manifest commit in manifest branch: 42d12739db45137fba4458b48ab58b387ec4f319
Missing manifest commit in manifest branch: a55e5a24910fb1124b196daf206466b09ed3ca5c
Missing manifest commit in manifest branch: e36150b9084f10cd678ceb67074effd53ecde559
Missing manifest commit in manifest branch: 1beda7e14b4887fd8574275206d16e708dd8bcb5
Missing manifest commit in manifest branch: 8e0d2c3b80db552b48e9cd823ee85860a2b9ad9b
Missing manifest commit in manifest branch: 7977b20207119090ecb6d2737788a7e9d7ea9075
Missing manifest commit in manifest branch: b860997bd6931b4c0a0e6ac8421e3bf214904552
Missing manifest commit in manifest branch: d81b6512255d7baad79fa3bb8ed2054b7e956f41
Missing manifest commit in manifest branch: 96a41b5bbd5e6247833dd3f00e0619c50f0feea4
Missing manifest commit in manifest branch: 9d8df8388cddd0aec4d654e3c12430c14a970221
Missing manifest commit in manifest branch: 5d915674ee91e135a1ababc6f63834499fa3b2fd
Missing manifest commit in manifest branch: 6aa378a404e658f1e81eebb8deb4c79a7a687204
Missing manifest commit in manifest branch: 0b33c6e02a9ddb1e9d4ed99ea4ef1d6022ec93b5
Missing manifest commit in manifest branch: 9bab47b17d7bac083c0b20a87f93d461e8c355b3
Missing manifest commit in manifest branch: 82ec67197e635c68470c7e1a362691f1ab8025c9
Missing manifest commit in manifest branch: 38c0b019f6f039e6f9b5dcf464e4b37babee3f27
Missing manifest commit in manifest branch: 63f4d9098f52c67f783d2fbdc4109f734c2293d3
Missing manifest commit in manifest branch: 0e160c004c3be8f52a344b6a3dc9667b9e60451f
Missing manifest commit in manifest branch: 55392662e551778013dd64c066a67e4c4864bc83
Missing manifest commit in manifest branch: 2af96a0696981a227f4685b0162cf76a7282c275
Missing manifest commit in manifest branch: 15fa05d752308c58ff20e27ea88796addf09205b
Missing manifest commit in manifest branch: 14dde6a656fb67186e88c9f2bb361dc69133c315
Missing manifest commit in manifest branch: 9dfe97329af819cd998224899f341e5e1b308358
Missing manifest commit in manifest branch: 87f7ce6419a487849ea45faf6f065d82cdb6fa96
Missing manifest commit in manifest branch: f05ac8d248eb0f73ffd815ccbfc72e81158f6388
Missing manifest commit in manifest branch: 4d54ff2bd19887a1893c018a22451482a88704fe
Missing manifest commit in manifest branch: 727265a5a59e24fcd60169ab4920ac83289923a6
Missing manifest commit in manifest branch: 3bbf13b0c64d4667b34062303b00e6a0c3c97493
Missing manifest commit in manifest branch: 51b688829834af8ba45fb7b9f080b128666c74fa
Missing manifest commit in manifest branch: 0bbee13d985ebda763aa38de44e8662d836083a8
Missing manifest commit in manifest branch: 59d2d189d87c7601f4a71c01a3be9ed8482ab217
Missing manifest commit in manifest branch: 234b008a92d42a3a2d2e29bfb57e962fe08a671c
Missing manifest commit in manifest branch: f14a41d36646daa4495c4e99938d96a371a2e9c1
Missing manifest commit in manifest branch: 64d0b879112e78a785a080b6de8a640c77d108dd
Missing manifest commit in manifest branch: 80e2c59024650da681d157a4d270b141dc386770
Missing manifest commit in manifest branch: c192090db3db93d82578043429ec989dc0abcc27
Missing manifest commit in manifest branch: 2ec155cd5a39ba10292a6abefb66320f23ae4322
Missing manifest commit in manifest branch: d11895cce280128287b173f066c4043a0bc2bf80
Missing manifest commit in manifest branch: 0deea435bbb585ea10da1aae6c3641518f922e30
Missing manifest commit in manifest branch: 290609836b17649b4ef813877d5d42d1033fd7ca
Missing manifest commit in manifest branch: 92feb2a3ef353d3818fe4d4934501d0fe93abf37
Missing manifest commit in manifest branch: cd6edeaf426afe2ab71fe861401d3730c3c4dc30
Missing manifest commit in manifest branch: 86b90b17bbd5d151e87907c6b9609c3b3da4854f
Missing manifest commit in manifest branch: 19b4a62236ca70af57d0401e40bf1c5e6c35d3ac
Missing manifest commit in manifest branch: 2c6b5c5174e253683ef76a5c64aae01ab3209895
Missing manifest commit in manifest branch: 6b40521552cf4ff1d9c61a06607881b902b9ff22
Missing manifest commit in manifest branch: b8eed5065deeb30e3c2e86cd22ab78c12d23b16a
Missing manifest commit in manifest branch: e58fc4a1354729d76e4e7755891d6ff0065ed380
Missing note for git commit: 45420c4599fb7e7d8e198dfd163f5c1a46357251
Checking 314496 changesets
Removing metadata commit 722ff9fd60e1bc1f559f75aebb81604cd319afa3 with no hg2git entry
Removing metadata commit 92cebcedd319edfa9fac2098ff414e4423e08e8b with no hg2git entry
Removing metadata commit c1a289a9a0b895b2134eae13b89cb398a3b2864c with no hg2git entry
Removing metadata commit c547bff64ac746a9c3b71fe22203d99388e0dca9 with no hg2git entry
Removing metadata commit aa2a3e2188101b4300445aa632d545a2220b49f5 with no hg2git entry
Removing metadata commit d46f6fe065ba7d6098413d90da14b7c49da5e99c with no hg2git entry
Removing metadata commit 090dd2f953f2dee77bd9798dbff51570f6ebe904 with no hg2git entry
Removing metadata commit 416433f3d63773f8721249dd513e02ef494c8d82 with no hg2git entry
Removing metadata commit 35ab11a478d42dec5753b7d5d165d37b088c86ba with no hg2git entry
Removing metadata commit ed99d281a045d9f89fd76d251d204b438cfe5f56 with no hg2git entry
Removing metadata commit cb031e319446c439c74b18026c69bd5120e4c29c with no hg2git entry
Removing metadata commit 596921561d92adb8ae9d444f1200335c85ed3091 with no hg2git entry
Removing metadata commit e259d65eb0943e6c02071c7a4b899a7b54c7496b with no hg2git entry
Removing metadata commit 0bc666fc3bebd4e2f3c1a957a7d651e5d92dc72c with no hg2git entry
Removing metadata commit 55e59ef89506c7e19a6acda9db7b1377cf2fcdb3 with no hg2git entry
Removing metadata commit e875adf8dae15901f2ef35a899ab4b943b41cf3e with no hg2git entry
Removing metadata commit 85756ba1ad7cd1e7b55d8157b841383812e8f262 with no hg2git entry
Removing metadata commit 785fbbe47b91ed2e809c2aa6dbf318720aa31714 with no hg2git entry
Removing metadata commit 77c5437fe4101f80c594563a121e9c243b6a1738 with no hg2git entry
Removing metadata commit 6f13673f36eb8e1048b2c796ddc811c84ad8c4fa with no hg2git entry
Removing metadata commit c4deb6621b91442beb067e272752c251c37a85e7 with no hg2git entry
Removing metadata commit 562bc87c97e56b7c338f9c86c394df8f20923881 with no hg2git entry
Removing metadata commit 9d29c22af0db7016688f45dd3e9d8ac93c2a0fbd with no hg2git entry
Removing metadata commit 9930a049d5b5cb819495ed4fcddc4ce6dee768c1 with no hg2git entry
Removing metadata commit f632c31f596c2813147998a6a00d004260044a54 with no hg2git entry
Removing metadata commit bea0d5ad30420f52124845e69c7ce7443a406156 with no hg2git entry
Removing metadata commit 84dd4551158ba427147e65c6e910ef104660c26e with no hg2git entry
Removing metadata commit 9c361ec488d8bed44e00db6613028b197aac561b with no hg2git entry
Removing metadata commit f9a847d9d23c07c62f28e503ce968229c7533484 with no hg2git entry
Removing metadata commit 399584c9d7e597859faa7c0dc06c1c7a5252ce71 with no hg2git entry
Removing metadata commit 8a4837ab1c672764a2108006b17d87a6cd63cabf with no hg2git entry
Removing metadata commit 959107ae74f94d2c903fe1739d094b0da2fd1589 with no hg2git entry
Removing metadata commit f8e0d22faeded550eabe75f837f439926cb196d9 with no hg2git entry
Removing metadata commit 31a691d5bb07a1994ad03f253d019bede7973b11 with no hg2git entry
Removing metadata commit 4546a86ba6c0d9de230750e52441ebf8c022f6f7 with no hg2git entry
Removing metadata commit 91e56cb844ebb67b66a65dcded394fb91791c892 with no hg2git entry
Removing metadata commit ed7e9c9ca303433637f38d774f19967bbe4c4423 with no hg2git entry
Removing metadata commit 8e3386b3bb029f7894c4169b10d55dd18783e933 with no hg2git entry
Removing metadata commit ae0899f3da47da00b0c2a5aa65b2071d8462cb0c with no hg2git entry
Removing metadata commit 54b91ebabd99ec9b9b26358f846190e82fc317a7 with no hg2git entry
Removing metadata commit 0a582ba79e5be3cfe3d4c45139b7f36805978302 with no hg2git entry
Removing metadata commit d8d163bcc419698fb65aa04eb9ca1d21539d66f6 with no hg2git entry
Removing metadata commit c33bbb71b8fdcca3ac9892944bc7e1110446163b with no hg2git entry
Removing metadata commit c70e3995651d49d0d09d099af0daa88aa6470f3d with no hg2git entry
Removing metadata commit b22529555317b012314a1afeeb9a71aaa1e387d0 with no hg2git entry
Removing metadata commit 41d3968f8ccbfe6f8e259e97e37ab41880c1bfcd with no hg2git entry
Removing metadata commit 7d563cb491a6c17708e9f93e098fd532f4692062 with no hg2git entry
Removing metadata commit f73a26d7dbcd4d5966b486c08a1080cd04b82837 with no hg2git entry
Removing metadata commit 8bb085bd130421609797201839040a9baeef9bca with no hg2git entry
Removing metadata commit 9c5ab7d89ef22b46c1543f29e5f6e42acb7d9130 with no hg2git entry
Removing metadata commit a8e93e396f6d1fc873f84958a5ec9b4a49241a32 with no hg2git entry
Removing metadata commit 352d2ac5abe09a9286e9e967220afa0562c3bbbd with no hg2git entry
Removing metadata commit 9e6b6967326543d88b83e6153b724148bc9f53ed with no hg2git entry
Removing metadata commit afc528a8db22f4d9cef1e71c9e0f9e9812466147 with no hg2git entry
Removing metadata commit 6ec69177e94f6387f932ecc77f55ae3bda5553a6 with no hg2git entry
Removing metadata commit 6b3d5d334eaffdfbb3e1aaa0b256799969252b33 with no hg2git entry
Removing metadata commit 46201ca00eae3dbf90cc31caae1692e2ffdf94e3 with no hg2git entry
Removing metadata commit 1bd728a8c02c1c99895f50b38f865160690e1248 with no hg2git entry
Removing metadata commit 904958c090cbb5ebf3fc89412b8017385fcffa84 with no hg2git entry
Removing metadata commit 336b462738516c1d2cba7445990582fb7d992afa with no hg2git entry
Removing metadata commit 40036797cdd610bf54fd339379b299c5b9f69e28 with no hg2git entry
Removing metadata commit ec91e269825c5627383eceae6003b9f38511735d with no hg2git entry
Removing metadata commit 5d93afe0187b7dbbd946834f8ab410cc0032f00b with no hg2git entry
Removing metadata commit 8f996c216a7641e276bf4cdbf33cadf6b6159230 with no hg2git entry
Removing metadata commit c0ad878d94e0c62d978e88a14998240ae14d3b7a with no hg2git entry
Removing metadata commit c5be4b9dd98384ade180eff293688de0be9e8a69 with no hg2git entry
Removing metadata commit bda148cc5dcace62592723861dc99b1c8a34f5cf with no hg2git entry
Removing metadata commit a7a60eb4ed1e79895239a26adf5624f46f6c3099 with no hg2git entry
Removing metadata commit 9505596c554f9fd7e21a22491c58086970ebb6fc with no hg2git entry
Removing metadata commit d52a854dff8e7a8ec00131ec3ffe0902f678d7f8 with no hg2git entry
Removing metadata commit e554a64ae52082353d69550259edc44bd84aa0c5 with no hg2git entry
Removing metadata commit 49c0e64266530eaef641d25685dfb73aef189b23 with no hg2git entry
Removing metadata commit 248a5dd2549c8fe634589e02562f64269cc33a08 with no hg2git entry
Removing metadata commit 04a1818731d58cbab6ab892038c815b226cacf8e with no hg2git entry
Removing metadata commit 5503b181463cab75bba36ba1e56817edcc84001e with no hg2git entry
Removing metadata commit b94e09b0abaa66257861f81fbb1d816db12bb6f5 with no hg2git entry
Removing metadata commit 04dd25e1c0d0fefa09b6f56f90780bb860f14391 with no hg2git entry
Removing metadata commit 9df738b9b308f82adedbc9842748ffb778a48516 with no hg2git entry
Removing metadata commit ae970bf0f8a097f120cd0f3799eec75dcca539bc with no hg2git entry
Adjusting 71 metadata commits
Updating hg2git for 71 metadata commits
Your git-cinnabar repository appears to be corrupted. There
are known issues in older revisions that have been fixed.
Please try running the following command to reset:
  git cinnabar reclone

Please note this command may change the commit sha1s. Your
local branches will however stay untouched.
Please report any corruption that fsck would detect after a
reclone.
Checking head references...
Removing non-head reference to 99a801862ecd389d3917bfad0556a7122dbc78c7 in branch GECKO380_2015050320_RELBRANCH
Removing non-head reference to 04f89d19556d18323bdb581d0734bacebb13c2c9 in branch GECKO430b7_2015112614_RELBRANCH
Adding missing head c1b8d8e1188000587661eb9f842d5245e9f4fa2c in branch default
Removing non-head reference to c0abc2a6e11f52761366e029eb1bae4c9864a8a3 in branch default
Removing non-head reference to dd509db16a138beb777599662ad88e3207bc2d2c in branch default
Removing non-head reference to 87e23922be375985d0b1906ed5ba5f095f323a38 in branch default
Removing non-head reference to 9ddf0da90fb3bc1ae29966dc596013fc54a44bd2 in branch default
Removing non-head reference to ac70678c5f5eb8d7b4477001600b13d5770960ac in branch default
Removing non-head reference to 451a185791433bce1a6a894c27f3da60a3119431 in branch default
Removing non-head reference to 5d61714cf5c3ec2440e9491ba30893ac37ca06c2 in branch default
Removing non-head reference to 691b1524602d12afc2336474d8ddd4eff1476c4d in branch default

Doing a reclone fixes everything. Perhaps we're missing a ref?

Cannot fetch from the repo got through `git hgdebug reclone`

I ran git hgdebug reclone and then switched to my main repo, added a remote called hg to that with the path to my clone, and when I fetched, I got:

$ git fetch hg
warning: no common commits
remote: error: Could not read 3ec464b55782fb94dbbb9b5784aac141f3e3ac01
remote: fatal: Failed to traverse parents of commit eabda6aae98d14c71d7e7b95a66896868ff9500b
error: git upload-pack: git-pack-objects died with error.
fatal: git upload-pack: aborting due to possible repository corruption on the remote side.
remote: aborting due to possible repository corruption on the remote side.
fatal: protocol error: bad pack header

I'm currently running 39b3979, and my clone's tip sha1 is e709605b82b3dc85a13e86d27bec06f340b5e1a2. And here is the output of git hgdebug fsck:

$ git hgdebug fsck
Ignoring known sha1 mismatch for changeset ffdee4a4eb7fc7cae80dfc4cb2fe0c3178773dcf

error when building nokogiri

Not sure how these are related, but when I try to install a ruby gem with native extensions, it fails and the configure log mentions git cinnabar.

➜  engineering-blogs git:(master) ✗ gem install nokogiri -v '1.6.6.2'
...
Running 'configure' for libxml2 2.8.0... ERROR, review '/Users/Nicholas/.rvm/gems/ruby-2.0.0-p598/gems/nokogiri-1.6.0/ext/nokogiri/tmp/x86_64-apple-darwin14.3.0/ports/libxml2/2.8.0/configure.log' to see what happened.
...
➜  engineering-blogs git:(master) ✗ cat /Users/Nicholas/.rvm/gems/ruby-2.0.0-p598/gems/nokogiri-1.6.0/ext/nokogiri/tmp/x86_64-apple-darwin14.3.0/ports/libxml2/2.8.0/configure.log
configure: error: cannot find sources (entities.c) in /Users/Nicholas/mozilla/git-cinnabar or ..

lol, wut?

Improve fetch and push times when local repository has many heads

When you have a clone of e.g. mozilla-beta, which has around a thousand heads, and fetch or push to another repository (like mozilla-inbound or try), the number of round trips to determine the common changesets between local and remote is high enough that it can make the operation slow.

git clone hg::https://hg.mozilla.org/mozilla-central very slow due to mmap

I'm seeing multi-hour git clone operations. A little profiling with instruments reveals that all the time is being spent in mmap:

Cloning from scratch doesn't work on OSX

I get:

$ git clone hg::http://hg.mozilla.org/mozilla-central mozilla-central.git2
Cloning into 'mozilla-central.git2'...
Your git configuration has core.ignorecase set to "true".
Usually, this means git detected the file system is case insensitive.
Git-remote-hg does not support this setup.
Either use a case sensitive file system or run the following command:
  git config --global core.ignorecase false
If you do not want this configuration to be global, you cannot clone,
but you use the following setup:
  mkdir directory
  cd directory
  git init
  git config core.ignorecase false
  git remote add origin hg::hg_repo_url
  git remote update origin
If you already cloned, you should reclone from scratch with this version
of git-remote-hg.

And this is after I set that config var to false.

Allow not to store mercurial metadata in some cases

The typical case I have in mind is when pushing to Mozilla's try repository. Those are useless heads to keep locally forever, and it would be better not to store them at all.
This could be triggered by a pref, or by pushing a detached head or HEAD.

Allow to remove metadata for "draft" pushes

c0f74f5 made the default to not store metadata when pushing on non-publishing mercurial repositories.
In Mozilla workflows, this avoids creating permanent extra heads locally for things pushed to try. But all the previous pushes are still stored locally, and need to be garbage collected.

I also want to allow removing metadata for heads that are not in any remote repository (like, after removing a remote, or when the remote was rewritten (rare, but happens, see https://bugzilla.mozilla.org/show_bug.cgi?id=1142858 )

Push failed - repository changed while preparing changes - please try again

I got this when pushing to a freshly cloned buildbotcustom repo:

$ git push
Traceback (most recent call last):
  File "/Users/ehsan/src/git-remote-hg/git-remote-hg", line 469, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/Users/ehsan/src/git-remote-hg/git-remote-hg", line 456, in main
    pushed = push(repo, store, pushes, repo_heads, branchmap.keys())
  File "/Users/ehsan/src/git-remote-hg/git-remote-hg", line 269, in push
    ret = repo.unbundle(cg, repo_heads, '') != 0
  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mercurial/wireproto.py", line 388, in unbundle
    _('push failed:'), output)
mercurial.error.ResponseError: ('push failed:', 'repository changed while preparing changes - please try again')
error: failed to push some refs to 'hg::ssh://hg.mozilla.org/build/buildbotcustom'

Here is my git config file:

[core]
    repositoryformatversion = 0
    filemode = true
    bare = false
    logallrefupdates = true
    ignorecase = false
    precomposeunicode = true
[remote "origin"]
    url = hg::http://hg.mozilla.org/build/buildbotcustom
    pushurl = hg::ssh://hg.mozilla.org/build/buildbotcustom
    fetch = +refs/heads/*:refs/remotes/origin/*
  prune = true
[branch "tip"]
    remote = origin
    merge = refs/heads/tip

Setup container-based infrastructure for travis-ci builds

Something you might be interested in is container-based infrastructure:
https://docs.travis-ci.com/user/migrating-from-legacy/

It's easy to setup but gives a huge speed-up in terms how long it takes for your tests to start. Also caching of files is a nice feature.

cinnabar doesn't play well with hg bookmarks

I cloned ssh://hg.mozilla.org/hgcustom/version-control-tools/ and got a repo with the "bookmarks/@" branch by default. I have the following change:

commit 5c615b1336ea2ced2289d5c420a877b1540f6f1a (HEAD, master, refs/cinnabar/branches/default/099d04eceefe649996e5a3fd9c6dda7a98085bb6)
Author: Ehsan Akhgari <[email protected]>
Date:   Wed Apr 22 12:08:24 2015 -0400

    Add htsai as an alias to hsinyi

commit b3f1919abd12d13de548aecc8c7f3c92aee95e35 (origin/branches/default/tip, origin/bookmarks/@, origin/HEAD, bookmarks/@, refs/cinnabar/tip, refs/cinnabar/refs/heads/branches/default/tip, refs/cinnabar/refs/heads/bookmarks/@)
Author: Gregory Szorc <[email protected]>
Date:   Tue Apr 21 22:06:52 2015 -0700

    ansible: interact with load balancer with doing MozReview deployments

And when I try to push it, I get:

$ git push origin HEAD:branches/default/tip
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files (+1 heads)
remote:
remote:
remote: ************************** ERROR ****************************
remote: Multiple heads detected on branch 'default'
remote: Only one head per branch is allowed!
remote: *************************************************************
remote:
remote:
remote: transaction abort!
remote: rollback completed
remote: abort: pretxnchangegroup.singlehead hook failed
To hg::ssh://hg.mozilla.org/hgcustom/version-control-tools/
 ! [remote rejected] HEAD -> branches/default/tip (nothing changed on remote)
error: failed to push some refs to 'hg::ssh://hg.mozilla.org/hgcustom/version-control-tools/'

Fix remaining sha1 mismatches in mozilla repositories

There are currently two sha1 mismatches left in mozilla repositories, related to timezones with precision to the second.

TypeError during clone

$ git clone hg::https://gem5-gpu.cs.wisc.edu/repo/benchmarks/libopencl libopencl

Cloning into 'libopencl'...
progress 8 files
progress 4 manifests
Traceback (most recent call last):
File "/home/pioneeraxon/Development/git-remote-hg/git-remote-hg", line 469, in
sys.exit(main(sys.argv[1:]))
File "/home/pioneeraxon/Development/git-remote-hg/git-remote-hg", line 422, in main
getbundle(repo, store, heads, branchmap.keys())
File "/home/pioneeraxon/Development/git-remote-hg/git-remote-hg", line 235, in getbundle
store.git_tree(sha1)
File "/home/pioneeraxon/Development/git-remote-hg/githg/init.py", line 692, in git_tree
mode, typ, tree, path = one(Git.ls_tree(manifest_commit, 'git'))
TypeError: 'NoneType' object is not iterable
Checking connectivity... fatal: bad object 0000000000000000000000000000000000000000
fatal: remote did not send all necessary objects

Subrepository support

Subrepositories [0] are not supported.

This will be easy to set up using git submodules (Just recursively clone each hg:: from .hgsub file using "git submodule add").

[0] : http://mercurial.selenic.com/wiki/Subrepository

TypeError: must be string or buffer, not None when trying to push

Traceback (most recent call last):
  File "/Users/ehsan/src/git-remote-hg/git-remote-hg", line 650, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/Users/ehsan/src/git-remote-hg/git-remote-hg", line 611, in main
    pushed = push(repo, store, pushes, repo_heads, branchmap.names())
  File "/Users/ehsan/src/git-remote-hg/git-remote-hg", line 286, in push
    common = findcommon(repo, store, repo_heads, repo_branches)
  File "/Users/ehsan/src/git-remote-hg/git-remote-hg", line 146, in findcommon
    known = repo.known(unhexlify(h) for h in hg_sample)
  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mercurial/wireproto.py", line 158, in plain
    encargsorres, encresref = batchable.next()
  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mercurial/wireproto.py", line 250, in known
    yield {'nodes': encodelist(nodes)}, f
  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mercurial/wireproto.py", line 175, in encodelist
    return sep.join(map(hex, l))
  File "/Users/ehsan/src/git-remote-hg/git-remote-hg", line 146, in <genexpr>
    known = repo.known(unhexlify(h) for h in hg_sample)
TypeError: must be string or buffer, not None
error: failed to push some refs to 'hg::ssh://hg.mozilla.org/integration/mozilla-inbound'

I got this when I was trying to push ehsan/gecko-dev@a9a0d9a to inbound.

AttributeError: 'NoneType' object has no attribute 'get' when fetching mozilla-release

Fetching release
Traceback (most recent call last):
  File "/Users/ehsan/src/git-remote-hg/git-remote-hg", line 589, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/Users/ehsan/src/git-remote-hg/git-remote-hg", line 445, in main
    branchmap = BranchMap(store, branchmap, heads)
  File "/Users/ehsan/src/git-remote-hg/git-remote-hg", line 316, in __init__
    extra = store.read_changeset_data(sha1).get('extra')
AttributeError: 'NoneType' object has no attribute 'get'
error: Could not fetch release

heap-use-after-free in git-cinnabar-helper in cases involving moved directories