bozaro / git-lfs-migrate

Simple project for convert old repository for using git-lfs feature

Overview

Simple project for convert old repository for using git-lfs feature.

How to use

Run from binaries

For quick run you need:

• Install Java 1.8 or later

• Download binaries archive from: https://github.com/bozaro/git-lfs-migrate/releases/latest

• After unpacking archive you can run server executing:

  java -jar build/deploy/git-lfs-migrate.jar -s source-repo.git -d target-repo.git -l http://test:test@lfs-server/ "*.psd" "*.zip" "*.bin"

For example, you can convert bozaro/git-lfs-migrate to bozaro/git-lfs-migrate-converted by commands:

#!/bin/bash
# Clone original repository
git clone --mirror [email protected]:bozaro/git-lfs-migrate.git

# Convert repository with moving .md and .jar file to LFS
#
# Usage: <main class> [options] LFS file glob patterns
#   Options:
#     -c, --cache
#        Source repository
#        Default: .
#         --check-lfs
#      Check LFS server settings and exit
#      Default: false
#   * -d, --destination
#        Destination repository
#     -g, --git
#       GIT repository url (ignored with --lfs parameter)
#     -h, --help
#        Show help
#        Default: false
#     -l, --lfs
#        LFS server url (can be determinated by --git paramter)
#   * -s, --source
#        Source repository
#     -u, --upload-threads
#        HTTP upload thread count
#        Default: 4
#     -t, --write-threads
#        IO thread count
#        Default: 2
#     --glob-file
#        File containing glob patterns
java -jar git-lfs-migrate.jar \
     -s git-lfs-migrate.git \
     -d git-lfs-migrate-converted.git \
     -g [email protected]:bozaro/git-lfs-migrate-converted.git \
     "*.md" \
     "*.jar"

# Push coverted repository to new repository
cd git-lfs-migrate-converted.git
git fsck && git push --mirror [email protected]:bozaro/git-lfs-migrate-converted.git

After that you with have:

• New repository bozaro/git-lfs-migrate-converted
• All *.md and *.jar in this repository will stored in LFS storage
• All revisions on this repository will have modified or created .gitattributes file with new lines like:
  

*.md    filter=lfs diff=lfs merge=lfs -text
*.jar   filter=lfs diff=lfs merge=lfs -text

Supported Git url formats:

• https://user:[email protected]/foo/bar.git
• http://user:[email protected]/foo/bar.git
• git://user:[email protected]/foo/bar.git
• ssh://[email protected]/foo/bar.git
• [email protected]:foo/bar.git

Build from sources

To build from sources you need install JDK 1.8 or later and run build script.

For Linux:

./gradlew deployZip

For Windows:

call gradlew.bat deployZip

When build completes you can convert repository executing:

java -jar build/deploy/git-lfs-migrate.jar -s source-repo.git -d target-repo.git -l http://test:test@lfs-server/ "*.psd" "*.zip" "*.bin"

git-lfs-migrate's Issues

Reattach notes after migration

I have notes attached to the commits. After migration all notes are gone. How I can reconnect them back?
Thank you.

SimpleHttpConnectionManager warning

Oct 19, 2015 5:16:20 PM org.apache.commons.httpclient.SimpleHttpConnectionManager getConnectionWithTimeout
WARNING: SimpleHttpConnectionManager being used incorrectly. Be sure that HttpMethod.releaseConnection() is always called and that only one thread and/or method is using this connection manager at a time.

Is this related to the git lfs' migrate command?

Is this tool doing the same thing as the git lfs migrate? Or is it different?

If they are different, which one is the "best"/newest? Which one should be used?
If they are not different, shouldn't this be marked as deprecated?

Need flag for skip SSL certificate validatoin for self-signed certificates

Problem with git-lfs-migrate and BitBucket 4.11 with LFS support

Hello,

I'm trying to use git-lfs-migrate to migrate an internal Subversion repository to Git for use with Atlassian BitBucket 4.11. This repository has a lot of binary files and I'd like to remove them from the git history using git-lfs-migrate. I'm using Windows 7, Java 1.8.0_91, Git LFS 1.5.5, git 2.10.0.windows.1.

In the commands and output below, I've changed the repository name to repo.git, the BitBucket project to 'project', the BitBucket server name to somebitbucketserver.net and the BitBucket IP address to 111.111.111.111 for illustration purposes.

When I issue any of the following commands:
java -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git --lfs https://somebitbucketserver.net/myproject/repo.git/info/lfs --no-check-certificate --glob-file binaries.txt
java -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git --lfs https://somebitbucketserver.net --no-check-certificate --glob-file binaries.txt
java -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git --lfs https://user:[email protected]/myproject/repo.git/info/lfs --no-check-certificate --glob-file binaries.txt
java -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git --lfs https://user:[email protected] --no-check-certificate --glob-file binaries.txt

I always get this error:

[main] INFO git.lfs.migrate.Main - LFS server: Batch API request exception
org.apache.http.conn.HttpHostConnectException: Connect to somebitbucketserver.net:443 [somebitbucketserver.net/111.111.111.111] failed: Connection refused: connect
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at ru.bozaro.gitlfs.client.internal.HttpClientExecutor.executeMethod(HttpClientExecutor.java:26)
at ru.bozaro.gitlfs.client.Client.doRequest(Client.java:275)
at ru.bozaro.gitlfs.client.Client.lambda$postBatch$7(Client.java:118)
at ru.bozaro.gitlfs.client.Client.doWork(Client.java:252)
at ru.bozaro.gitlfs.client.Client.postBatch(Client.java:118)
at git.lfs.migrate.Main.checkLfsAuthenticate(Main.java:113)
at git.lfs.migrate.Main.main(Main.java:66)
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:337)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
... 17 more
[main] ERROR git.lfs.migrate.Main - LFS server: Invalid base URL

I suspect it has something to do with authentication because in order to view the repository via HTTPS in a browser, I have to authenticated using our third-party SSO service. However, repos are fully accessible via SSH. Which led me to try the following in a git bash session after I added my SSH key to ssh-agent:

java -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git -g ssh://[email protected]:7999/project/repo.git --glob-file binaries.txt --no-check-certificate
java -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git -g [email protected]:project/repo.git --glob-file binaries.txt --no-check-certificate
java -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git -g ssh://[email protected]:7999/project/repo.git --glob-file binaries.txt
java -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git -g [email protected]:project/repo.git --glob-file binaries.txt

These commands always gives this error:

[main] INFO git.lfs.migrate.Main - LFS server: Batch API request exception
java.io.IOException: Command returned with non-zero exit code 255: [ssh, [email protected]:7999, -oBatchMode=yes, -C, git-lfs-authenticate, project/repo.git, upload]
at ru.bozaro.gitlfs.client.auth.ExternalAuthProvider.getAuthUncached(ExternalAuthProvider.java:82)
at ru.bozaro.gitlfs.client.auth.CachedAuthProvider.getAuth(CachedAuthProvider.java:38)
at ru.bozaro.gitlfs.client.Client.doWork(Client.java:248)
at ru.bozaro.gitlfs.client.Client.postBatch(Client.java:118)
at git.lfs.migrate.Main.checkLfsAuthenticate(Main.java:113)
at git.lfs.migrate.Main.main(Main.java:66)
[main] ERROR git.lfs.migrate.Main - LFS server: Invalid base URL

I also tried:
java -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git --lfs ssh://[email protected]:7999/project/repo.git --glob-file binaries.txt --no-check-certificate

Which errors with:

Feb 02, 2017 5:22:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (org.apache.http.conn.UnsupportedSchemeException) caught when processing request to {}->ssh://somebitbucketserver.net:7999: ssh protocol is not supported
Feb 02, 2017 5:22:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->ssh://somebitbucketserver.net:7999
Feb 02, 2017 5:22:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (org.apache.http.conn.UnsupportedSchemeException) caught when processing request to {}->ssh://somebitbucketserver.net:7999: ssh protocol is not supported
Feb 02, 2017 5:22:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->ssh://somebitbucketserver.net:7999
Feb 02, 2017 5:22:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (org.apache.http.conn.UnsupportedSchemeException) caught when processing request to {}->ssh://somebitbucketserver.net:7999: ssh protocol is not supported
Feb 02, 2017 5:22:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->ssh://somebitbucketserver.net:7999
[main] INFO git.lfs.migrate.Main - LFS server: Batch API request exception
org.apache.http.conn.UnsupportedSchemeException: ssh protocol is not supported
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:108)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at ru.bozaro.gitlfs.client.internal.HttpClientExecutor.executeMethod(HttpClientExecutor.java:26)
at ru.bozaro.gitlfs.client.Client.doRequest(Client.java:275)
at ru.bozaro.gitlfs.client.Client.lambda$postBatch$7(Client.java:118)
at ru.bozaro.gitlfs.client.Client.doWork(Client.java:252)
at ru.bozaro.gitlfs.client.Client.postBatch(Client.java:118)
at git.lfs.migrate.Main.checkLfsAuthenticate(Main.java:113)
at git.lfs.migrate.Main.main(Main.java:66)
Feb 02, 2017 5:22:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (org.apache.http.conn.UnsupportedSchemeException) caught when processing request to {}->ssh://somebitbucketserver.net:7999: ssh protocol is not supported
Feb 02, 2017 5:22:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->ssh://somebitbucketserver.net:7999
Feb 02, 2017 5:22:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (org.apache.http.conn.UnsupportedSchemeException) caught when processing request to {}->ssh://somebitbucketserver.net:7999: ssh protocol is not supported
Feb 02, 2017 5:22:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->ssh://somebitbucketserver.net:7999
Feb 02, 2017 5:22:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (org.apache.http.conn.UnsupportedSchemeException) caught when processing request to {}->ssh://somebitbucketserver.net:7999: ssh protocol is not supported
Feb 02, 2017 5:22:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->ssh://somebitbucketserver.net:7999
[main] ERROR git.lfs.migrate.Main - LFS server: Invalid base URL

Is there some way to get this working other than resorting to filter-branch or cluttering my repo with .gitattributes files from bfg --convert-to-git-lfs?

Thanks,
Mike

JNI Exception: NoClassDefFoundError for ru/bozaro/gitlfs/client/auth/AuthProvider

Hi I'm trying to run the migration on my repository and am unfortunately getting the following error with the latest JDK (jdk1.8.0_92) on my Mac (10.11.3). Error occurs both when I download the precompiled jar or when I build myself. Thanks!

$ java -jar git-lfs-migrate.jar -s source.git -d dest.git -l <URL> "*.dae" "*.psd" "*.zip" "*.bin" "*.mov" "*.mp4" "*.mp3" "*.wav" "*.jpg" "*.png" "*.jpeg" "*.obj" "*.fbx" "*.pc2" "*.asset" "*.tif"

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: ru/bozaro/gitlfs/client/auth/AuthProvider
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
    at java.lang.Class.getMethod0(Class.java:3018)
    at java.lang.Class.getMethod(Class.java:1784)
    at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: ru.bozaro.gitlfs.client.auth.AuthProvider
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 7 more

java.lang.OutOfMemoryError: GC overhead limit exceeded

Hi,

When converting come really large repos, when the count of objects reaches 400K-500K, I get this error:

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceede
d
at sun.nio.cs.UTF_8.newDecoder(Unknown Source)
at org.eclipse.jgit.util.RawParseUtils.decode(RawParseUtils.java:1127)
at org.eclipse.jgit.util.RawParseUtils.decodeNoFallback(RawParseUtils.ja
va:1073)
at org.eclipse.jgit.util.RawParseUtils.decode(RawParseUtils.java:1033)
at org.eclipse.jgit.treewalk.TreeWalk.pathOf(TreeWalk.java:1291)
at org.eclipse.jgit.treewalk.AbstractTreeIterator.getEntryPathString(Abs
tractTreeIterator.java:462)
at git.lfs.migrate.GitConverter$4.getEntries(GitConverter.java:198)
at git.lfs.migrate.GitConverter$4.depends(GitConverter.java:221)
at git.lfs.migrate.Main.loadTaskGraph(Main.java:316)
at git.lfs.migrate.Main.processRepository(Main.java:166)
at git.lfs.migrate.Main.main(Main.java:86)

Disabling the GC overhead or using another GC algorithm gives the error about Heap size not enough. I keep increasing the heap size but I still keep getting it.

Thanks,
Waqas

Multi thread support causes problems with github lfs

Hello again! :)

Since multi thread support is added I'm getting:

Exception in thread "main" java.io.IOException: Error writing request body to server
    at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3478)
    at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3461)
    at git.lfs.migrate.GitConverter.upload(GitConverter.java:320)
    at git.lfs.migrate.GitConverter.access$300(GitConverter.java:26)
    at git.lfs.migrate.GitConverter$5.convert(GitConverter.java:254)
    at git.lfs.migrate.Main.lambda$processMultipleThreads$0(Main.java:123)
    at git.lfs.migrate.Main$$Lambda$1/1059072014.run(Unknown Source)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
    Suppressed: java.io.IOException: insufficient data written
        at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.close(HttpURLConnection.java:3500)
        at git.lfs.migrate.GitConverter.upload(GitConverter.java:322)
        ... 9 more

and

Exception in thread "main" java.net.SocketException: Unexpected end of file from server
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:792)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1535)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
    at git.lfs.migrate.GitConverter.upload(GitConverter.java:323)
    at git.lfs.migrate.GitConverter.access$300(GitConverter.java:26)
    at git.lfs.migrate.GitConverter$5.convert(GitConverter.java:254)
    at git.lfs.migrate.Main.lambda$processMultipleThreads$0(Main.java:123)
    at git.lfs.migrate.Main$$Lambda$1/1059072014.run(Unknown Source)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

I thinks this has something to do with API limitation on github or S3. The error occurs at random, sometimes 100 files can be uploaded, sometimes some thousands.

I would suggest to reduce it to 4 threads as default, seems to run stable here. (currently testing with a 250Mbit/s upload connection)

Greets
Manuel

Unsupported object type

Hello

I get this error everytime I try to migrate:

main] INFO git.lfs.migrate.Main -   convert revision: a4425df8143f2ad7932c1aa1ae501f236c60fad8 -> 8fcd0a90d35487aeac22e0c75e78440b6d08d5d9 (1092/1103)
Exception in thread "main" java.lang.IllegalStateException: Unsupported object type: bdc67f81d37831df3451fca5c496202e8f7b7403 (org.eclipse.jgit.revwalk.RevTag)
    at git.lfs.migrate.GitConverter.lambda$convert$0(GitConverter.java:75)
    at git.lfs.migrate.GitConverter$$Lambda$1/293907205.apply(Unknown Source)
    at java.util.HashMap.computeIfAbsent(HashMap.java:1118)
    at git.lfs.migrate.GitConverter.convert(GitConverter.java:60)
    at git.lfs.migrate.Main.processRepository(Main.java:69)
    at git.lfs.migrate.Main.main(Main.java:46)

Thanks
Manuel

Mapping of old sha1 to new sha1

After running git-lfs-migrate how to get mapping of old sha1 to new sha1 of all commits?

Also, what is the file git-lfs-migrate.mapdb? If it has old commit id to new commit id mapping how to dump it to text or csv file?

NoClassDefFoundError while Migrating existing repository data to LFS [Using git-lfs-migrate]

Hi,
I have completed following steps (https://github.com/git-lfs/git-lfs/wiki/Tutorial#migrating-existing-repository-data-to-lfs) to migrate existing repository data to LFS, mentioned here.

• Install Java 1.8 or later
• Download the latest binaries from here
• Do a mirror clone of the repository to rewrite: git clone --mirror [email protected]:bozaro/git-lfs-migrate.git

I ended up with NoClassDefFoundError while trying to run the following step

• Rewrite e.g. all *.mp4 video files in the repository:

Following is my full stack. Please note that, I have changed the .git file name and project name, due to security issues.

admin@admin-ThinkPad:~/MyProjects/MiProject_Repo_Mirror$ java -jar git-lfs-migrate.jar      -s miproject_android_studio.git      -d miproject_android_studio_lfs-migrate-converted.git      -g https://[email protected]/miproject/miproject_android_studio_lfs-migrate-converted.git      "*.sqlite"      "*.sqlite.zip"
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/http/client/HttpClient
	at java.lang.Class.getDeclaredMethods0(Native Method)
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
	at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
	at java.lang.Class.getMethod0(Class.java:3018)
	at java.lang.Class.getMethod(Class.java:1784)
	at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
	at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.http.client.HttpClient
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 7 more
admin@admin-ThinkPad:~/MyProjects/MiProject_Repo_Mirror$ 

Does not generate correct .gitattributes entry for filenames with spaces

git-lfs-migrate seems to generate a .gitattributes file with spaces in the file names left intact, causing "not a valid attribute name" errors upon cloning. Apparently it should replace spaces with [:space:] instead. See git-lfs/git-lfs#298 and git-lfs/git-lfs#327.

Repo size increased from 2 to 9 Gig

I am using the following versions:
Centos 6.5
git 2.0.5
git-lfs-migrate 0.2.5

I am trying to prepare a repository that is in Gerrit which has eight large files over 100 MB. So in order to migrate and push it to Github with LFS support, I used git-lfs-migrate. I noticed that the size of the converted repository has increased from 2 to 9 Gig. There is a new "lfs" subdirectory with 2 GB and "objects" subdirectory has increased from 2 to 7.4 Gig. Does anyone have an idea what might be the issue? Here are my commands:

$ git clone --mirror ssh://username@gerrit/repo.git
$ java -jar /home/user/git-lfs-migrate.jar -s repo.git -d repo-converted.git ".csv" ".xls" "*.rtf"
$ cd repo-converted.git
$ git fsck

Also, I noticed that all the commits in the converted repository have a new hash. I am guessing that this is an expected behavior.

Migration process gets stuck for a long time

I successfully converted a single repository, but since then, any other repository I try to convert exhibits the same behavior. It proceeds to the following step, then just hangs there for a very long time, even on small repositories with very few commits:

[main] INFO git.lfs.migrate.Main - LFS server: OK
[main] INFO git.lfs.migrate.Main - Converting object without dependencies in 2 threads...
[pool-2-thread-2] INFO git.lfs.migrate.Main -   processed: 1/9
[pool-2-thread-1] INFO git.lfs.migrate.Main -   processed: 436/618
[pool-2-thread-2] INFO git.lfs.migrate.Main -   processed: 541/618
[main] INFO git.lfs.migrate.Main -   processed: 618/618
[main] INFO git.lfs.migrate.Main - Converting object with dependencies in single thread...
[main] INFO git.lfs.migrate.Main -   processed: 1/121
[main] INFO git.lfs.migrate.Main -   processed: 121/121

straceing it doesn't reveal very much:

) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
futex(0x7f13b0aa29d0, FUTEX_WAIT, 1957, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
futex(0x7f13b0aa29d0, FUTEX_WAIT, 1957, NULL [main] INFO git.lfs.migrate.Main - Recreating refs...
[main] INFO git.lfs.migrate.Main -   convert ref: c62a3b3fd0c6424f18ee364e114b3e0fcf21c6d6 -> 4e96476725c1a43f7e25a8382544ed4a42dc11ba (HEAD)
[main] INFO git.lfs.migrate.Main -   convert ref: c62a3b3fd0c6424f18ee364e114b3e0fcf21c6d6 -> 4e96476725c1a43f7e25a8382544ed4a42dc11ba (refs/heads/master)
[main] INFO git.lfs.migrate.Main - Convert time: 204587
) = 0
exit_group(0)                           = ?
+++ exited with 0 +++

That might have something to do with java binaries? It doesn't use up any CPU, it's not waiting on IO, it just sleeps.

Support an arbitrary glob, not just a suffix

git-lfs can take an arbitrary glob path as an argument. In my case I need to track something like **/config/*.xml (only those XMLs, not all XMLs). Can you please upgrade git-lfs-migrate to take an LFS-compatible glob instead of a list of suffixes?

Uploading progress

Is there an option to add uploading progress? For a large repos it is not clear if the migration stuck or something is really happening.

attach old sha1 to the comment in new repo

Attach old sha1 to the comment in new repo.Show like below:
before converted:
commit 86af00687ffd455c3c5e54c67864f6a8fee8e3f0(old sha1)
Author: cjp [email protected]
Date: Wed Feb 22 18:00:38 2017 +0800

kernel->file: ignore some file

after converted:
commit 86af00687ffd455c3c5e54c67864f6a8fee8e3f1(new sha1)
Author: cjp [email protected]
Date: Wed Feb 22 18:00:38 2017 +0800

kernel->file: ignore some file
86af00687ffd455c3c5e54c67864f6a8fee8e3f0(old sha1)

fama23

Performance degradation on huge repositories

There are performance degradation on huge repositories (>250 000 objects).
Look like root cause is too big file count. Much better generate pack files per every 10 000 objects.

Example should specify -g repo should exist, other possibilities

This wasn't initially clear to me from the documentation.

Because this may not be completely obvious, I'd suggest saying to leave the .jar file in the directory in which it's unzipped, or to add the ./vendors/ subdir to classpath.

I think also a slightly more complex example would help clarify usage some.

java -jar 
      ./path/to/git-lfs-migrate.jar  \ #should be in same location as vendors 
      -s ./the/path/for/source-repo.git \
      -d ./and/path/for/destination-repo.git \ 
      -g [email protected]:username/public-repo-name.git \ #this empty repo should already exist 
        \ # ^-- for instance [email protected]:bozaro/my-new-repo.git 
     "*.pet" \ # and as many extensions as you like 
     "*.cat" \
     "*.dog"

ru.bozaro.gitlfs.client.exceptions.RequestException: ... - 422 (status code 422)

I'm trying to migrate from a gerrit server to github instance. Both are inside the company firewall. When I run the tool it make a lot of progress before throwing an exception and then hanging/sitting until I CTRL-C out of.

I clone the source repository and ran the following...

java                                                                    \
    -Xmx4g                                                              \
    -jar ~/projects/IS_Streams/DevOps/GHE/migrate/git-lfs-migrate.jar   \
    -s ~/projects/IS_Streams/DevOps/GHE/Streams/.git                    \
    -d ~/projects/IS_Streams/DevOps/GHE/Streams-Test/.git               \
    -g [email protected]:org/repo.git                                 \
    "*.zip" "*.tar.gz"

The last part of the output is

[main] INFO git.lfs.migrate.Main -   processed: 57497/57497, uploaded: 418/423
Exception in thread "main" java.util.concurrent.ExecutionException: ru.bozaro.gitlfs.client.exceptions.RequestException: https://media.github.corp.com/networks/8503/lfs/objects/62bd609d9b26461fcd7fcc5efe6dcbac1a1c46ff70a771a544bcf3a7a2ba00a5 - 422 (status code 422)
        at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:368)
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1906)
        at git.lfs.migrate.Main$HttpUploader.close(Main.java:360)
        at git.lfs.migrate.Main.processRepository(Main.java:156)
        at git.lfs.migrate.Main.main(Main.java:77)
Caused by: ru.bozaro.gitlfs.client.exceptions.RequestException: https://media.github.corp.com/networks/8503/lfs/objects/62bd609d9b26461fcd7fcc5efe6dcbac1a1c46ff70a771a544bcf3a7a2ba00a5 - 422 (status code 422)
        at ru.bozaro.gitlfs.client.Client.doRequest(Client.java:315)
        at ru.bozaro.gitlfs.client.Client.putObject(Client.java:238)
        at ru.bozaro.gitlfs.client.BatchUploader.lambda$objectTask$13(BatchUploader.java:66)
        at ru.bozaro.gitlfs.client.BatchUploader$$Lambda$22.0000000074005140.exec(Unknown Source)
        at ru.bozaro.gitlfs.client.internal.BatchWorker.processObject(BatchWorker.java:262)
        at ru.bozaro.gitlfs.client.internal.BatchWorker.lambda$submitTask$3(BatchWorker.java:223)
        at ru.bozaro.gitlfs.client.internal.BatchWorker$$Lambda$23.0000000074005F30.run(Unknown Source)
        at ru.bozaro.gitlfs.client.internal.BatchWorker.executeInPool(BatchWorker.java:298)
        at ru.bozaro.gitlfs.client.internal.BatchWorker.access$400(BatchWorker.java:34)
        at ru.bozaro.gitlfs.client.internal.BatchWorker$1.run(BatchWorker.java:317)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(Thread.java:785)

Let me know if I should gather more info or if there is anything I can do to help.

File ending case sensitive

I've just run the migration tool on a project with, among many others, *.dll specified as files to migrate to LFS. This worked for the majority of files, but there was one file somefilename.DLL which wasn't.

There was a warning about this being tracked, but not a pointer when I cloned the project.

This suggests that *.dll in the .gitattributes is handled in a case insensitive way by git lfs, but that git-lfs-migrate is case sensitive. It should probably behave the same way as git lfs and include any casing of the filters.

There is no way to convert multiple repositories (repository with submodules)

Created by email from Eli Smaga:

Hello,

How do I use your lfs-migrate tool to convert a repo and the repo's submodule so that all of the branch, tag, and submodule links are updated with the new SHA's?

You mention doing that in the following post, but I don't see a way to do that. When I clone with the --> mirror option it doesn't get any of the submodule information so I can't convert the repo and submodule at the same time.

git-lfs/git-lfs#326 (comment)

Thanks for your help,
Eli

Bitbucket LFS

I'm trying this out with the Bitbucket LFS Beta released today, but having some trouble (UUIDs in the output have been sanitised):

java -jar ~/git-lfs-migrate.jar -s source.git -d dest.git -g [email protected]:user/repo.git --check-lfs 
[main] INFO git.lfs.migrate.Main - LFS server: Batch API request exception
com.fasterxml.jackson.databind.exc.InvalidFormatException: Can not construct instance of java.net.URI from String value ("https://bitbucket.org/{}/{120898b2-99b3-40e2-932d-d9622b245ad4}/info/lfs"): not a valid textual representation, problem: Illegal character in path at index 22: https://bitbucket.org/{}/{120898b2-99b3-40e2-932d-d9622b245ad4}/info/lfs
 at [Source: [B@548a102f; line: 1, column: 267] (through reference chain: ru.bozaro.gitlfs.common.data.Link["href"])
        at com.fasterxml.jackson.databind.exc.InvalidFormatException.from(InvalidFormatException.java:74)
        at com.fasterxml.jackson.databind.DeserializationContext.weirdStringException(DeserializationContext.java:923)
        at com.fasterxml.jackson.databind.deser.std.FromStringDeserializer.deserialize(FromStringDeserializer.java:136)
        at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:490)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:465)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:380)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1123)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:298)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:133)
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3788)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2872)
        at ru.bozaro.gitlfs.client.auth.ExternalAuthProvider.getAuthUncached(ExternalAuthProvider.java:84)
        at ru.bozaro.gitlfs.client.auth.CachedAuthProvider.getAuth(CachedAuthProvider.java:38)
        at ru.bozaro.gitlfs.client.Client.doWork(Client.java:248)
        at ru.bozaro.gitlfs.client.Client.postBatch(Client.java:118)
        at git.lfs.migrate.Main.checkLfsAuthenticate(Main.java:98)
        at git.lfs.migrate.Main.main(Main.java:68)
Caused by: java.lang.IllegalArgumentException: Illegal character in path at index 22: https://bitbucket.org/{}/{120898b2-99b3-40e2-932d-d9622b245ad4}/info/lfs
        at java.net.URI.create(URI.java:852)
        at com.fasterxml.jackson.databind.deser.std.FromStringDeserializer$Std._deserialize(FromStringDeserializer.java:214)
        at com.fasterxml.jackson.databind.deser.std.FromStringDeserializer.deserialize(FromStringDeserializer.java:122)
        ... 14 more
Caused by: java.net.URISyntaxException: Illegal character in path at index 22: https://bitbucket.org/{}/{120898b2-99b3-40e2-932d-d9622b245ad4}/info/lfs
        at java.net.URI$Parser.fail(URI.java:2848)
        at java.net.URI$Parser.checkChars(URI.java:3021)
        at java.net.URI$Parser.parseHierarchical(URI.java:3105)
        at java.net.URI$Parser.parse(URI.java:3053)
        at java.net.URI.<init>(URI.java:588)
        at java.net.URI.create(URI.java:850)
        ... 16 more
[main] ERROR git.lfs.migrate.Main - LFS server: Invalid base URL

I also tried specifying the LFS URI (-l https://user:[email protected]/user/repo.git/info/lfs) instead of the git repo, but that produces a similar result, a little further down the stack:

java -jar ~/git-lfs-migrate.jar -s source.git -d dest.git -l https://user:[email protected]/user/repo.git/info/lfs --check-lfs
[main] INFO git.lfs.migrate.Main - LFS server: Batch API request exception
com.fasterxml.jackson.databind.exc.InvalidFormatException: Can not construct instance of java.net.URI from String value ("https://bitbucket.org/{}/{120898b2-99b3-40e2-932d-d9622b245ad4}/info/lfs/object/verify?upload_id=6ae885b3-6de4-458b-a770-b35e8ef0674a"): not a valid textual representation, problem: Illegal character in path at index 22: https://bitbucket.org/{}/{120898b2-99b3-40e2-932d-d9622b245ad4}/info/lfs/object/verify?upload_id=6ae885b3-6de4-458b-a770-b35e8ef0674a
 at [Source: org.apache.http.conn.EofSensorInputStream@29d89d5d; line: 1, column: 121] (through reference chain: ru.bozaro.gitlfs.common.data.BatchRes["objects"]->java.util.ArrayList[0]->ru.bozaro.gitlfs.common.data.BatchItem["actions"]->java.util.LinkedHashMap["verify"]->ru.bozaro.gitlfs.common.data.Link["href"])
        at com.fasterxml.jackson.databind.exc.InvalidFormatException.from(InvalidFormatException.java:74)
        at com.fasterxml.jackson.databind.DeserializationContext.weirdStringException(DeserializationContext.java:923)
        at com.fasterxml.jackson.databind.deser.std.FromStringDeserializer.deserialize(FromStringDeserializer.java:136)
        at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:490)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:465)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:380)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1123)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:298)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:133)
        at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBind(MapDeserializer.java:436)
        at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:344)
        at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:26)
        at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:490)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:465)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:380)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1123)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:298)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:133)
        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:277)
        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:249)
        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:26)
        at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:490)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:465)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:380)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1123)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:298)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:133)
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3788)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2849)
        at ru.bozaro.gitlfs.client.internal.JsonPost.processResponse(JsonPost.java:52)
        at ru.bozaro.gitlfs.client.Client.doRequest(Client.java:311)
        at ru.bozaro.gitlfs.client.Client.lambda$postBatch$7(Client.java:118)
        at ru.bozaro.gitlfs.client.Client.doWork(Client.java:252)
        at ru.bozaro.gitlfs.client.Client.postBatch(Client.java:118)
        at git.lfs.migrate.Main.checkLfsAuthenticate(Main.java:98)
        at git.lfs.migrate.Main.main(Main.java:68)
Caused by: java.lang.IllegalArgumentException: Illegal character in path at index 22: https://bitbucket.org/{}/{120898b2-99b3-40e2-932d-d9622b245ad4}/info/lfs/object/verify?upload_id=6ae885b3-6de4-458b-a770-b35e8ef0674a
        at java.net.URI.create(URI.java:852)
        at com.fasterxml.jackson.databind.deser.std.FromStringDeserializer$Std._deserialize(FromStringDeserializer.java:214)
        at com.fasterxml.jackson.databind.deser.std.FromStringDeserializer.deserialize(FromStringDeserializer.java:122)
        ... 33 more
Caused by: java.net.URISyntaxException: Illegal character in path at index 22: https://bitbucket.org/{}/{120898b2-99b3-40e2-932d-d9622b245ad4}/info/lfs/object/verify?upload_id=6ae885b3-6de4-458b-a770-b35e8ef0674a
        at java.net.URI$Parser.fail(URI.java:2848)
        at java.net.URI$Parser.checkChars(URI.java:3021)
        at java.net.URI$Parser.parseHierarchical(URI.java:3105)
        at java.net.URI$Parser.parse(URI.java:3053)
        at java.net.URI.<init>(URI.java:588)
        at java.net.URI.create(URI.java:850)
        ... 35 more
[main] ERROR git.lfs.migrate.Main - LFS server: Unsupported batch API

I'm not sure how the URI is constructed, but it's obviously barfing on the brackets { }, which are not encoded. The repos do work using the standard git-lfs client, but it looks to me like perhaps some of the API responses from Bitbucket are not correctly populated - I suspect occurrences of {} are supposed to contain the user name, and occurrences of {120898b2-99b3-40e2-932d-d9622b245ad4} are supposed to contain the repo name, though I'm surprised that the standard client works if that's the case. Thoughts?

Not working with VSO

Hi there,

It seems that this tool cannot work with VSO hosted Git repos.

• Git Version: 2.11.0
• Git Lfs Version: 2.0.1
• Git-Lfs-Migrate version: latest

The error message are shown below:

[main] INFO git.lfs.migrate.Main -   processed: 4647/4647
Exception in thread "main" java.util.concurrent.ExecutionException: org.apache.http.client.ClientProtocolException
        at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
        at git.lfs.migrate.Main$HttpUploader.close(Main.java:359)
        at git.lfs.migrate.Main.processRepository(Main.java:170)
        at git.lfs.migrate.Main.main(Main.java:84)
Caused by: org.apache.http.client.ClientProtocolException
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at ru.bozaro.gitlfs.client.internal.HttpClientExecutor.executeMethod(HttpClientExecutor.java:26)
        at ru.bozaro.gitlfs.client.Client.doRequest(Client.java:275)
        at ru.bozaro.gitlfs.client.Client.putObject(Client.java:238)
        at ru.bozaro.gitlfs.client.BatchUploader.lambda$objectTask$13(BatchUploader.java:66)
        at ru.bozaro.gitlfs.client.internal.BatchWorker.processObject(BatchWorker.java:262)
        at ru.bozaro.gitlfs.client.internal.BatchWorker.lambda$submitTask$3(BatchWorker.java:223)
        at ru.bozaro.gitlfs.client.internal.BatchWorker.executeInPool(BatchWorker.java:298)
        at ru.bozaro.gitlfs.client.internal.BatchWorker.access$400(BatchWorker.java:34)
        at ru.bozaro.gitlfs.client.internal.BatchWorker$1.run(BatchWorker.java:317)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.http.ProtocolException: Transfer-encoding header already present
        at org.apache.http.protocol.RequestContent.process(RequestContent.java:93)
        at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:182)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
        ... 15 more

I tried to ignore this error. After that, I cannot checkout from the converted repo. Error message shown below:

$ git checkout -f
Downloading SMLPuller/lib/bond-3.0.jar (117.25 KB)
Error downloading object: SMLPuller/lib/bond-3.0.jar (da67920233505056889f412e32bb3b91af5a16dfc513a1e37e3264948828681c)

Errors logged to /tmp/test-target/.git/lfs/objects/logs/20170323T163817.1322287.log
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: SMLPuller/lib/bond-3.0.jar: smudge filter lfs failed

The content of /tmp/test-target/.git/lfs/objects/logs/20170323T163817.1322287.log is shown below:

cat /tmp/test-target/.git/lfs/objects/logs/20170323T163817.1322287.log
git-lfs/2.0.1 (GitHub; linux amd64; go 1.8)
git version 2.11.0

$ git-lfs filter-process
Error downloading object: SMLPuller/lib/bond-3.0.jar (da67920233505056889f412e32bb3b91af5a16dfc513a1e37e3264948828681c)

Smudge error: Error downloading SMLPuller/lib/bond-3.0.jar (da67920233505056889f412e32bb3b91af5a16dfc513a1e37e3264948828681c): batch response: Post /tmp/target.git/info/lfs/objects/batch: unsupported protocol scheme ""
github.com/git-lfs/git-lfs/errors.newWrappedError
        /tmp/docker_run/src/github.com/git-lfs/git-lfs/amd64/obj-x86_64-linux-gnu/src/github.com/git-lfs/git-lfs/errors/types.go:166
github.com/git-lfs/git-lfs/errors.NewSmudgeError
        /tmp/docker_run/src/github.com/git-lfs/git-lfs/amd64/obj-x86_64-linux-gnu/src/github.com/git-lfs/git-lfs/errors/types.go:252
github.com/git-lfs/git-lfs/lfs.PointerSmudge
        /tmp/docker_run/src/github.com/git-lfs/git-lfs/amd64/obj-x86_64-linux-gnu/src/github.com/git-lfs/git-lfs/lfs/pointer_smudge.go:68
github.com/git-lfs/git-lfs/lfs.(*Pointer).Smudge
        /tmp/docker_run/src/github.com/git-lfs/git-lfs/amd64/obj-x86_64-linux-gnu/src/github.com/git-lfs/git-lfs/lfs/pointer.go:64
github.com/git-lfs/git-lfs/commands.smudge
        /tmp/docker_run/src/github.com/git-lfs/git-lfs/amd64/obj-x86_64-linux-gnu/src/github.com/git-lfs/git-lfs/commands/command_smudge.go:63
github.com/git-lfs/git-lfs/commands.filterCommand
        /tmp/docker_run/src/github.com/git-lfs/git-lfs/amd64/obj-x86_64-linux-gnu/src/github.com/git-lfs/git-lfs/commands/command_filter_process.go:65
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).execute
        /tmp/docker_run/src/github.com/git-lfs/git-lfs/amd64/obj-x86_64-linux-gnu/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:477
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).Execute
        /tmp/docker_run/src/github.com/git-lfs/git-lfs/amd64/obj-x86_64-linux-gnu/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:551
github.com/git-lfs/git-lfs/commands.Run
        /tmp/docker_run/src/github.com/git-lfs/git-lfs/amd64/obj-x86_64-linux-gnu/src/github.com/git-lfs/git-lfs/commands/run.go:68
main.main
        /tmp/docker_run/src/github.com/git-lfs/git-lfs/amd64/obj-x86_64-linux-gnu/src/github.com/git-lfs/git-lfs/git-lfs.go:35
runtime.main
        /usr/local/go/src/runtime/proc.go:185
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:2197

ENV:
LocalWorkingDir=/tmp/test-target
LocalGitDir=/tmp/test-target/.git
LocalGitStorageDir=/tmp/test-target/.git
LocalMediaDir=/tmp/test-target/.git/lfs/objects
LocalReferenceDir=
TempDir=/tmp/test-target/.git/lfs/tmp
ConcurrentTransfers=3
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic
UploadTransfers=basic
GIT_DIR=.git
GIT_PREFIX=

ssh protocol for LFS

I'm trying to migrate to LFS with ssh protocol but instead I get an error:
ssh protocol is not supported
How to make it working? Thank you.

Batch request size is zero on retry.

If the initial conversion fails and a retry is attempted, the requests are all made with a "size": 0

This seems to be because data is saved into git-lfs-migrate.mapdb. When that data is reused in git.lfs.migrate.GitConverter#createRemoteFile the main body of that function is skipped so the size never gets set to anything other than zero.

Bitbucket.org's implementation of LFS asserts that the uploaded file matches in both OID and size.

LFS server: Invalid base URL when converting bit bucket repo

I am trying to convert an existing 1.8GB Bit Bucket repo to another BB repo with LFS for all the binary files.

I am running the command:

$ java -jar git-lfs-migrate.jar \
>      -s source-repo.git \
>      -d destination-repo.git \
>      -g [email protected]:companyname/destination-repo.git \
>      "*.md" \
>      "*.jar" \
>      "*.PcbLib" \
>      "*.SchLib" \
>      "*.zip" \
>      "*.EDF" \
>      "*.PDF" \
>      "*.xls" \
>      "*.xlxs" \
>      "*.step" \
>      "*.docx" \
>      "*.exe" \
>      "*.bin" \
>      "*.jar" \
>      "*.war"

And I get the following error:

[main] INFO git.lfs.migrate.Main - LFS server: Batch API request exception
com.fasterxml.jackson.databind.exc.InvalidFormatException: Can not construct instance of java.net.URI from String value ("https://bitbucket.org/{}/{dc5f1f05-e4eb-430f-8d06-79bbdc00712f}/info/lfs"): not a valid textual representation, problem: Illegal character in path at index 22: https://bitbucket.org/{}/{dc5f1f05-e4eb-430f-8d06-79bbdc00712f}/info/lfs
 at [Source: [B@5ab9e72c; line: 1, column: 259] (through reference chain: ru.bozaro.gitlfs.common.data.Link["href"])
        at com.fasterxml.jackson.databind.exc.InvalidFormatException.from(InvalidFormatException.java:74)
        at com.fasterxml.jackson.databind.DeserializationContext.weirdStringException(DeserializationContext.java:923)
        at com.fasterxml.jackson.databind.deser.std.FromStringDeserializer.deserialize(FromStringDeserializer.java:136)
        at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:490)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:465)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:380)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1123)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:298)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:133)
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3788)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2872)
        at ru.bozaro.gitlfs.client.auth.ExternalAuthProvider.getAuthUncached(ExternalAuthProvider.java:84)
        at ru.bozaro.gitlfs.client.auth.CachedAuthProvider.getAuth(CachedAuthProvider.java:38)
        at ru.bozaro.gitlfs.client.Client.doWork(Client.java:248)
        at ru.bozaro.gitlfs.client.Client.postBatch(Client.java:118)
        at git.lfs.migrate.Main.checkLfsAuthenticate(Main.java:113)
        at git.lfs.migrate.Main.main(Main.java:66)
Caused by: java.lang.IllegalArgumentException: Illegal character in path at index 22: https://bitbucket.org/{}/{dc5f1f05-e4eb-430f-8d06-79bbdc00712f}/info/lfs
        at java.base/java.net.URI.create(URI.java:883)
        at com.fasterxml.jackson.databind.deser.std.FromStringDeserializer$Std._deserialize(FromStringDeserializer.java:214)
        at com.fasterxml.jackson.databind.deser.std.FromStringDeserializer.deserialize(FromStringDeserializer.java:122)
        ... 14 more
Caused by: java.net.URISyntaxException: Illegal character in path at index 22: https://bitbucket.org/{}/{dc5f1f05-e4eb-430f-8d06-79bbdc00712f}/info/lfs
        at java.base/java.net.URI$Parser.fail(URI.java:2915)
        at java.base/java.net.URI$Parser.checkChars(URI.java:3086)
        at java.base/java.net.URI$Parser.parseHierarchical(URI.java:3168)
        at java.base/java.net.URI$Parser.parse(URI.java:3116)
        at java.base/java.net.URI.<init>(URI.java:600)
        at java.base/java.net.URI.create(URI.java:881)
        ... 16 more
[main] ERROR git.lfs.migrate.Main - LFS server: Invalid base URL

If I run the migration command without the -g line, it will run without error. But when I push it and then try a checkout on another machine, I get tons of errors that objects are missing from server. So it sounds like I really need to figure out the -g line.

What is causing the LFS string to be empty (and filled with curly braces) and what do I need to do to get it returning a proper string that will work?

Thanks in advance!

Convert zero length file

git lfs command does not convert zero length file to a pointer file
ref. https://github.com/github/git-lfs/blob/cd68f26c103b9ad474bd569c98a5e490e696b544/test/test-zero-len-file.sh#L37

But, git-lfs-migrate(v0.2.2) converts it.

$ mkdir git-lfs-migrate-test
$ cd git-lfs-migrate-test/
$ git init
$ touch empty.dat
$ git add empty.dat
$ git commit -m 'add files'
$ cd ..

$ git clone --mirror git-lfs-migrate-test git-lfs-migrate-test.git
$ java -jar git-lfs-migrate.jar \
     -s git-lfs-migrate-test.git \
     -d git-lfs-migrate-test-converted.git \
     "*.dat"
$ cd git-lfs-migrate-test-converted.git
$ git log -p | grep empty.dat -A 10
diff --git a/empty.dat b/empty.dat
new file mode 100644
index 0000000..fc8e42b
--- /dev/null
+++ b/empty.dat
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
+size 0

When I push it to github, I can't clone it.

$ git lfs clone https://github.com/kna/git-lfs-migrate-test-converted
Cloning into 'git-lfs-migrate-test-converted'...
remote: Counting objects: 4, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 4 (delta 0), reused 4 (delta 0), pack-reused 0
Unpacking objects: 100% (4/4), done.
Checking connectivity... done.
Git LFS: (1 of 1 files) 0 B / 0 B                                                                                                                                     Skipped checkout for empty.dat, content not local. Use fetch to download.

Errors logged to /tmp/git-lfs-migrate-test-converted/.git/lfs/objects/logs/20161014T121953.084344008.log
Use `git lfs logs last` to view the log.


$ cat /tmp/git-lfs-migrate-test-converted/.git/lfs/objects/logs/20161014T121953.084344008.log
git-lfs/1.4.2 (GitHub; linux amd64; go 1.7)
git version 1.9.1

$ git-lfs clone https://github.com/kna/git-lfs-migrate-test-converted
Skipped checkout for empty.dat, content not local. Use fetch to download.

smudge: Error
github.com/github/git-lfs/errors.newWrappedError
        /tmp/docker_run/src/github.com/github/git-lfs/amd64/obj-x86_64-linux-gnu/src/github.com/github/git-lfs/errors/types.go:166
github.com/github/git-lfs/errors.NewDownloadDeclinedError
        /tmp/docker_run/src/github.com/github/git-lfs/amd64/obj-x86_64-linux-gnu/src/github.com/github/git-lfs/errors/types.go:317
github.com/github/git-lfs/lfs.PointerSmudge
        /tmp/docker_run/src/github.com/github/git-lfs/amd64/obj-x86_64-linux-gnu/src/github.com/github/git-lfs/lfs/pointer_smudge.go:62
github.com/github/git-lfs/lfs.PointerSmudgeToFile
        /tmp/docker_run/src/github.com/github/git-lfs/amd64/obj-x86_64-linux-gnu/src/github.com/github/git-lfs/lfs/pointer_smudge.go:27
github.com/github/git-lfs/commands.checkoutWithChan
        /tmp/docker_run/src/github.com/github/git-lfs/amd64/obj-x86_64-linux-gnu/src/github.com/github/git-lfs/commands/command_checkout.go:194
github.com/github/git-lfs/commands.checkoutFromFetchChan.func1
        /tmp/docker_run/src/github.com/github/git-lfs/amd64/obj-x86_64-linux-gnu/src/github.com/github/git-lfs/commands/command_checkout.go:70
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:2086

ENV:
LocalWorkingDir=/tmp/git-lfs-migrate-test-converted
LocalGitDir=/tmp/git-lfs-migrate-test-converted/.git
LocalGitStorageDir=/tmp/git-lfs-migrate-test-converted/.git
LocalMediaDir=/tmp/git-lfs-migrate-test-converted/.git/lfs/objects
LocalReferenceDir=
TempDir=/tmp/git-lfs-migrate-test-converted/.git/lfs/tmp
ConcurrentTransfers=3
TusTransfers=false
BasicTransfersOnly=false
BatchTransfer=true
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic
UploadTransfers=basic

git-lfs-migrate should not convert zero length file to a pointer file.

JFrog Artifactory git-lfs upload failure

Hi,

by testing git-lfs-migrate.jar on your repository by using Artifactory as lfs-server the process runs through but no data is uploaded to Artifactory.
I used the following script

#!/bin/bash
java -jar java/git-lfs-migrate.jar \
     -s git-lfs-migrate.git \
     -d git-lfs-migrate-converted.git \
     -l http://<user>:<pwd>@<server>/artifactory/api/lfs/<lfs-repo> \
     "*.md" "*.jar"

and the output was

[main] INFO git.lfs.migrate.Main - LFS server: OK
[main] INFO git.lfs.migrate.Main - Reading full objects list...
[main] INFO git.lfs.migrate.Main -   found: 1
[main] INFO git.lfs.migrate.Main -   found: 601
[main] INFO git.lfs.migrate.Main - Found objects: 601
[main] INFO git.lfs.migrate.Main - Converting object without dependencies in 2 threads...
[pool-2-thread-2] INFO git.lfs.migrate.Main -   processed: 1/140, uploaded: 0/0
[main] INFO git.lfs.migrate.Main -   processed: 140/140, uploaded: 17/17
[main] INFO git.lfs.migrate.Main - Converting graph in single thread...
[main] INFO git.lfs.migrate.Main -   completed: 1/461
[main] INFO git.lfs.migrate.Main -   completed: 461/461
[main] INFO git.lfs.migrate.Main - Recreating refs...
[main] INFO git.lfs.migrate.Main -   convert ref: 2145c973f03617cd07361717785fd58604c94419 -> c176ab2c82ee7a780d2c193a70e8fa79ac2e9f89 (HEAD)
[main] INFO git.lfs.migrate.Main -   convert ref: 31edfcd65064468d35e66f4018c3c91af728601d -> 43430174791681f653fb7fe0651e929ae0a0eecd (refs/heads/globs)
[main] INFO git.lfs.migrate.Main -   convert ref: 2145c973f03617cd07361717785fd58604c94419 -> c176ab2c82ee7a780d2c193a70e8fa79ac2e9f89 (refs/heads/master)
[main] INFO git.lfs.migrate.Main -   convert ref: 5e19ff822c082878c5217af5dd95e15abe15ae76 -> 556239f2cf1f3357f239b1055fc4e3ebb8ec67fd (refs/pull/15/head)
[main] INFO git.lfs.migrate.Main -   convert ref: 7186b36f2d69c756e9ae8fdde9eb07d2ad53d8f7 -> ba0e59f7f2e28e647c14fa9fecb7f85a410844cd (refs/pull/16/head)
[main] INFO git.lfs.migrate.Main -   convert ref: 5c7486fb65b0ada289b481ce7262d7945b1d5994 -> 4b461df37747873ca75036e992d4622c0beea83d (refs/pull/19/head)
[main] INFO git.lfs.migrate.Main -   convert ref: 3125deb04614b6b5a2d4607794d92b282a2c32f3 -> 176a60a55d62f0be03343da5ae789f4e2e26491f (refs/pull/22/head)
[main] INFO git.lfs.migrate.Main -   convert ref: bd2a5582dfa38143efadb9e75dbb6b658adb53fd -> 0c38a82f20a7b28398130998eee32327098258aa (refs/pull/22/merge)
[main] INFO git.lfs.migrate.Main -   convert ref: 7b44b48351d3422a6a30b6c4d6086387a200b794 -> eeb20695c4fececf9be7984823c2d39be8d23b31 (refs/pull/6/head)
[main] INFO git.lfs.migrate.Main -   convert ref: ea0e304cc0ccdef4f21213a4167fb2337f65a069 -> 8991fd979168f213a094c3e6c20fcbd4f8e865cc (refs/pull/7/head)
[main] INFO git.lfs.migrate.Main -   convert ref: 6c3cd1def3c4877022e29d0a0de4a6ec0485874e -> 2d6bcf34b69ef2bc5e3c86445809854387565c09 (refs/pull/9/head)
[main] INFO git.lfs.migrate.Main -   convert ref: 005e6e010617e37be2bf25eeeddf50bac8f09a6a -> a6b9aafcf637a51496d174531f2a1cb06288de6c (refs/tags/0.0.1)
[main] INFO git.lfs.migrate.Main -   convert ref: 2a64aa0c7796b5fa324ab7ccfb049685b52bdf67 -> a06ba123abccada0e5c691b6ef2c4e16cd73f1d7 (refs/tags/0.0.10)
[main] INFO git.lfs.migrate.Main -   convert ref: 52fcfa20c8b14b99f001812cd93e5ea1eb476408 -> 676d0bea5c721ad57d115251a5f8e48cc5c9193e (refs/tags/0.0.11)
[main] INFO git.lfs.migrate.Main -   convert ref: 7b6a43f12116f81dafb85cc03b1a622cfbb36362 -> ff54ae8f856128747c0a9b53725f2cef5349b19a (refs/tags/0.0.12)
[main] INFO git.lfs.migrate.Main -   convert ref: fce590df6f4f86b03f42de4a69a7d1a40fd06fc0 -> e63e71506bbf737f43810a9311184446e4a2b09e (refs/tags/0.0.13)
[main] INFO git.lfs.migrate.Main -   convert ref: d2fd13d26c2cfd2e81936221bc4d04e982172a34 -> b18354ec9137522b11305f13e34c146168fcc13e (refs/tags/0.0.14)
[main] INFO git.lfs.migrate.Main -   convert ref: 61243791234be1cb92ad45ed3961ccea8ca6bbc0 -> 095fefc0bef3da728a289bc606507db21d515415 (refs/tags/0.0.15)
[main] INFO git.lfs.migrate.Main -   convert ref: ebc13e8293ff20b77a19e2d212e48435b6a1035a -> f733468cf7c1d1881dd1ede33a90f83b0862b668 (refs/tags/0.0.2)
[main] INFO git.lfs.migrate.Main -   convert ref: bdddc4246b3e31b3a645e1973ff59679c407a5f1 -> 657fe828f8c5385ada6a0a80919f3757689912ef (refs/tags/0.0.3)
[main] INFO git.lfs.migrate.Main -   convert ref: dbac21c2c1a0c6d600ee5bde6e60c7f510d3f82b -> 6a1a1a6b21f4b2207a21d5b2732e8ace1f2ee78b (refs/tags/0.0.4)
[main] INFO git.lfs.migrate.Main -   convert ref: 36f2fab6edddbd9703b41ca7b0c2251fd575da0b -> 8d847b4878447a42901012e893e8a4903cd034af (refs/tags/0.0.5)
[main] INFO git.lfs.migrate.Main -   convert ref: 04701dba2a82f59df9415ebdae96f772253e725b -> 048695c95064bc8f72034c95dc9fefbb8ab5c192 (refs/tags/0.0.6)
[main] INFO git.lfs.migrate.Main -   convert ref: 724e0d35deac4209755e00103c10b42408b049b4 -> 4dd2059afd59e9f26a8f2b0731c050c4988ebe4f (refs/tags/0.0.7)
[main] INFO git.lfs.migrate.Main -   convert ref: d2e19aa87cace25936becf4cbac15cc16ce4df79 -> 984383f01c3d3aeeee49e0bf4b0008bc68afa173 (refs/tags/0.0.8)
[main] INFO git.lfs.migrate.Main -   convert ref: ba899229359132c6d8f51a7d207e392a2f883211 -> d934b46ca5928b38757bbece7370e0d0c75f95f9 (refs/tags/0.0.9)
[main] INFO git.lfs.migrate.Main -   convert ref: 7e7cccccb1022c77ad5cb07fe3fd996d26a30fd5 -> 46c4f55918d35a1dea2dcb5c1a1814a12431ef3a (refs/tags/0.1.0)
[main] INFO git.lfs.migrate.Main -   convert ref: 61243791234be1cb92ad45ed3961ccea8ca6bbc0 -> 095fefc0bef3da728a289bc606507db21d515415 (refs/tags/0.1.1)
[main] INFO git.lfs.migrate.Main -   convert ref: 6c3cd1def3c4877022e29d0a0de4a6ec0485874e -> 2d6bcf34b69ef2bc5e3c86445809854387565c09 (refs/tags/0.2.0)
[main] INFO git.lfs.migrate.Main -   convert ref: 357092b16879390d69ba0c1a1e4481a10831f2de -> e103cfe2a0da872a92b4a4acc7cad314ebf75ed4 (refs/tags/0.2.1)
[main] INFO git.lfs.migrate.Main -   convert ref: 10b441e662f1cf02f3899db1da281457c5af44cc -> 7925efb628965e4a0e5fe6b1229769fd5402a6e3 (refs/tags/0.2.2)
[main] INFO git.lfs.migrate.Main -   convert ref: cd0c7829dafa0dabffb0c4eb32ca6f790e142338 -> c0dbac11c8524a9975aa40f2634c9179c7367061 (refs/tags/0.2.3)
[main] INFO git.lfs.migrate.Main -   convert ref: 2145c973f03617cd07361717785fd58604c94419 -> c176ab2c82ee7a780d2c193a70e8fa79ac2e9f89 (refs/tags/0.2.4)
[main] INFO git.lfs.migrate.Main - Convert time: 1431

I tried to use the GitHub lfs-test-server which worked without problems (all lfs objects where uploaded). For both lfs-servers, the outputs of git-lfs-migrate.jar are the same.
Sadly I did not find a way to increase the verbosity to get more information.

Error "duplicate entry names" while migrating existing repository data to LFS

There is not enough logged information for the error "duplicate entry names".
See http://stackoverflow.com/q/43187374/5154345

Exception in thread "main" java.util.concurrent.ExecutionException: ru.bozaro.gitlfs.client.exceptions.RequestException

I receive this stacktrace when using git-lfs-migrate 0.1.0:

Exception in thread "main" java.util.concurrent.ExecutionException: ru.bozaro.gitlfs.client.exceptions.RequestException
        at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
        at git.lfs.migrate.Main$HttpUploader.close(Main.java:292)
        at git.lfs.migrate.Main.processRepository(Main.java:88)
        at git.lfs.migrate.Main.main(Main.java:60)
Caused by: ru.bozaro.gitlfs.client.exceptions.RequestException
        at ru.bozaro.gitlfs.client.Client.doRequest(Client.java:314)
        at ru.bozaro.gitlfs.client.internal.BatchWorker.submitBatchTask(BatchWorker.java:162)
        at ru.bozaro.gitlfs.client.internal.BatchWorker.lambda$tryBatchRequest$0(BatchWorker.java:124)
        at ru.bozaro.gitlfs.client.internal.BatchWorker.executeInPool(BatchWorker.java:298)
        at ru.bozaro.gitlfs.client.internal.BatchWorker.access$400(BatchWorker.java:34)
        at ru.bozaro.gitlfs.client.internal.BatchWorker$1.run(BatchWorker.java:317)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

The command run was:

java -jar /mnt/tools/git-lfs-migrate.jar -t 16 -s lfs-test.git -d lfs-clean.git -l http://10.100.166.77:8081/artifactory/api/lfs/git-lfs .7z .CUR .EXE .NED .VSD .avi .bak .bz2 .chm .dll .dmp .doc .docx .dot .exe .git .gz .jar .jpg .jpi .mpp .msi .pdf .png .ppt .pptx .rar .vdx .vsd .war .xls .xlsm .xlsx .zip

Convert files no extension

Hi,

Is there a way to convert files without an extension? For example, I tried this:

java -jar git-lfs-migrate.jar \ -s source.git \ -d dest.git \ -g https://example/lfs-test.git
\ “.clean” \ “.log” \ “SAMPLEDATA”

But the file SAMPLEDATA did not get converted to LFS. Is there a way to accomplish this?

Thanks.

Corrupt objects after migration?

I tried to run a gc after migrating but it failed:

$ git gc --aggressive --no-prune
error: refs/tags/v0.0.0 does not point to a valid object!
error: refs/tags/v0.0.0 does not point to a valid object!
error: refs/tags/v0.0.0 does not point to a valid object!
error: Could not read 5ddc0b062c709c4a8985a553396571ac5fb22200
fatal: Failed to traverse parents of commit e28d56914098097ba25d178db128e332e417ef6c
error: failed to run repack

This used to work just fine!

I think I recently upgraded from 2145c97 to d930ef5

Exception in thread "main" java.io.FileNotFoundException: https://GHUser:[email protected]/objects

Hello Again!

I'm currently struggling with the LFS-URL if LFS from github is used.

I tried different URLs but it seems I didn't got it right:

-l https://yyy:[email protected]/QuantumBytesinc/ROAC.git/info/lfs
-l https://yyy:[email protected]/QuantumBytesinc/ROAC/
-l https://yyy:[email protected]/QuantumBytesinc/
-l https://yyy:[email protected]

The stacktrace is:

Uploading 610fbe75bf02eef89f776aabd2508d581fbdc5057d21a80476d0f0189269eab0, size: 640 to: /QuantumBytesinc/ROAC.git/info/objects
Exception in thread "main" java.io.FileNotFoundException: https://yyy:[email protected]/QuantumBytesinc/ROAC.git/info/objects
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1889)
    at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1884)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1883)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1456)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
    at git.lfs.migrate.GitConverter.upload(GitConverter.java:263)
    at git.lfs.migrate.GitConverter.lambda$convertLFS$2(GitConverter.java:206)
    at git.lfs.migrate.GitConverter$$Lambda$3/867398280.apply(Unknown Source)
    at java.util.HashMap.computeIfAbsent(HashMap.java:1118)
    at git.lfs.migrate.GitConverter.convertLFS(GitConverter.java:188)
    at git.lfs.migrate.GitConverter.convertTree(GitConverter.java:151)
    at git.lfs.migrate.GitConverter.lambda$convert$0(GitConverter.java:70)
    at git.lfs.migrate.GitConverter$$Lambda$1/494586676.apply(Unknown Source)
    at java.util.HashMap.computeIfAbsent(HashMap.java:1118)
    at git.lfs.migrate.GitConverter.convert(GitConverter.java:60)
    at git.lfs.migrate.GitConverter.convertTree(GitConverter.java:153)
    at git.lfs.migrate.GitConverter.lambda$convert$0(GitConverter.java:70)
    at git.lfs.migrate.GitConverter$$Lambda$1/494586676.apply(Unknown Source)
    at java.util.HashMap.computeIfAbsent(HashMap.java:1118)
    at git.lfs.migrate.GitConverter.convert(GitConverter.java:60)
    at git.lfs.migrate.GitConverter.convertTree(GitConverter.java:153)
    at git.lfs.migrate.GitConverter.lambda$convert$0(GitConverter.java:70)
    at git.lfs.migrate.GitConverter$$Lambda$1/494586676.apply(Unknown Source)
    at java.util.HashMap.computeIfAbsent(HashMap.java:1118)
    at git.lfs.migrate.GitConverter.convert(GitConverter.java:60)
    at git.lfs.migrate.GitConverter.convertTree(GitConverter.java:153)
    at git.lfs.migrate.GitConverter.lambda$convert$0(GitConverter.java:70)
    at git.lfs.migrate.GitConverter$$Lambda$1/494586676.apply(Unknown Source)
    at java.util.HashMap.computeIfAbsent(HashMap.java:1118)
    at git.lfs.migrate.GitConverter.convert(GitConverter.java:60)
    at git.lfs.migrate.GitConverter.convertTree(GitConverter.java:153)
    at git.lfs.migrate.GitConverter.lambda$convert$0(GitConverter.java:70)
    at git.lfs.migrate.GitConverter$$Lambda$1/494586676.apply(Unknown Source)
    at java.util.HashMap.computeIfAbsent(HashMap.java:1118)
    at git.lfs.migrate.GitConverter.convert(GitConverter.java:60)
    at git.lfs.migrate.GitConverter.convertTree(GitConverter.java:153)
    at git.lfs.migrate.GitConverter.lambda$convertTreeRoot$1(GitConverter.java:130)
    at git.lfs.migrate.GitConverter$$Lambda$2/322836221.apply(Unknown Source)
    at java.util.HashMap.computeIfAbsent(HashMap.java:1118)
    at git.lfs.migrate.GitConverter.convertTreeRoot(GitConverter.java:128)
    at git.lfs.migrate.GitConverter.convertCommit(GitConverter.java:116)
    at git.lfs.migrate.GitConverter.lambda$convert$0(GitConverter.java:67)
    at git.lfs.migrate.GitConverter$$Lambda$1/494586676.apply(Unknown Source)
    at java.util.HashMap.computeIfAbsent(HashMap.java:1118)
    at git.lfs.migrate.GitConverter.convert(GitConverter.java:60)
    at git.lfs.migrate.Main.processRepository(Main.java:112)
    at git.lfs.migrate.Main.main(Main.java:89)
Caused by: java.io.FileNotFoundException: https://yyy:[email protected]/QuantumBytesinc/ROAC.git/info/objects
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1835)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
    at git.lfs.migrate.GitConverter.upload(GitConverter.java:256)
    ... 41 more

Thanks a lot for help!

text vs crlf in .gitattributes

Hi, I noticed the lines this tool adds to .gitattributes differ from lines added by git lfs track.
The migrate tool adds -crlf where as track adds -text.

Is this something that needs to be fixed, or is this correct?
Thank

Convert files in a designated path

java -jar git-lfs-migrate.jar -s source.git -d dest.git -g https://example/lfs-test.git "path1/" "path2/.bin" .
it need to be supported.

403 Forbidden: TokenExpiredError

This happened while running a very long conversion on BitBucket. Is this a case of the same token (maybe the same connection object?) being used for the duration of the entire run? And if so, could you renegotiate the session and get a new token in the handler for this exception?

My SSH knowledge is very poor, so I can't say if this theory makes any sense whatsoever.

git/lfs/migrate/Main : Unsupported major.minor version 52.0

Hi,

when I run git-lfs-migrate, I get:

Exception in thread "main" java.lang.UnsupportedClassVersionError: git/lfs/migrate/Main : Unsupported major.minor version 52.0

This is my Java VM:
java -version
java version "1.6.0_35"
OpenJDK Runtime Environment (IcedTea6 1.13.7) (6b35-1.13.7-1ubuntu0.14.04.1)
OpenJDK 64-Bit Server VM (build 23.25-b01, mixed mode)

Any idea what might be wrong?

lfs authentication error for [email protected]

If I specify user and password with "-l" option as "https://[email protected]:[email protected]" it exits with error. Instead of connecting to lfsserver.com it tries to connect to domain.com

Master branch is created in converted repo even if it does not exist in the initial repo

The master branch is automatically created by the conversion => this prevents the subbranches of master to be created.

Usecase:
There is no master branch in the original repository; the default one is main. There are several branches named master/<name>.
During the conversion, master is automatically created and, in the end, it points to the same commit the main branch points to. Besides that, the subbranches of master cannot be created, which leads to missing branches and dangling commits.

The workaround is to rename all the master/<name> branches to placeholder/<name>, run the conversion, delete the master branch (because in the initial repo the default is main) and rename back the branches to master/<name>.

❓ Why is master created in the converted repo since there is no master in the initial repo and why does it point to the same commit the default branch points to?