miromannino / contributions-importer-for-github

This tool helps users to import contributions to GitHub from private git repositories, or from public repositories that are not hosted in GitHub.

Hi there! I’m Miro, a Full-Stack Software Engineer working at New York University in Abu Dhabi. Originally from Italy, I earned my MS and BS from the University of Bologna. I like computer science, I like art, and wherever they meet together.

A more complete list of my projects: https://miromannino.com/portfolio/

contributions-importer-for-github's Issues

Imports commits of others too

When I import commits from bitbucket to github, I get way too many commits.
It is importing all the commits regardless of who is commiting.
My commits include a merge commit from others. I'm guessing that is the problem because it is not importing commits out of the merged ones.

I expected it to import commits which had my email id only. Apparently, my github calendar has a few "fake" commits (commits that are not actually mine) which is not the intended purpose of this project.

Language is java instead of kotlin

My project is an android app written in kotlin but my mock repo shows its 100% java. How to make it reflect the correct language?

Commits not reflected in GitHub

I ran the script provided in the documentation and it seemed to have the expected behavior - it analyzed all the commits and ignored those not done by me. However, I did not see the commits reflected on my profile after the script was complete. It is entirely possible that I've missed a step somewhere, so if you have any guidance it would be appreciated!

Is there a way to keep original commit messages?

If there isn't I would love to implement this in my free time

Been using this tool for quite some time now, it's awesome

And thanks for this repo :D

Commit times are offset by time zone

Not a big deal, but commit times are changed by the hour difference from UTC of the user's time zone.

Example:
Original commit:

Author: John Doe <[email protected]>
Date:   Mon Mar 25 19:09:42 2019 -0400

Imported commit:

Author: John Doe <[email protected]>
Date:   Mon Mar 25 23:09:42 2019 -0400

Add ScalaGenerator [patch included!]

I can't open PR from my private repo, so could you apply this patch or open PR yourself with this patch?

From f73e2fc527450f713b78154a08b113656a01a9f4 Mon Sep 17 00:00:00 2001
Date: Thu, 14 Jan 2021 19:05:29 +0200
Subject: [PATCH] Add ScalaGenerator

---
 .../generators/ScalaGenerator.py              | 26 +++++++++++++++++++
 .../generators/__init__.py                    |  2 ++
 2 files changed, 28 insertions(+)
 create mode 100644 git_contributions_importer/generators/ScalaGenerator.py

diff --git a/git_contributions_importer/generators/ScalaGenerator.py b/git_contributions_importer/generators/ScalaGenerator.py
new file mode 100644
index 0000000..d1f58a6
--- /dev/null
+++ b/git_contributions_importer/generators/ScalaGenerator.py
@@ -0,0 +1,26 @@
+#!/usr/bin/python3
+
+from . import Generator
+
+
+class ScalaGenerator(Generator):
+    # one day this will generate awesome random code
+
+    min_content_size = 4
+
+    def __init__(self):
+        pass
+
+    def insert(self, content, num):
+        if len(content) <= self.min_content_size:
+            content.clear()
+            content.append('object Main' + self.random_string(5) + ' {')
+            content.append('    def main(args: Array[String]): Unit = {')
+            content.append('    }')
+            content.append('}')
+        for i in range(num):
+            content.insert(-2, '        println("' + self.random_string(5) + '");')
+
+    def delete(self, content, num):
+        for i in range(min(num, len(content) - self.min_content_size)):
+            content.pop(-3)
diff --git a/git_contributions_importer/generators/__init__.py b/git_contributions_importer/generators/__init__.py
index 6cbdf10..4cca874 100644
--- a/git_contributions_importer/generators/__init__.py
+++ b/git_contributions_importer/generators/__init__.py
@@ -3,6 +3,7 @@
 from .Generator import Generator
 from .JsGenerator import JsGenerator
 from .JavaGenerator import JavaGenerator
+from .ScalaGenerator import ScalaGenerator
 from .CssGenerator import CssGenerator
 from .CppGenerator import CppGenerator
 from .CGenerator import CGenerator
@@ -20,6 +21,7 @@
     '.tex': Generator,
     '.js': JsGenerator,
     '.java': JavaGenerator,
+    '.scala': ScalaGenerator,
     '.css': CssGenerator,
     '.scss': CssGenerator,
     '.cpp': CppGenerator,

Or as a file:
Add_ScalaGenerator.txt

git apply --stat Add_ScalaGenerator.txt (Github rejects to upload *.patch files)

Contributions not imported

I downloaded the package, and wrote a script. The script seems to run fine, I can see the analysing, commit and apply changes logs, but no contribution history is updated on my profile.

# run_script.py 
import git
from git_contributions_importer import *

# private repo
repo = git.Repo("../../infuse/clover-dashboard-react")
# mock repo
mock_repo = git.Repo("../../daniel/mock-repo")

importer = Importer([repo], mock_repo)
importer.set_author(['[email protected]', '[email protected]'])
  
importer.import_repository()

Could it be I missed some setup steps?

Contributions not being shown on Github

Hi, I have seen the two closed issues on this topic and am hoping to understand why the contributions dont show up.

In my script I have

repo = git.Repo("../../Projects/otter")
# Your mock repo
mock_repo = git.Repo("../../Projects/mock-repo-otter")
importer = Importer([repo], mock_repo)

I run the script and see my commits being analyzed with others not by me skipped

I cd to the mock repo and run git push

In my mock repo I see latest activity "...seconds ago"

But the commits arent reflected in the contribution chart.

Why is this ?

importer.set_start_from_last should exclude last commit

Hello,

when using importer.set_start_from_last(True),
only commits before the date are ignored
but if the dates are equals it should be ignored too and it's not the case,
this cause the last commit to be repeated twice

commit 07072356f7ddc123e46e8ee9c3c622ff976c00c5 (HEAD -> master)
Date:   Thu Jun 3 13:27:36 2021 +0000

commit 59b18244053533fc1510df6d5e33ad51f06bf551 (origin/master, origin/HEAD)
Date:   Thu Jun 3 13:27:36 2021 +0000

thanks

Script hangs in some repos

I already managed to run the script for some repos, but in hangs on others.
Debugged a bit and checked out till the first version, but the same thing happens.
Something else I noticed is that these repos had the .git/branches dir empty, not sure if it's relevant.
I can see this, when i stop the script manually
Starting ^CTraceback (most recent call last): File "run_script.py", line 15, in <module> importer.import_repository() File "/Users/ydrargyros/Projects/Contributions-Importer-For-Github/git_contributions_importer/Importer.py", line 74, in import_repository for c in self.get_all_commits(last_committed_date): File "/Users/ydrargyros/Projects/Contributions-Importer-For-Github/git_contributions_importer/Importer.py", line 133, in get_all_commits if c.committed_date < ignore_before_date or (self.ignore_before_date != None and c.committed_date < self.ignore_before_date): File "/usr/local/lib/python3.8/site-packages/gitdb/util.py", line 253, in __getattr__ self._set_cache_(attr) File "/usr/local/lib/python3.8/site-packages/git/objects/commit.py", line 143, in _set_cache_ self._deserialize(BytesIO(stream.read())) File "/usr/local/lib/python3.8/site-packages/git/cmd.py", line 446, in read data = self._stream.read(size) KeyboardInterrupt

When i removed the /branches
Starting ^CTraceback (most recent call last): File "run_script.py", line 15, in <module> importer.import_repository() File "/Users/ydrargyros/Projects/Contributions-Importer-For-Github/git_contributions_importer/Importer.py", line 74, in import_repository for c in self.get_all_commits(last_committed_date): File "/Users/ydrargyros/Projects/Contributions-Importer-For-Github/git_contributions_importer/Importer.py", line 133, in get_all_commits if c.committed_date < ignore_before_date or (self.ignore_before_date != None and c.committed_date < self.ignore_before_date): File "/usr/local/lib/python3.8/site-packages/gitdb/util.py", line 253, in __getattr__ self._set_cache_(attr) File "/usr/local/lib/python3.8/site-packages/git/objects/commit.py", line 142, in _set_cache_ _binsha, _typename, self.size, stream = self.repo.odb.stream(self.binsha) File "/usr/local/lib/python3.8/site-packages/git/db.py", line 42, in stream hexsha, typename, size, stream = self._git.stream_object_data(bin_to_hex(sha)) File "/usr/local/lib/python3.8/site-packages/git/cmd.py", line 1085, in stream_object_data hexsha, typename, size = self.__get_object_header(cmd, ref) File "/usr/local/lib/python3.8/site-packages/git/cmd.py", line 1057, in __get_object_header return self._parse_object_header(cmd.stdout.readline()) KeyboardInterrupt
Thanks for this nice tool

Is there a way to use multiple repos?

In the Python script you set the repo variable to the repo you want to pull commits from. In my case, I have 5 different repos in Bitbucket. Is there any way to grab commits from all five?

Is it possible to have a valid test script?

Hello Sir,
This tool sounds amazing, I tried to run several scripts in the a similar format like the examples in the README.md and in /tests/basic_test.py but I get errors

Is it possible to make an actual running script for a mock project?

unable to import 'git' pylint (import-error)

I'm getting this weird unable to import 'git' error, I tried troubleshooting on my end but no luck. Was hoping someone may know a quick fix to this.

Add tests

Contributions from enterprise to github

Can we import Github Enterprise private repo's contribution to Github mock repo?

Each run creates duplicates

Hello,

Is there a way to prevent duplicate commits through defining a start date?

I ran this initially and everything worked great, but after a couple more days of development, I re-ran the script so it could show the latest few day's commits, but I noticed it also pulled the previous commits again, so the contributions show more commits than intended.

Bug: Contributions are fetched only for last commited date

With your last commit ec4ec71 you have set ignoreBeforeDate to last commit date, this prevents your plugin to analyse older commits. I think you should add an option to check if a user wants to import all commits or just last one

OSError: [Errno 9] Bad File Descriptor

Hello,

I've been using this code for almost a year now. It has worked great up until a few months ago. I started getting the error message below. It seems like its having trouble with a specific commit but I can't tell. Has anyone seen or reported something like this before?

Analyzing commit at 2022-10-18 17:54:58
Commit changes: insertions: {'.xml': 100} deletions: {'.xml': 100}
Apply changes: insertions: {'.xml': 1} deletions: {'.xml': 1}
OSError: [Errno 9] Bad file descriptor

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "c:\Users\Matthew.Hiebing\Documents\1_GitHubRepos\BitbucketCommits\Contributions-Importer-For-Github\run_script.py", line 22, in
importer.import_repository()
File "c:\Users\Matthew.Hiebing\Documents\1_GitHubRepos\BitbucketCommits\Contributions-Importer-For-Github\git_contributions_importer\Importer.py", line 104, in import_repository
self.content.save()
File "c:\Users\Matthew.Hiebing\Documents\1_GitHubRepos\BitbucketCommits\Contributions-Importer-For-Github\git_contributions_importer\Content.py", line 33, in save
with open(full_path, 'w') as f:
OSError: [Errno 9] Bad file descriptor
(bitbucketToGitHub)

Any way to include commits from multiple repos?

I work on multiple projects at the same time and importing repo's one by one with a new repo every time takes a lot of time. If I merge everything into a same repo, the history becomes haphazard.

I would like to make this chronological. One thing I can think is have everything in memory, sort and then start committing when everything is parsed.

• Can this be done any other way?
• Is anything already done for such scenarios in this awesome project?
• What do you guys do?

I used a for loop to pick commits from multiple repositories.

# Contributions-Importer-For-Github/run.py
import git
from git_contributions_importer import *

def importCommits(path):
    # Your private repo or Bitbucket repo
    print("processing", path)
    repo = git.Repo(path)
    # Your mock repo
    mock_repo = git.Repo("D:/company/mock-repo")
    importer = Importer([repo], mock_repo)

    importer.set_author(['[email protected]', '[email protected]'])
    importer.set_start_from_last(False)
    importer.import_repository()


branches = ["folder1", "folder2", "folder3"]
projects = ["subfolder1", "subfolder2", "subfolder3", "subfolder4", "subfolder5", "subfolder6"]

for branch in branches:
    for project in projects:
        importCommits(f"D:/company//{branch}//{project}")

importCommits(f"some//other//path")

This is the messed up chronology i get with git log --pretty=format:"%ar". This change is whenever the folder (and hence the repo changes).

6 months ago
6 months ago
7 months ago
7 months ago
7 months ago
7 months ago
8 months ago
8 months ago
8 months ago
4 weeks ago
3 months ago
3 months ago
3 months ago
4 months ago
5 months ago
5 months ago
9 months ago

**New repo starts**
10 days ago
10 days ago
12 days ago
12 days ago
12 days ago
12 days ago
12 days ago
2 weeks ago
9 weeks ago
9 weeks ago
3 months ago
3 months ago
3 months ago
3 months ago
4 months ago
4 months ago
4 months ago
4 months ago
5 months ago
5 months ago
8 months ago
8 months ago
8 months ago
8 months ago
9 months ago

**New repo starts again**
2 days ago
2 days ago
2 days ago

Error when running script

I followed the instructions listed in GitHub, and currently receiving this odd " NoSuchPathError " , any help would be appreciated.

Issue updating contributor graph

Hi!

I've run the python script specified in the README (after entering in the appropriate info). While running the script, the commits from the original repo appear in the logs, but when I go to push the mock repo, the contributor graph doesn't appear.

Easier installation

Provide an easier installation with pip and brew, and possibly others if needed

Add CLI

Sync ignores commits after adding manual commit to mock repo

After adding a manual commit to the mock repo, repo sync will ignore any unsynced commits that were made before the manual commit in the mock repo.

Last contribution is always reimported when time zone is GMT

When local time zone is GMT, the last imported contribution is always re-imported. This issue also becomes apparent after applying PR #8 which resolves bug #5 (remove incorrect time zone offset in commit date).

Support author regex

If PR is merged on github website the author would be something like

[email protected]

I am not 100% when exactly this happens, but this prefix number is static per user (or by user+repo, I'll check it).

The idea is to support not only constant authors but also author email regex, e.g.:

\d+\+original-username@users\.noreply\.github\.com

I've already patched it in my mock repo, su if community will be interested in it, I'll open a PR.

Error when running script

When I run the script I got an error, here the full log output:

Starting

Analysing commit at 2018-12-04 13:04:38
    Commit changes: insertions: {'.md': 3} deletions: {}
    Apply changes: insertions: {'.md': 1} deletions: {}
    Commit at: 2018-12-04 13:04:38

Analysing commit at 2018-12-04 17:16:54
    Commit changes: insertions: {'': 100, '.ejs': 100, '.map': 2, '.json': 100, '.css': 86, '.js': 100} deletions: {}
    Apply changes: insertions: {'': 1, '.ejs': 1, '.map': 1, '.json': 1, '.css': 1, '.js': 1} deletions: {}
Traceback (most recent call last):
  File "import.py", line 21, in <module>
    importer.import_repository()
  File "/Contributions-Importer-For-Github/git_contributions_importer/Importer.py", line 98, in import_repository
    apply(self.content, broken_stats)
  File "/Contributions-Importer-For-Github/git_contributions_importer/generators/__init__.py", line 45, in apply
    gen.insert(content.get(ext), num)
  File "/Contributions-Importer-For-Github/git_contributions_importer/generators/JsonGenerator.py", line 17, in insert
    content.clear()
AttributeError: 'list' object has no attribute 'clear'