gotsunami / gitlab-copy Goto Github PK
View Code? Open in Web Editor NEWBatch copy issues between GitLab instances, with some cool features
License: MIT License
Batch copy issues between GitLab instances, with some cool features
License: MIT License
Compiled master from source to have support for API v4
Testrun was fine, but with -y I got the following runtime error:
Any help would be welcome !
schwinn@belpc077:~/git/src/github.com/gotsunami/gitlab-copy/bin$ ./gitlab-copy -y gitlab.yml
source: aschwinn/silecs at https://gitlab.cern.ch/api/v4
target: al.schwinn/silecs at https://gitlab.com/api/v4
--
source: finding issues ... OK
source: 46 issues (26 opened, 20 closed)
source: 3 milestone(s): 3.0.0, 2.1.2, 2.1.0
source: 7 label(s): mergeWithCERN, silecs-cli-client, silecs-codegen, silecs-communication-cpp, silecs-diagnostic-cpp, silecs-model, virtual-controller
source: 237 notes
--
Copying labels ...
Copying issues ...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x729b45]
goroutine 1 [running]:
main.(*migration).migrateIssue(0xc4200892f0, 0x1, 0x866901, 0xc42022d560)
/home/schwinn/git/src/github.com/gotsunami/gitlab-copy/cmd/gitlab-copy/migration.go:138 +0x3c5
main.(*migration).migrate(0xc4200892f0, 0x1, 0x1)
/home/schwinn/git/src/github.com/gotsunami/gitlab-copy/cmd/gitlab-copy/migration.go:375 +0xf22
main.main()
/home/schwinn/git/src/github.com/gotsunami/gitlab-copy/cmd/gitlab-copy/main.go:156 +0x933
Hi,
i very happy to found this. I wrote the gitlab.yml and executed the command but i get the following error GET https://gitlab.com/soemthing/api/v4/projects/xx/some-project: 401 {error: You need to sign in or sign up before continuing.}
I am logged in and the token is freshly made with all permissions.
i hope you can help me here.
edit: my adress in the yaml file is https://gitlab.com/mydomain
and not https://gitlab.mydomain.com
is that a problem?
Kind Regards
Julian
Relates to #13
Is there a way that a flag could be added to migrate only open milestones? Right not it copies all, but it would be nice to be able to migrate just open ones.
Thank you!
Shouldn't it be using V4?
Let's mock part of the GitLab library used and write unit tests. Too many failures reported related to nil pointer dereference.
Coping labels from one project to another works good except of the label descriptions. When opening the label settings I can see the copied labels with correct colors, but they don't contain any description.
This should be fixed for using the tool in production.
-v
CLI flag to increase log level to DEBUG
Check whether a new version of the tool is available. With a periodic check and a dedicated update
command.
According to https://docs.gitlab.com/ee/user/gitlab_com/index.html#gitlabcom-specific-rate-limits, there is now a 60/min rate limit on note creation.
When using this tool, I get errors like:
2021/11/09 11:28:46 target: error creating note for issue #7: POST https://gitlab.com/api/v4/projects/.../issues/7/notes: 429 {message: {error: This endpoint has been requested too many times. Try again later.}}
This is unfortunate because the next time the tool is started it will start from the next issue, so there may be notes missing from the previous issue.
Maybe a 1s delay can be added when adding notes?
Current project doesn't supports Gitlab instances which has self-signed SSL certificates.
That would be great if you can add support. Thank you.
Hello @matm,
When I run gitlab-copy
to transfer between different instances/hosts of Gitlab, linked attachments on the issues or notes aren't transferred.
Also, this comment points out the same problem: #15 (comment)
Update this dependency before working on #18.
Hi there,
Is there a way to copy Notes/Comments from Gitlab Issues WITHOUT copying over all of the notes and comments? I have had a look through the project and I can't see anyway to do this other than editing the source code.
Thanks!
I copied a repo with 671 issues from private server of old developers to gitlab.com. The new repo has 670 issues. The copying has skipped issue 199, meaning all issues after that are off by 1, so cross-referencing of issues above 199 are wrong, which complicates matters! :-D
I can't see anything special about that issue, it just contains a couple of lines and a few comments...
When doing the copy, the label color is created on target with default color, and doesn't retain its original color from source.
On a side note, great tool, saved us a lot of time. Thank you.
I have got a trouble during install, this is the result:
#go get -u github.com/gotsunami/gitlab-copy
´´´package github.com/gotsunami/gitlab-copy: no Go files in /root/go/src/github.com/gotsunami/gitlab-copy
´´´
If a label is not yet used, it is not copied over. Would be great to have it.
and an option to close the old one.
I want to define labels at the site level and just copy to each project. How can I copy labels without specifying a project?
Right now, issues are copied from one project to another. Let's add a -move
option on the CLI so that source issues are deleted after successful creation on target project.
Hello @cacciaresi @egmanoj @matm @jflemer-ndp @aboutofpluto
It looks like last update was 4 years ago.
Is there a plan to update the project or is it completely abandoned?
Thanks
This is not an issue, but rather a request for info/documentation. I'm trying to copy issues from one repository to another in the same gitlab instance. The instance needs users to log in first. Is there a way to accomplish this?
tools/version.sh
and use a different way of passing Version
, GitRevision
, GitBranch
and Built
dist
target to generate checksum filesThis will make the automatic building of the tool pass on GitHub actions.
I've run gitlab-copy
between two GitLab v8.8.3 to transfer issues from one to the other.
Unfortunately, files that were attached to the source issues are not transferred to the new server. When referring to them in an issue (for example, showing a screenshot), the link is broken.
To prevent getting 401 errors from GitLab when defining API path.
Well it might not be considered as good style to not assign an issue to a milestone, but gitlab-copy should not crash with a nil pointer dereference
when copying an issue that is not assigned to a milestone.
Workaround: assing all issues to a milestone
Otherwise, nice tool - keep on going! :)
Hi,
I think it would be usefull to have an option in the gitlab.yml file to indicate that we want to copy issues with a specific label.
Like this, we don't have to deal with issue numbers when we have new issues to copy from one project to another one, we only have to apply a tag.
Add a labelsOnly
entry in config file to easily copy labels between two projects:
from:
url: https://gitlab.mydomain.com
token: atoken
project: namespace/project
labelsOnly: true
to:
url: https://gitlab.sameorotherdomain.com
token: anothertoken
project: namespace/project
Label attributes are conserved.
I am trying to run gitlab-copy according to this config:
from:
url: https://mylocalgitlab.com
token: glpat-sdfdsfsdfsdfsdfsdf
project: namespace/project
to:
url: https://anotherinstance.com
token: glpat-sdfsdfsdffsdfsdfsfd
project: namespace/project
If both instance (from and to) is same instance it works.
If instances are different, like in the example, it gave source project 'namespace/project' not found
error.
Is gitlab-copy
working across different instances?
Hi,
great tool.
I would like to see the feature to copy milestones only - just as the labelsOnly.
Is there a chance, that this feature gets implemented within 1 or 2 weeks?
Kind regards, Karl
Uploads are not carried across. A lot of gitlab issues contain screenshots, so it would help if these were copied over too.
I can understand why you added the "User Name @userid Wrote..." on each comment, that makes sense to not have to workout the actual db ids of the users. However, I wonder if the YML could include some type of user id mapping, so that you don't have as much work to do to workout the mapping in the code.
go.mod
)vendor
directorymaster
to v3.0.1go-gitlab
API changes-k
CLI flag to skip TLS verification step (hence allowing to connect to GitLab instances with self-signed certificates)README.md
accordinglyI tried to move all issues in one repo. from my gitlab instance to Gitlab Main. But I just got message like:
source: finding issues ... OK
source: 0 issues (0 opened, 0 closed)
source: 0 notes
And if I put source and target same my instance url then it works
I think this issue related to user mapping. I can't understand If I just put target in all users tokens then how will library identify.
golang.org/x/net
)gopkg.in/yaml.v3 v2.1.0 => v3.0.1
golang.org/x/net v0.4.0 => v0.7.0
Very happy to have found this utility. It helped me to solve capturing a snapshot of a repo that was too large to export/import by bringing over just the current state and then using this tool to get the issues brought over.
After successfully testing gitlab-copy and getting it to work on sample repos that both lived under the same base host (and shared a token) I wasn't initially successful when I tried using https://gitlab.host.a to https://gitlab.host.b. The error shown was that the project couldn't be found.
I finally figured out how to build and use a local version of the code (I am a complete 'go' novice) so I could add print statements to see why and found that the Migration had the DstClient token and baseURL stored in the SrcClient.
It seeems the new refactoring of the GitLaber in to gitlab/client.go causes the fromgl
to get overwritten by the togl
values in migration/issues.go when creating. a new migration.
In gitlab/client.go the init() sets the DefaultClient
DefaultClient = new(Client).
In migrations/issue.go:
fromgl := gitlab.DefaultClientFrom.New(nil, c.SrcPrj.Token)
...
togl := gitlab.DefaultClientTo.New(nil, c.DstPrj.Token)
The setting of togl changes the ptr-value of DefaultClient & fromgl as well
Again, I'm new to Go so I may be missing something obvious, however here's what the print output shows. (names modified to be generic and tokens redacted)
show-failure-config.yml:
from:
url: https://git.from.co
project: group_name/myproject
token: from-token-string
to:
url: https://gitlab.to.com
token: to-token-string
project: group_to/subgroup/otherproject
And the output:
~/go/src/github.com/gotsunami/gitlab-copy/bin/gitlab-copy show-failure-config.yml
DUMMY MODE: won't apply anything (stats only)
--
Creating a new Migration...
DefaultClient address: &{<nil>}
setting fromgl from 'gitlab.DefaultCient.New'
fromgl address: &{0xc4200d01e0}
DefaultClient address: &{0xc4200d01e0}
setting togl from 'gitlab.DefaultCient.New'
togl address: &{0xc4200d03c0}
DefaultClient address: &{0xc4200d03c0}
fromgl address: &{0xc4200d03c0}
====
fromgl.BaseURL is: https://gitlab.to.com/api/v4/
togl.BaseURL is: https://gitlab.to.com/api/v4/
2021/07/29 19:41:23 source project 'group_name/myproject' not found
For my immediate needs I worked around the issue by creating two additional DefaultClient vars: DefaultClientTo and DefaultClientFrom and initializing them in init() and using them to set the fromgl and togl to become the Src and Dst in EndPoint. However I think the right fix is more in line with creating a new GitLaber client for each item. I wasn't sure why the package var DefaultClient was involved.
As a final note, even after I got it working with my hack, I wasn't able to copy attachments to notes. I saw this had been reported as fixed with #15 but it wasn't working for me. I figured I'd be happy with what I got.
When using the gitlab-copy script, for each new issue emails are sent to the people that are referenced in that issue. Preferably this should be disabled during import time (when possible).
error such as below raised when I migrate issue by v0.7-rc2. this error occured on windows and linux.
panic: runtime error: invalid memory address or nil pointer dereference
goroutine 1 [running]:
main.(*migration).migrateIssue
/home/ab/work/go/src/github.com/gotsunami/gitlab-copy/cmd/gitlab-copy/migration.go:138 main.(*migration).migrate
/home/ab/work/go/src/github.com/gotsunami/gitlab-copy/cmd/gitlab-copy/migration.go:375 main.main
/home/ab/work/go/src/github.com/gotsunami/gitlab-copy/cmd/gitlab-copy/main.go:156
Thank alot for this tool, it saved me a bunch of work !
However for gitlab API v4, it seems to have problems on closing bugs.
I just closed them by hand now, which was no big deal for only ~20 bugs. Here the console output:
schwinn@belpc077:~/git/src/github.com/gotsunami/gitlab-copy$ ./bin/gitlab-copy -y gitlab.yml
source: aschwinn/silecs at https://gitlab.cern.ch/api/v4
target: al.schwinn/silecs at https://gitlab.com/api/v4
--
source: finding issues ... OK
source: 46 issues (26 opened, 20 closed)
source: 3 milestone(s): 2.1.0, 3.0.0, 2.1.2
source: 7 label(s): virtual-controller, mergeWithCERN, silecs-cli-client, silecs-codegen, silecs-communication-cpp, silecs-diagnostic-cpp, silecs-model
source: 237 notes
--
Copying labels ...
Copying issues ...
2018/09/04 09:45:30 target: error closing issue #1: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882209: 404 {message: 404 Not found}
2018/09/04 09:45:38 target: error closing issue #2: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882210: 404 {message: 404 Not found}
2018/09/04 09:45:44 target: error closing issue #3: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882212: 404 {message: 404 Not found}
target: created issue #4: Build unit-tests to compile ( and link ? ) a DU-wrapper [opened]
target: created issue #5: Provide a way to have lab-specific code-generation [opened]
2018/09/04 09:45:59 target: error closing issue #6: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882218: 404 {message: 404 Not found}
2018/09/04 09:46:05 target: error closing issue #7: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882222: 404 {message: 404 Not found}
2018/09/04 09:46:09 target: error closing issue #8: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882225: 404 {message: 404 Not found}
target: created issue #9: Add support for boolean data-type [opened]
2018/09/04 09:46:15 target: error closing issue #10: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882228: 404 {message: 404 Not found}
target: created issue #11: show in type-selection which FESA-type corresponds to which PLC-type [opened]
target: created issue #12: Use xsd naming for C++ "access types" [opened]
target: created issue #13: Restructure C++ Classes in order to reflect SilecsClasses [opened]
target: created issue #14: replace all tabs by spaces [opened]
target: created issue #15: Usage of c-templates for different register-data types [opened]
target: created issue #16: Support for enums missing [opened]
target: created issue #17: restructure class "SilecsCluster" and rename it to "SilecsDesign" [opened]
target: created issue #18: Allow to give alias-names for fesa-classes and DU's [opened]
target: created issue #19: Provide template-builder which can re-use arguments [opened]
target: created issue #20: Introduce package "virtual controller" [opened]
2018/09/04 09:46:43 target: error closing issue #21: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882327: 404 {message: 404 Not found}
target: created issue #22: Re-Structure C++ Block model [opened]
target: created issue #23: move generated code for fesa-class into "generated" folder [opened]
target: created issue #24: Split library into snap7 / modbus / etc [opened]
target: created issue #25: Investigate: How to make use of BLOCK_MODE in FESA [opened]
2018/09/04 09:46:55 target: error closing issue #26: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882336: 404 {message: 404 Not found}
2018/09/04 09:46:57 target: error closing issue #27: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882337: 404 {message: 404 Not found}
target: created issue #28: Provide lab-specific install-script [opened]
target: created issue #29: Provide a compartibility-mode for old checksums [opened]
target: created issue #30: Create merge requests for differences between CERN silecs-communication-cpp and GSI version [opened]
2018/09/04 09:47:12 target: error closing issue #31: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882342: 404 {message: 404 Not found}
target: created issue #32: Create merge requests for differences between CERN silecs-diagnostic-cpp and GSI version [opened]
2018/09/04 09:47:21 target: error closing issue #33: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882351: 404 {message: 404 Not found}
2018/09/04 09:47:25 target: error closing issue #34: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882354: 404 {message: 404 Not found}
2018/09/04 09:47:28 target: error closing issue #35: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882356: 404 {message: 404 Not found}
2018/09/04 09:47:32 target: error closing issue #36: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882358: 404 {message: 404 Not found}
target: created issue #37: Evaluate possibility to connect memory register with outputs in Siemens scl code [opened]
2018/09/04 09:47:36 target: error closing issue #38: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882362: 404 {message: 404 Not found}
target: created issue #39: Make use of python package 'unittest' [opened]
target: created issue #40: move string to value method(s) to silecs-communication utils [opened]
2018/09/04 09:47:48 target: error closing issue #41: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882367: 404 {message: 404 Not found}
target: created issue #42: isInitialized_ check in codegen should get it's own method [opened]
2018/09/04 09:47:54 target: error closing issue #43: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882372: 404 {message: 404 Not found}
2018/09/04 09:47:59 target: error closing issue #44: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882374: 404 {message: 404 Not found}
2018/09/04 09:48:04 target: error closing issue #45: PUT https://gitlab.com/api/v4/projects/7953820/issues/13882376: 404 {message: 404 Not found}
target: created issue #46: Prefix Block Makros in generated ClassHeader with Classname [opened]
Can the order of issues be preserved? This will help preserve issue numbers when copying issues from an existing project to a project that doesn't have any.
When migrating issues from one project to another labels are not applied to issues that are already closed. Open issues are correctly labelled.
I am looking for a method to copy my issues within the same project:
from:
url: https://gitlab.com
token: my-very-secret-token
project: my-team/my-project
issues:
- 1156
to:
url: https://gitlab.com
token: my-very-secret-token
project: my-team/my-project
unfortunately this gives me an error: target: issue 1156 already exists, skipping...
Is this feature possible and do you have time to add this?
Thanks for the wonderful tool!
I tried a YAML setup like this to copy labels between projects:
from:
url: http://gitlab.foobar.com
token: foo
project: 1
labelsOnly: true
to:
url: http://gitlab.foobar.com
token: foo
project: 2
Unfortunately, only the first 20 labels were copied, due to pagination.
An partial quick-fix might be to use per_page=100 to at least increase the number of labels copied, and update "... in order to copy all labels from one project to another ..." at https://github.com/gotsunami/gitlab-copy#more-features.
See:
https://docs.gitlab.com/ee/api/labels.html#list-labels
https://docs.gitlab.com/ee/api/index.html#offset-based-pagination
I'm sorry, I don't speak Go at all so I can't offer a pull request. Thanks for all your hard work!
This occurs when an issue's description or a note's body is too large. This makes the issue creation on target fail. This error can be caught and fixed by saving a shortened description or body content.
Let's use dep
for now.
Currently if an issue is copied all notes are copied with it including developers assigning the original issue to a milestone. Is it possible to include an option to not have these notes be created as comments on destination issue so as to prevent users from being spammed with 60+ todos/emails/notifications when multiple issues they worked on in the past have been moved to a new project. Chances are they are not working on that project anymore either and don't need to be re-notified.
This image below shows what the comments look like after being copied over. Having the issue simply read "@z changed milestone to Open Source Intelligence Gathering 4 hours ago" without the original developer tag or what the user "wrote".
If the gitlab instance uses a company CA-signed certificate (which are often not then signed by a global CA) then certificate cannot be validated and the process fails
Can you add a option type to the config.yml to skip verification? Kind of equivalent to wget's no-check-certificate flag
Output when such a case arises looks like:
DUMMY MODE: won't apply anything (stats only)
--
2018/05/30 21:19:38 network error: Get https:/api/v3/projects/namespace%2Fproject: x509: certificate signed by unknown authority
Hi @matm,
With the latest update, 0.8.1, I have run the gitlab-copy to copy from one Gitlab instance to another one.
First issue is copied successfully. But when it started copying second issue, I got this error:
...
Copying issues ...
2023/03/10 06:03:28 [DEBUG] GET https://myremoteserver.com/api/v4/projects/3/issues?page=1&per_page=100&sort=asc
2023/03/10 06:03:30 [DEBUG] GET https://myremoteserver.com/api/v4/projects/3/issues?page=2&per_page=100&sort=asc
2023/03/10 06:03:31 [DEBUG] GET https://myremoteserver.com/api/v4/projects/3/issues?page=3&per_page=100&sort=asc
2023/03/10 06:03:31 [DEBUG] GET https://myremoteserver.com/api/v4/projects/3/issues/1
2023/03/10 06:03:31 [DEBUG] GET https://mylocalserver.com/api/v4/projects/68/issues
target: issue 1 already exists, skipping...2023/03/10 06:03:31 [DEBUG] GET https://myremoteserver.com/api/v4/projects/3/issues/2
2023/03/10 06:03:32 [DEBUG] GET https://mylocalserver.com/api/v4/projects/68/issues
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x50 pc=0xcc334d]
goroutine 1 [running]:
github.com/gotsunami/gitlab-copy/migration.(*Migration).migrateIssue(0xc00008b110, 0xc0000c8008?)
/home/mathias/progs/gitlab-copy/migration/issue.go:135 +0x32d
github.com/gotsunami/gitlab-copy/migration.(*Migration).Migrate(0xc00008b110)
/home/mathias/progs/gitlab-copy/migration/issue.go:406 +0x778
main.main()
/home/mathias/progs/gitlab-copy/cmd/gitlab-copy/main.go:172 +0x1065
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.