gavinr / github-csv-tools

Import and export GitHub issues via CSV

Import and export GitHub issues via CSV

Usage

Prerequisite: Install Node.js, then run this to install:

npm install -g github-csv-tools

After install, githubCsvTools --help for info on how to use, or see below.

Instructions for exporting or importing:

To Import Issues

Currently imports title, body, labels, status (closed or open) and milestones. See the test folder for example input formats.

githubCsvTools myFile.csv

To Export Issues

githubCsvTools

Option	Default	Notes
-f, --exportFileName	YYYY-MM-DD-hh-mm-ss-issues.csv	The name of the CSV you'd like to export to.
-a, --exportAttributes	number, title, labels, state, assignees, milestone, comments, created_at, updated_at, closed_at, body	Comma-separated list of attributes (columns) in the export**.
-c, --exportComments	n/a	Include comments in the export. If using in combination with --exportAttributes, id must be included.
-e, --exportAll	n/a	Export all possible values from the GitHub API. If not included, a subset of attributes (see --exportAttributes above) that are known to be compatible with GitHub import will be included in the export.

** List of all possible options for exportAttributes includes every option in the GitHub API Export. Values in child objects can be referenced using a "dot" - for example, a user's avatar_url can be accessed via user.avatar_url.

Tokens

For all actions, the tool will ask you to input a GitHub token. To obtain this token:

1. Go to https://github.com/settings/tokens
2. Click "Generate New Token"
3. Check on repo
4. Copy/paste the token provided when the tool asks for it.

Other Options

Option	Notes
-V, --version	Output the version number
-g, --github_enterprise	Your GitHub Enterprise URL. https://your-internal-githubenterprise.com/api/v3 (Reminder: do not forget the /api/v3 at the end)
-t, --token	The GitHub token. https://github.com/settings/tokens
-o, --organization	The User or Organization slug that the repo lives under. Example: For /myOrg/my-repo, this value would be myOrg.
-r, --repository	The repository name (slug). Example: For /myOrg/my-repo, this value would be my-repo.
--csvDelimiter	The CSV delimiter character (defaults to ',')
-v, --verbose	Include additional logging information.
-h, --help	See all the options and help.

Development

1. Clone the repo.
2. Browse to repo, then run npm install -g

Changelog

See CHANGELOG.md

Hosted version

This software can be used without download/install by going to repoio.com.

Thanks

• octokit/rest.js
• nodeCSV
• commander
• co
• Tim Patterson's Post on Atlassian.com

github-csv-tools's Issues

Unable to find output .csv

Hi there,

This is the perfect tool I am looking for but I am currently unable to find the outputted .csv file. Where does it go by default? I am using the latest node.js and im on macOS Catalina.

Thanks,

Please add percentage process (1% 2%) when running

Please add percentage process (1% 2%) when running as I had to wait for more than 30 mins but have no idea what was happening.
Thanks

Artifact Explorer

Would appreciate someone analyizing how close this new tool from GitHub is to this repo.
https://github.blog/2021-05-18-github-artifact-exporter-open-source-release/

If you don't specify the fields, you get all of them and that causes a problem

The problem is that if your source repo has a milestone assigned to those fields, they are listed as a URI in them on the export. When you try to import that CSV into your dest repository, it complains and throws a weird Github API error that it doesn't know what that URI is.

A better approach would be to post-process the URIs and/or to create a corresponding milestone in the dest repository if not already there, and then to scrub the CSV data with the new milestone. Or simply to by default exclude any fields that only exist in the source domain unless otherwise specified.

getting only the milestone Title

Hi, I am trying to only get the milestone Title for the csv.

Do you have any idea, how to update the code, that instead of all the Milestone columns, I get something like this:

const defaultColumns = values.exportAttributes || [
"created_at",
"labels",
"milestone.title",
"number",
"title",
"body",
"state",
];

thanks = )

great tool by the way^^

Exporting Issues to CSV

we should allow for this.

Exported body is only the first line of the original body

Export - Ability to filter issues

#41 There are a variety of attributes that we may want to filter on, and I'd like to have a more generic switch so that we do not have to create a custom cli switch for each (milestone, state, assignee, creator, mentioned, labels)

I'm open to suggestions on what the api should be. My initial thought it:

githubCsvTools --exportFilter attribute,value

... where attribute is one of the attributes that you can filter on, like state, etc, and value is the value that you'd like to filter.

Fresh installation is not working for me

➜  Downloads  node --version
v8.6.0
➜  Downloads  npm install -g github-csv-tools
/Users/user/.nvm/versions/node/v8.6.0/bin/githubCsvTools -> /Users/user/.nvm/versions/node/v8.6.0/lib/node_modules/github-csv-tools/index.js
+ [email protected]
updated 1 package in 0.611s
➜  Downloads  githubCsvTools --help          
env: node\r: No such file or directory
➜  Downloads  which githubCsvTools 
/Users/user/.nvm/versions/node/v8.6.0/bin/githubCsvTools

Differentiate between issues and PRs

I was able to successfully download the CSV but it contains both PRs and issues from my repo. Though PRs are also the same as issues in GitHub but is there a way in the extract we can differentiate between the two?

More complex samples

@gavinr can you provide a csv example for a more complex ticket? Eg one with paragraphs and lists and alike?

Reaching limit when importing

I am trying to import csv with 88 entries
and I got this error after 20 entries:

[@octokit/plugin-throttling] `onAbuseLimit()` is deprecated and will be removed in a future release of `@octokit/plugin-throttling`, please use the `onSecondaryRateLimit` handler instead
Abuse detected for request POST /repos/{owner}/{repo}/issues
Error
{ HttpError: You have exceeded a secondary rate limit and have been temporarily blocked from content creation. Please retry your request again later.
    at fetch.then (/usr/local/lib/node_modules/github-csv-tools/node_modules/@octokit/request/dist-node/index.js:86:21)
    at process._tickCallback (internal/process/next_tick.js:68:7)

Include first comment in CSV export

I understand that exporting the whole conversation thread of an issue is probably too complicated, but it would be cool if at least the first comment could be exported, as it mostly acts as the body of the issue. Would you be willing to include this?
Really helpful tool, btw. Thanks for building it!

Add an option to format Issue number as link

Would it be desirable to add an option to format the Issue number using the spreadsheet HYPERLINK function? For example, if enabled, convert the issue number to =HYPERLINK("https://github.com/gavinr/github-csv-tools/issues/31", "31") in the output CSV? That would enable an easy jump to the actual Issue page on GitHub. It appears that function is supported across various spreadsheet applications, including MS Excel, Apache OpenOffice, and Apple Numbers.

Add better examples to the documentation

We have details on the command options, but many people learn by example and we do not have many good examples. We should add a few of these to the README or some other documentation page.

npm install error

Hello,
When I'm trying to install the package using the following command : npm install -g github-csv-tools
I'm getting the error bellow
npm ERR! code ERR_TLS_CERT_ALTNAME_INVALID
npm ERR! errno ERR_TLS_CERT_ALTNAME_INVALID
npm ERR! request to https://npm.pkg.gitub.com//github-csv-tools failed, reason: Hostname/IP does not match certificate's altnames: Host: npm.pkg.gitub.com. is not in the cert's altnames: DNS:*.mytrafficmanagement.com, DNS:mytrafficmanagement.com

can you help me please.
Best regards
NKH

Error when running inside an enterprise

I installed the githubCsvTools globally and started the process as typing githubCsvTools in the command prompt

The program requested for Token, Org name, Repo name and errored out with below response

error RequestError [HttpError]: request to https://api.github.com/repos/ORG_NAME/REPO_NAME/issues?state=all failed, reason: self signed certificate in certificate chain
at C:\Users\test\AppData\Roaming\npm\node_modules\github-csv-tools\node_modules@octokit\request\dist-node\index.js:108:11
at processTicksAndRejections (internal/process/task_queues.js:93:5)
at async Job.doExecute (C:\Users\test\AppData\Roaming\npm\node_modules\github-csv-tools\node_modules\bottleneck\light.js:405:18) {
name: 'HttpError',
status: 500,
request: {
method: 'GET',
url: 'https://api.github.com/repos/ORG_NAMEREPO_NAME/issues?state=all',
headers: {
accept: 'application/vnd.github.v3+json',
'user-agent': 'github-csv-tools octokit-rest.js/18.6.0 octokit-core.js/3.5.1 Node.js/12.13.1 (win32; x64)',
authorization: 'token [REDACTED]'
},
request: { hook: [Function: bound bound register] }
}
}

Problem with line breaks

Hi,

Trying to use this module here and get the following error when I call the command via CLI

env: node\r: No such file or directory

Ive found another module that had the same issue but I could not solve it using the methods described there: nwutils/nw-builder#75

I never coded a npm module before so I dont know where to start to look to fix it...

Hope you have time to look into it

thanks

--exportAttributes labels

When I write the command

githubCsvTools --exportAttributes state,created_at,labels,milestone.title,milestone.created_at,milestone.due_on,number,title,body

I get all the label attributes, I just want to have the labels.name

but if I type labels.name i get nothing

Support for importing from other online Git services? (e.g. GitLab)

Hi, just used this tool to move over some issues from an organization GitLab instance to my public GitHub. I was just wondering if there's any plans to support CSV formats from other providers such as GitLab or Bitbucket? For GitLab's CSV in particular, the header names are capitalized and I needed to lowercase "Title" to "title" to get it to work, then realized I forgot to lowercase other fields such as "Description" and those didn't get ported over.

Just thought I'd post this here and if it's out of scope of the project feel free to close. :)

Import fails when there are multiple tag values

I was hitting various HTTP when trying to import a large csv of issues. Using the test csv files and some trial and error, it seems that the failure occurs as soon as it encounters the first instance of multiple tags.

So in this example, Test 1 is successfully created as an issue, but then fails.

title,body,labels
Test 1, This is the test1 desc, bug
Test 2, This is the test2 desc, "question, bug"

Here is was is returned

RequestError [HttpError]: Validation Failed: {"value":"bug","resource":"Label","field":"name","code":"invalid"}
    at C:\Users\...\github-csv-tools\node_modules\@octokit\request\dist-node\index.js:86:21
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async Job.doExecute (C:\Users\...\github-csv-tools\node_modules\bottleneck\light.js:405:18) {
  name: 'HttpError',
  status: 422,
  response: {
    url: 'https://github.enterprise/api/v3/repos/org/repo/issues',
    status: 422,

data: {
      message: 'Validation Failed',
      errors: [Array],
      documentation_url: 'https://developer.github.com/enterprise/2.21/v3/issues/#create-an-issue'
    }

 request: {
    method: 'POST',
    url: 'https://github.enterprise/api/v3/repos/org/repo/issues',
    headers: {
      accept: 'application/vnd.github.v3+json',
      'user-agent': 'github-csv-tools octokit-rest.js/18.9.0 octokit-core.js/3.5.1 Node.js/12.13.0 (win32; x64)',
      authorization: 'token [REDACTED]',
      'content-type': 'application/json; charset=utf-8'
    },
    body: '{"title":"Test 2","body":"This is the test2 desc","labels":["question"," bug"]}',
    request: { hook: [Function: bound bound register] }
  }

After export with comments, import generates separate issues.

Recently I had to migrate everything over to a new organization and this tool helped massively.

I found that, although the issue numbers were exported they were ignored (which I see now is in the code as title is the anchor) however it also generated a number of issues with the same title.

Is the behaviour intended to be able to import issues and comments or is it operating as expected?

(I did sort it out by manually moving the comments into a single issue line)

Rate Limiting on Import

Hi there! Me again...

So the throttling is better but I was apparently still too liberal. I'm still hitting rate limit issues with the current configuration (my test case is about 78 issues that I am importing, FYI).

Also, I'm checking for items in the queue before exiting the program but if I have requests that lag, it aborts early. So that needs adjusting too.

I will work on it and send you another PR; just wanted to let you know in case you get to it first.

Is there support for threaded replies

I'm looking to bring over issues from a different system that has multiple replies to the original issue. How is that handled using this application?

Title required by GitHub, but not found in CSV.

Really stocked by import a csv file (from azure) with (backlog)issues.
After alle information as token, organisation and repo, it gives me
Title required by GitHub, but not found in CSV.

But the first line in the csv is:
ID,Work Item Type,Title,Assigned To,State,Tags

I tested the csv online and in excel. Both work perfect, so the format is right. Even after conversion a rewritten file gives the same message.

I attached a small version with 1 sample item (Upload refush csv, renamed to txt). It give the same message
AzureIssues.txt

Can you help me to solve this, please.

Add Sample CSV

Would it be possible to add a sample CSV to the repository (or the README)?

Importing milestones causes 'Validation Failed' error

I get the error (below) when trying to import my csv with a milestone column.
I have no errors when I remove the milestone column.
The milestone already exists in my repo. Trying to use another existing milestone or a new milestone produces the same error.

RequestError [HttpError]: Validation Failed: {"value":"SprintJuly
1","resource":"Issue","field":"milestone","code":"invalid"}
at /usr/local/lib/node_modules/github-csv-tools/node_modules/
@octokit/request/dist-node/index.js:86:21
at processTicksAndRejections (internal/process/task_queues.js
:97:5)
at async Job.doExecute (/usr/local/lib/node_modules/github-cs
v-tools/node_modules/bottleneck/light.js:405:18) {

osx error: .env node\r

just to register, to make it work in OSX, one need to go to first line of installed githubCsvTools file, remove the line after and add a new line.

Attempted npm re-install does no upgrade the version

I've tried to upgrade from 2.0.0 to the latest with the following:

sudo npm install -g npm
sudo npm uninstall -g github-csv-tools
sudo npm install -g github-csv-tools

The version number doesn't appear to have changed at all.
I'm likely doing something wrong but I've not used npm before so please be gentle.

Members of a GitHub Org can no longer import

Prior to the last update (#82) both members and Owners could import. Now only Owners of an Org can import to repos in that Org. After reviewing the last update it appears that line 3 of helpers.js might be the cause.

Is there a way to only get issues and not pull requests?

Is there a way to only get issues and not pull requests?

Set ENV OCTOKIT_SSL_VERIFY_MODE=0 but still getting self-signed certificate error when exporting from Github Enterprise

I'm running using npx in Windows:

npm --version
6.14.9

npm install -g github-csv-tools
C:\Users\*****\AppData\Roaming\npm\githubCsvTools -> C:\Users\*****\AppData\Roaming\npm\node_modules\github-csv-tools\index.js
+ [email protected]
updated 1 package in 0.649s

$env:OCTOKIT_SSL_VERIFY_MODE=0

npx github-csv-tools --exportAll --github_enterprise https://*****/api/v3 --token ***** --organization ***** --repository ***** --verbose

npx: installed 35 in 2.765s
error RequestError [HttpError]: request to https://*****/api/v3/repos/*****/*****/issues?state=all failed, reason: self signed certificate in certificate chain
    at C:\Users\*****\AppData\Roaming\npm-cache\_npx\9540\node_modules\github-csv-tools\node_modules\@octokit\request\dist-node\index.js:108:11
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async Job.doExecute (C:\Users\*****\AppData\Roaming\npm-cache\_npx\9540\node_modules\github-csv-tools\node_modules\bottleneck\light.js:405:18) {
  status: 500,
  request: {
    method: 'GET',
    url: 'https://*****/api/v3/repos/*****/*****/issues?state=all',
    headers: {
      accept: 'application/vnd.github.v3+json',
      'user-agent': 'github-csv-tools octokit-rest.js/18.6.7 octokit-core.js/3.5.1 Node.js/14.15.2 (win32; x64)',
      authorization: 'token [REDACTED]'
    },
    request: { hook: [Function: bound bound register] }
  }
}

On CSV Import, receive error "Title required by GitHub, but not found in CSV."

When I try to import the attached CSV, I receive an error "Title required by GitHub, but not found in CSV."

Is this CSV malformed somehow?

OH_drawing_issues.csv

Export CSV with comma

When i export to CSV and i have some comma , into my issue's text visualisering is not clear definited

Importing to specific projects

Is there any way to import issues to the specific project that is linked to many repositories?

Support the import of pull requests? Our fork supports some functionality

Hello, is there any interests to support the importation of Pull Requests? We are aware there are some "gotchas" when trying to import PRs with GitHub's API as certain edge cases are not supported. However, we have modified our fork of the code to support some cases of importing PRs. We have added documentation to our fork in the README.md file. If there is interest, please checkout our fork with the code that supports importing PRs here and if all is well let us know to open a PR to this repository. We are willing to make small changes to our fork so that a PR can be approved and merged into this repository but we can not spend much more time on our fork.

Thanks!

Branch with our changes: https://github.com/CityOfLosAngeles/github-csv-tools/tree/import-pull-requests
Diff with our changes as of now: master...CityOfLosAngeles:import-pull-requests.

Request error

Hey there,
After entering the repository, I am getting Http Error. Any idea why this is happening?

Error: Error reading file

Hi there,

our goal is to import a bulk of issues to our enterprise github organisation. However, each time that we try, we get the error in the screenshot. We have tried modifying the csv file with different seperators, and also different inputs for the organization.
Could somebody help out to see what we do wrong here?
Thanks!

How do we ensure labels pick up some colour ?

We used this utility to bulk import but all new labels it created are of same colour.
How do we ensure it takes different colours for different labels ?

test 1

Alert the user on error if including spaces in "--exportAttributes"

Great tool. As feedback the tool could look at spaces, or alert the user as the following was shoing an error:

githubCsvTools -f doar2.csv -c -e -t ghp_TkQxXXXXXXXXXXXXXv6s2Gpbm1eX9yY -o myorgname -r myrepo --exportAttributes id, title, labels, comments, body

error:
Error reading file.

I had to call with no spaces in the:
--exportAttributes id,title,labels,comments,body

Import Milestones

Modify import operation to not import milestones by default.

Add option, --importMilestones that will take a milestone title column, and create the milestones first.

Be sure not to create the same milestone twice during the import option, save the IDs and use those.

Silent fail when spaces are present in tags list —

I imported a csv with 131 rows. The issues field had spaces in them, e.g. "Engagement, Plugin, slam"

github-csv-tools ran as expected listing "Imported issue:" for each row present.
Output ended with Created 131 issues, and had 0 failures.

However, 0 issues were present in the Github Issues for that repo.

I edited the file to remove all spaces between items and to remove any empty items e.g. ,,.
Re-ran import in verbose mode.
github-csv-tools ran as expected listing "Imported issue:" for each row present.
Created 131 issues, and had 0 failures.

However, there are only 110 items in the Issue. Some issues silent failed and I don't know why.

tool adds /repos/* to url?

Hi, I have an enterprise url im trying to get issues from, but the tool adds /repos to my url, and it does not resolve in my enterprise github. how can i remove this added /repos ?

 url: 'https://github.internal/repos/Dev/foo/issues?state=all',

Linting

Enable ESLint on the repo

githubCsvTools immediately returns with no output

githubCsvTools ./issues.csv

Just returns doing nothing, doesn't ask for OAuth token or any other information. Any idea what I might be missing?

Allow comment import

Now that #82 is merged, the only thing stopping me from importing a repo with ~4000 issues is the lack of support for comments. The api endpoint supports them, but it is not clear to me how to structure the csv file to have comments in them. Is that something you are interested in supporting?

Time output

Error message:
Request quota exhausted for request GET https://api.github.com/repos/dotCMS/core/issues/16215/comments
Retrying after 2525 seconds!

You can access https://api.github.com/repos/dotCMS/core/issues/16215/comments and find that it can not respond.
For this situation, it needs to ignore and skip this url.

Import issues error if using -e flag

Can export the issues from a repo into the CSV generated file as expected but when importing to another repo an error status 422 is thrown.

Operation: githubCsvTools -e then githubCsvTools [path to file]

In version 3.1.2

Error message in console:

RequestError [HttpError]: Validation Failed: {"value":"\"url\":\"https://api.github.com/repos/.../labels/UX\"","resource":"Label","field":"name","code":"invalid"}

at ...\npm\node_modules\github-csv-tools\node_modules\@octokit\request\dist-node\index.js:86:21
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async Job.doExecute (...\npm\node_modules\github-csv-tools\node_modules\bottleneck\light.js:405:18) {
  status: 422,
  ...

Node version: v12.18.3
Windows system32

Only seeing the first comment when using -c flag

I can see the number of comments on an issue, but only the first appears in the CSV file when I use the -c flag. Any ideas? They aren't in the body column either