coreos / issue-sync

A tool for synchronizing issue tracking between GitHub and JIRA

issue-sync

Issue-sync is a tool for mirroring GitHub issues in JIRA. It grew out of a desire to maintain a public GitHub repo while tracking private issues in a JIRA board; rather than require people to keep up with both sources, we decided to make one the single source of truth. Note that issue-sync works only one way and will NOT mirror issues from JIRA to GitHub.

Usage

JIRA Configuration

To use, first ensure you have a JIRA server with the project you want to track on it - it can be a cloud account, or self-hosted. Also make sure you have a user account that can access the project and create issues on it; it's recommended that you create an account specifically for the issue-sync tool.

Add the following custom fields to the project: GitHub ID, GitHub Number, GitHub Labels, GitHub Status, GitHub Reporter, and Last Issue-Sync Update. These fields are required and the names must match exactly. In addition, GitHub ID and GitHub Number must be number fields, Last Issue-Sync Update must be a date time field, and the remainder must be text fields.

If you intend to use OAuth with JIRA, you must create an inbound application connection and add a public key. Instructions can be found in OAuth for Rest APIs.

Application Configuration

Arguments to the program may be passed on the command line or in a JSON configuration file. For the command line arguments, run issue-sync help. The JSON format is a single, flat object, with the argument long names as keys.

Configuration arguments are as follows:

Name	Value Type	Example Value	Required	Default
log-level	string	"warn"	false	"info"
github-token	string		true	null
jira-user	string	"[email protected]"	false	null
jira-pass	string		false	null
jira-token	string		false	null
jira-secret	string		false	null
jira-consumer-key	string		false	null
jira-private-key-path	string		false	null
repo-name	string	"coreos/issue-sync"	true	null
jira-uri	string	"https://jira.example.com	true	null
jira-project	string	"SYNC"	true	null
since	string	"2017-07-01T13:45:00-0800"	false	"1970-01-01T00:00:00+0000"
timeout	duration	500ms	false	1m

Configuration Key Descriptions

log-level is the minimum level which will be logged; any output below this value will be discarded.

github-token is a personal access token used to access GitHub as a specific user.

jira-user and jira-pass are the username (i.e. email) and password of the JIRA user which will be authenticated. See Authentication for more details.

jira-token and jira-secret are OAuth access tokens which will be used to perform an OAuth connection to JIRA. jira-consumer-key and jira-private-key-path are the RSA key used for OAuth. See Authentication for more details.

repo-name is the GitHub repo from which issues will be retrieved. It must be in the form owner/repo, for example coreos/issue-sync.

jira-uri is the base URL of the JIRA instance. If the JIRA instance lives at a non-root URL, the path must be included. For example, https://example.com/jira.

jira-project is the key (not the name) of the project in JIRA to which the issues will be synchronized.

since is the cutoff date issue-sync will use when searching for issues to synchronize. If an issue was last updated before this time, it will not be synchronized. Usually this is the last run of the tool. It is in ISO-8601 format.

timeout represents the duration of time for which an API request will be retried in case of failure. Human-friendly strings such as 30s are accepted as input, although the application will save it to the file in a number of nanoseconds.

Configuration File

By default, issue-sync looks for the configuration file at $HOME/.issue-sync.json. To override this location, use the --config option on the command line.

If both a configuration file and command line arguments are provided, the command line arguments override the configuration file.

After a successful run, the current configuration, with command line arguments overwritten, is saved to the configuration file (either the one provided, or $HOME/.issue-sync.json); the "since" date is updated to the current date when the tool is run, as well.

Authentication

If jira-user or jira-pass are provided, both are required, and the application will connect to JIRA via Basic Authentication.

Otherwise, OAuth will be used. In this case, the jira-consumer-key, which is the name of the RSA public key on the JIRA server, and the jira-private-key, which is the path to the RSA private key which matches, must be provided.

If the jira-token and jira-secret are provided, they are used as the OAuth access token.

If they are not provided, an OAuth handshake occurs, and an authorization URL will be given. The user will need to open the URL in their browser, and receive the authorization code provided. Once the code is entered into the application, an access token will be generated, and it will be added to the configuration for future use.

issue-sync's Issues

Project fairly quiet

Looking at commits, issues and pull requests this repository looks fairly quiet. Given that this lives in the coreos org, am I right in assuming that this project at the moment is no longer used/ maintained by it's original authors?

Updating comments causes slice bounds out of range

This block of code https://github.com/coreos/issue-sync/blob/master/lib/clients/jira.go#L258 will almost always cause slice bounds out of range. I believe the intent was to trim body when it is longer than max body length. Will follow up with a pull request to fix this.

Test Issue with Comments

This issue can be used to have comments on it for testing comments in issue-sync.

Comment regex fails when username has non-letter/number characters

Discovered that the regex that parses out the meta data at the top of jira comments fails when the github username has a dash (or any other non-letter/number) character.

Issue with Jira - "issue type is required"

Hello,
I just installed and run issue-sync and got this error message:

root@jira:/usr/lib/go-1.7/src/github.com/coreos/issue-sync# bin/issue-sync --config ./config.json -u admin -p vohx9Shoofen6cei
INFO[0000] config file loaded                            file="./config.json"
INFO[0000] log level set                                 app=issue-sync log-level=debug
DEBU[0000] Checking config variables...                  app=issue-sync
DEBU[0000] Using HTTP Basic Authentication               app=issue-sync
DEBU[0000] All config variables are valid!               app=issue-sync
DEBU[0000] JIRA clients initialized                      app=issue-sync
DEBU[0000] Collecting field IDs.                         app=issue-sync
DEBU[0000] All fields have been checked.                 app=issue-sync
DEBU[0000] Successfully connected to GitHub.             app=issue-sync
DEBU[0000] Collecting issues                             app=issue-sync
DEBU[0004] Collected all GitHub issues                   app=issue-sync
DEBU[0004] Collected all JIRA issues                     app=issue-sync
DEBU[0004] Creating JIRA issue based on GitHub issue #36  app=issue-sync
ERRO[0004] Error performing operation; retrying in 507ms: Request failed. Please analyze the request body for more details. Status code: 400  app=issue-sync
...
ERRO[0076] Error creating JIRA issue: Request failed. Please analyze the request body for more details. Status code: 400  app=issue-sync
DEBU[0076] Error body: {"errorMessages":[],"errors":{"issuetype":"issue type is required"}}  app=issue-sync
ERRO[0076] Error creating issue for #36. Error: {"errorMessages":[],"errors":{"issuetype":"issue type is required"}}  app=issue-sync
DEBU[0076] Creating JIRA issue based on GitHub issue #37  app=issue-sync
ERRO[0076] Error performing operation; retrying in 389ms: Request failed. Please analyze the request body for more details. Status code: 400  app=issue-sync

I created a new Jira project and configure Jira with:

• all required fields
• new field configuration
• new field configuration scheme
• new screen
• new screen scheme
• new issue type screen scheme
  The issue type screen scheme and field configuration scheme are applied to the new project.
  We are using Jira 7.5.0
  I have no idea, where this error comes from :-/

Test issue to test the status and reporter fields

New custom fields were created. Let's test them!

issue-sync sends datetime in wrong format for Last Issue-Sync Update

ERRO[0008] Error performing operation; retrying in 3.04s: Request failed. Please analyze the request body for more details. Status code: 400 Body: {"errorMessages":[],"errors":{"customfield_11310":"Error parsing time: 2018-07-30T11:47:18+0200"}} app=issue-sync

Upgrade Dependencies

Upgrade all dependencies.

Comment regular expression is fragile

In some comments in our repository the comments regexp in comments.go fails to match at all, and sync fails since fields is empty.

As a ugly quickfix I had to add

if len(fields) != 5 {
		log.Debug("Failed to update JIRA comment for ", jIssue.ID)
		return nil
	}

To UpdateComment to make sync work. Also writing the config to home failed on macOS, but worked when I specified a full file path. I can add a separate issue about that if needed.

Change Readme: Clarify that this tool only syncs content from Github to Jira and not the other way around

Is github 2fa supported?

I am considering using this for work but we have 2fa enabled on github accounts. Is this supported and does it make things more complicated?

Test Issue Please Continue to Ignore

A quick issue that I can update all I want just to test the issue updating feature.

Issue with JIRA Cloud (Kanban)

DEBU[0003] Creating JIRA issue based on GitHub issue #1 app=issue-sync
ERRO[0003] Error performing operation; retrying in 552ms: Request failed. Please analyze the request body for more details. Status code: 400 app=issue-sync

Wish i had more for ya :/

Add unit tests

I'm looking to use this for github -> jira workflow integration. I'd like to add unit tests to ascertain stability and help justify using this open source project at my org. I'll be following up this issue with pull requests as I add unit tests. I'm opening this issue now to start discussion with anyone who's interested in providing feedback/suggestions.

Refactor for testability

Refactor for improved testability (in preparation for adding unit tests).

example for compiling

attempting to build on Mac OS with homebrew go1.12.4 installed

seeing the following output
can't load package: package github.com/coreos/issue-sync: cannot find package "github.com/coreos/issue-sync" in any of:
/usr/local/Cellar/go/1.12.4/libexec/src/github.com/coreos/issue-sync (from $GOROOT)
/Users/paulcesario/go/src/github.com/coreos/issue-sync (from $GOPATH)

can you provide example or instructions on how to build?

Add GitHub Project Board to Jira State Syncing

This feature may be generally useful however I implemented to suit my immediate needs. I'd like to open up a discussion about adding this feature to issue-sync proper.

Usage Examples?

Are there any examples of this in real world use that we can see on both the GitHub and JIRA sides?

Move to Go modules

The go community (along with other associated tools) have moved on to using go modules as is it now built into the language. This issue is to track migration over to go modules (as opposed to glide).

When updating issues, only first 50 Jira issues are considered.

When comparing Jira and GH issues during an update, the code queries issues from GH and then tries to find the matching issues from Jira. But, if there are many GH issues then this branch is executed: https://github.com/coreos/issue-sync/blob/master/lib/clients/jira.go#L130.

It seems like we could filter the issues ourselves, but the query only returns the 50 first issues. If the GH id doesn't match, then a new issue is created although a matching Jira ticket could be found beyond the first hits.

My workaround for this was to move the Jira query https://github.com/coreos/issue-sync/blob/master/lib/issues.go#L40 inside the following GH issue loop, so we can query Jira one GH id at a time. I didn't see any major performance hit and the script stopped creating duplicate issues to Jira.