Add the linting rules and set up the project to test linting on CI

Upon registering a rule for a repository, we should fetch the repository's ID and save it to the database so that we'll still be able to work with those repositories even if they are renamed.

addProjectV2ItemById is not working on private repositories.

I have been calling the following mutation:

mutation($project: ID!, $issue: ID!) {
  addProjectV2ItemById(input: {projectId: $project, contentId: $issue}) {
    item {
      id
    }
  }
}

and it does work on public repositories with public issues, but when I call this same endpoint in a private repository I get the following error:
GraphqlResponseError: Request failed due to following response errors:.

I do not know if this problem comes from my side so I created a discussion in GitHub's organization.

addProjectV2ItemById does not work in private repositories #46505

ProjectNext query is deprecated and soon will stop working.

We need to update to use ProjectV2.

Example call

Documentation

Problem:
Known fact: projects may include issues from different repositories. When team is working with all issues in a project, it'd be nice if labels are "speak same language", e.g are consistent across repos. Otherwise it's hard to aggregate/group/sort and expect query to work consistently.

Example 1: team might use specific levels of prioritizations like "P1", "P2", "P3" (with some nice icons), while one repo has P1 -🔥 the other one would be just P1. While meaning is the same - there's no way to filter all issues by P1 priority, as filters technically are different from repo to repo.

Example 2: as a team, we're reviewing all tickets from the board, and adding according labels to all issues from different repo's which help then to group & sort issues efficiently - (like effort: 🔥🔥, impact: 💎💎, ¯\_(ツ)_/¯, etc) => since labels are inconsistent - then the marking doesn't allow us to sort/group them efficiently too

In order to keep project with several repos consistent, we need to figure out the way to define "project level" labels, which may be optionally distributed across the repositories.
I'm not sure exact constraints yet, how this will operate with existing labels etc, but we can discuss

Add support for custom project fields.

Add the ability to add extra values when moving the issue to a board.

Must be done after #43

Problem: github-issue-sync only syncs issues from future events, which means it will not sync existing issues until the next event is given for them.

Solution: create API endpoint for syncing all issues in a repository

Create a configuration file for the following projects:

• license-scanner
  • project: 16
  • Tool
  • license-scanner
• pipeline-scripts
  • project: 16
  • Tool
  • pipeline-scripts
• parity-processbot
  • project: 16
  • Tool
  • parity-processbot
• pr-custom-review
  • project: 16
  • Tool
  • pr-custom-review
• substrate-matrix-faucet
  • project: 16
  • Tool
  • substrate-matrix-faucet
• substrate-tip-bot
  • project: 16
  • Tool
  • substrate-tip-bot
• command-bot
  • project: 16
  • Tool
  • try-runtime-bot
• opstooling-js-style
  • project: 16
  • Tool
  • generic
• opstooling-js
  • project: 16
  • Tool
  • generic
• matrix-admin-bot
  • project: 16
  • Tool
  • matrix-admin-bot
• roadmap
  • project: 27
  • Status
  • Draft

It seems trivy is not the best option to analyze our repository as it doesn't differ production dependencies from dev dependencies for yarn projects.

As recommended by @mutantcornholio in #30 we should replace this check for Dependabot instead as it can differ production dependencies from dev dependencies.

Problem

Currently all sync rules are managed through the API. We provide some curl examples for how to use the API but there are alternatives for further enhancing the operators' usability experience.

Solution

To make the sync rules' management easier to work with for operators, we could

Alternative 1: Provide OpenAPI specification for the API. That way operators could use the specification in whatever REST client they want e.g. Postman, Insomnia, etc.

Alternative 2: Implement CRUD Rules UI.

Currently in order to create a rule we call https://github-issue-sync.parity-prod.parity.io/api/v1/rule/repository/{org_name}/{repo_name} while required to provide project_field and project_field_value. Setting those fields shouldn't be mandatory in order to create a rule.
The only mandatory field should be project_number

We need release/deployment instructions for maintainers like the one showcased in https://github.com/paritytech/parity-processbot/tree/2f8ffedb62a2abc1a3b50c983087e80c1d8677be#deployment

For other projects we usually run the deployment pipeline on CI for certain tags, as showcased in https://github.com/paritytech/parity-processbot/tree/2f8ffedb62a2abc1a3b50c983087e80c1d8677be#deployment.

This will be done after #2

Why?

The idea of this change is to provide 3 improvements:

• Remove the need of a server.
• Greatly improve Developer Experience (DX) so anyone can use it.
• Rewrite the app to have a more consistent codebase.

Remove the need of a server

Rewriting the GitHub app into an action would allow us to execute it as a single execution function, removing the need to have a running instance of this app in a server.

Greatly improve DX

Currently, when someone decides to modify anything with the app, they must contact an API endpoint with a credential. This implementation is difficult which ends up requiring the OpsTooling to do it ourselves. So, when implementing this system we are the ones who need to install it. By converting it into an action other users can use this app without the need to have us intervene.

Rewrite the app to have a more consistent codebase

The app has several entry points and connections, this document proposes a simplification of it. More info available in the architecture section.

Action

Entrypoint

The action's entrypoint will be set up by two kind of events:

on:
  issues:
    # updates when a issue is modified
    types:
      - opened
      - reopened
      - labeled
      - transferred
  # this can be manually triggered by going to the action's tab in the repository
  workflow_dispatch:
    inputs:
      syncIssues:
        description: 'Sync all issues available in the repository'
        required: true 
        type: boolean 

It will sync with two types of events, automatically, when a issue is modified, or manually, when the workflow is manually triggered by going to the action tab.

Configuration

At the time of configurating rules we can set everything in the action file.

with:
  # Token with the following permissions: write:projects. 
  token: ${{ secrets.PROJECTS_TOKEN }}
  # The number of the project which the issues will be synced to
  project: 123
  # The name of the project field which the issue will be assigned to
  target-project-field: Team
  # The value which will be set in the field, in this case the team's name
  target-project-field-value: Foo
  # Optional field, only execute on issues with a given label
  label:
    - ops
    - ci
    - leads

For the token we would require one that has access only to project, allowing us to link the issue to a particular project. Some experimentation is required in this area to find the ideal permission access but it should be easy to do.
The token could be added either as per repository or a global one for the organization.

Flow

The flow would be quite straightforward, if the action is initialized over a workflow_dispatch, it will fetch all the issues from the project and iterate over them assigning them the correct condition.
But, if it's initialized through a Issue event, it will check which condition triggered the action and will act accordingly. You can visualize the flow in the following diagram:

flowchart LR
Start-up --> SyncOrIssue{Is it a\nworkflow dispatch?}
SyncOrIssue --> |Yes|Fetch[(Fetch all\nissues ID)]
Fetch --> Assign[[Assign Issue\nto Project Ids]]
SyncOrIssue --> |No|Switch((Switch over\nthe issue state))
Switch --> opened --> Assign
Switch --> transfered --> Assign
Switch --> labeled --> Label[[Calculate condition]] --> Assign
Switch --> closed --> ChangeState[[Change Issue\nState in Project]]
subgraph issue-event
opened
transfered
labeled
closed
end

Architecture

The architecture will be defined in an entry point (the core) and will initialize all the required values, sending everything to a business layer that will be the one in charge of executing the side effect.

flowchart LR
A(Event) --> |Action is initialized| Core(Core)
Core(Core) --> |Initialized GitHub app| Business(Business layer)
Business --> Logger(Logger)
Business --> Octokit(Octokit)
    subgraph Application
        Business
        Logger
        Octokit
    end

The core will parse all the configuration coming from both the event and the action workflow file and parse it into an agnostic object:

// Example of the event data
interface IEventData {
	issueId?: number;
	event: "opened issue" | "closed issue" | "labeled issue" | "transfered issue" | "sync all issues"
	issueState?: "open" | "closed"
	config: {
		project: number,
		targetField: string,
		targetValue: string
	}
}

Everything related to interacting with GitHub (modifying the state, fetching configurations, etc) will be encapsulated in an wrapper that will be accessed by the business layer:

interface IGitHub {
	assignProjectToIssue(issueId: number, projectId: number): Promise<boolean>;
	getIssueState(issueId: number): Promise<"open" | "closed">;
	getAllIssuesId(): Promise<number[]>;
	getProjectIdFromIssue(issueId: number): Promise<number>;
	changeIssueStateInProject(issueId: number, state: "todo" | "in progress" | "blocked" | "done");
}

Everything related to interacting with the logging system will also be wrapped in its own wrapper:

interface ILogger {
	log(message: string): void;
	warn(message: string): void;
	error(message: string | Error): void;
}

This would allow us to have an agnostic business logic where we can write a clean and easy to understand logic.

const octokit: IGitHub;
const logger: ILogger;
/** Overly simplified example of how it would work **/
async function execute(eventData: IEventData) {
    const { issueId, event } = eventData;
    if (event === "opened issue") {
        await octokit.assignProjectToIssue(eventData.issueId, eventData.config.project);
        logger.log(`Assigned issue #${issueId} to project ${eventData.config.project}!`);
    } else if (event === "closed issue") {
        if (await octokit.getProjectIdFromIssue(issueId) > 0) {
            await octokit.changeIssueStateInProject(issueId, "done");
            logger.log(`Updated issue #${issueId} to done!`);
        }
    } else if (event === "sync all issues") {
        const issues = await octokit.getAllIssuesId();
        for (let i = 0; i < issues.length; i++) {
            await octokit.assignProjectToIssue(issues[i], eventData.config.project);
            logger.log(`Assigned issue #${issues[i]} to project ${eventData.config.project}!`);
        }
    }
}

This would also allow us to test the logic by substituting the dependencies. And if, for whatever reason, we want to port this code to be a GitHub app or to run in GitLab, the abstraction would make it quite straightforward.

Multiple project configuration

By setting up multiple executions, the action can be executed multiple times for different projects, allowing some better level of customization.

on:
  issues:
    # updates when a issue is modified
    types:
    - opened
    - reopened
    - labeled
    - transferred
  # this can be manually triggered by going to the action's tab in the repository
  workflow_dispatch:
    inputs:
      syncIssues:
        description: 'Sync all issues available in the repository'
        required: true 
        type: boolean 
jobs:
  lead-board:
    runs-on: ubuntu
    steps:
      - uses: paritytech/github-issue-sync
        with:
          token: ${{ secrets.PROJECTS_TOKEN }}
          project: 123
          label:
          - leads
  dev-board:
    runs-on: ubuntu
    steps:
      - uses: paritytech/github-issue-sync
        with:
          token: ${{ secrets.PROJECTS_TOKEN }}
          project: 321
          target-project-field: Team
          target-project-field-value: Foo
          label:
          - engineering
          - ci

Completion requirements:

• Works as an action.
• Has an updated readme
• Has unit tests.
• Has been tested in some system.

Problem: Right now our rules work on a per-repository basis. It would be nice if we could say that "not X and Y but all repositories for a given team should be synced according to this rule".

i.e. when I add a new repository to https://github.com/orgs/paritytech/teams/opstooling/repositories it could automatically be synced to the board instead of having to manually create a rule for it.

Solution: Create team-wide project rules which specify how any issue from repositories assigned to a given team will be synced to the board.

all opstooling + https://github.com/paritytech/roadmap

Add support to only link issues when a particular label is present (or when no labels are present).

This way we can have more than one action running and link them accordingly to different states and boards.

pre-commit run --color=always --all-files