codechecks / monorepo

Open source platform for code review automation ✅

Code Checks

Open source platform for code review automation ✅

CodeChecks.io is a framework for code review automation. It allows writing plugins (codechecks) in JavaScript that:

• execute directly on your CI (like CircleCI or TravisCI) and communicate with our backend
• can inspect all Pull Request data (title, content, changed files etc.)
• can attach feedback information back to GitHub as check (success/failure status, additional message)
• can upload, retrieve and calculate difference between arbitrary artifacts (like JSON values describing build size or screenshots of views of a frontend app)
• require only one GitHub App — Codechecks App installed in order to work

import { codechecks } from "@codechecks/client";

export default async () => {
  await codechecks.success({
    name: "Hello world!",
    shortDescription: "It's so easy!",
    longDescription: "# Hello! 👋",
  });
};

Particular codechecks can be distributed as npm packages and can be setup in a declarative way in codechecks.yml file.

Docs 📚

• Awesome codechecks
• Getting started guide
• Full documentation

Packages 📦

Package	Version	Description
@codechecks/client		TypeScript/JavaScript client used for executing codechecks

Note: Currently we are in the process of open sourcing whole platform.

Contributors ✨

Thanks goes to these wonderful people (emoji key):

Chris Kaczor 💻 🤔 📖 💼 🚇	cgewecke 🔌 🤔 💻	Evgeniy Timokhov 🔌	Michał Załęcki 💻 🔌	Craig Thompson 💻	Jakub Ziółkowski 💻 🔌	Deric Cain 💻
Sean Dawson 🔌

This project follows the all-contributors specification. Contributions of any kind welcome! Read more!

monorepo's Issues

Installing failes NPM audits

Docs for custom installation

Hey!

Could you please provide some documentation on how to create and install own custom GitHub codechecks app on closed enterprise GitHub installation? Or just provide a link to GitHub app sources, I cannot find them.

Automatically verify if user can run a given plugin on theirs node version

Basically this:

const pkg = require("check/package.json");
require("please-upgrade-node")(pkg);

Can't get project slug for https repo origin

I think this regex should cover https-based origins as well.

Question about speculative branches

Hi @krzkaczor :) hope all's well.

I'm missing historical data in a check installed on JoinColony/colonyNetwork. They use CircleCI and usually merge to develop. Their .yml looks like this:

checks:
  - name: eth-gas-reporter/codechecks
settings:
  branches:
    - develop
    - master

Everything about the check looks correct (to me):

• the target and base branches seem right.
• ci runs on commit
• no errors

Log output

CodeChecks Client v.0.1.10
https://codechecks.io

Executing 1 codechecks files
Base branch: 2a9ec03b

Executing codechecks.yml...

Checks: 1 succeeded, 1 total
Time:   2s
Done in 2.92s.

But nothing saves. An example of everything working perfectly on CircleCI can be seen at omise-go/plasma-contracts. They don't use speculative branching though.

My codecheck code looks like this.

Do you have any thoughts about why this might be / ideas for debugging it?

Support DroneCI

https://drone.io

Missing dependency on typescript

@codechecks/[email protected] has a dependency on node-ts ^8.0.2. But [email protected] has a peer dependency on typescript >=2.0, and [email protected] (which is what gets installed today) has a peer dependency on typescript >=2.7.

Since the CodeChecks client module is the one that depends on the module with the peer dependency, as I understand it the CodeChecks client module is the one that is responsible for depending on it as a non-peer dependency. There should be a dependency on typescript >=2.7 to satisfy node-ts, and this dependency needs to be bumped if/when node-ts releases a new version that matches ^8.0.2 and bumps its peer dependency version.

Without this dependency, I get this at the end of npm ls:

npm ERR! peer dep missing: typescript@>=2.7, required by [email protected]

Remove ts-node dep?

I can't see this being called anywhere, was this meant to be added as a devDep?

https://github.com/codechecks/monorepo/search?q=ts-node

Client version 0.2.0 — Breaking changes summary

Proposed breaking changes:

• do not implicitly initialize client but rather make it explicit like:

const client = new CodechecksClient();

instead of

import { client } from "@codechecks/client";

This will result in multiple benefits like a possibility to provide defaults for a check, client can execute checks in parallel etc.

• rename directory dist to lib in final package

• isFork should be a boolean in execution ctx

• rename saveValue/getValue to saveJSON/getJSON

• support scope setting. Useful for monorepos with multiple codechecks files. All codechecks artifacts and titles would share some scope

• by default, it exit with non-zero exit code if any report failed as discussed (#22)

Make common things in checks very easy to do

We should have an api to:

• execute processes (execa like)
• create tmp dirs (that perhaps should share same prefix)

Getting MaxBufferError: stdout maxBuffer exceeded

Getting MaxBufferError: stdout maxBuffer exceeded

not sure what caused this. it used to work. you can see the failure here : https://github.com/thesandboxgame/sandbox-smart-contracts/runs/1643040725?check_suite_focus=true

Is codecheck running out of memory. any way to fix this via some settings ?

Checks missing on cached push builds (CircleCI)

Just continuing a discussion at eth-gas-reporter with @krzkaczor about the codechecks client/server model and the difficulty of identifying "where" one is in the CI cycle.

My understanding of the issue is:

• CI caches push builds and applies them to opened pull requests. In these cases, no PR environment variable is set in the Circle container.
• The codechecks client relies on being able to extract the PR ID from the CI environment.
• The codechecks client relies on push builds being enabled to save historical data.

The end result is that checks don't automatically run on the initial commit of a pull request, and anyone who only builds on pull requests won't be able to store data in the codechecks backend.

@krzkaczor has suggested some work-arounds but I'd like to advocate for moving to a system that

• subscribes the user's repo to webhooks for push and pull request events
• does not use isPR to toggle reporting on and off. Instead it caches the relevant data on the codechecks backend for all builds (like Circle).
• does context detection using the server's webhook event history to get the appropriate commit hashes to index data by and trigger checks at the right time.

This approach is more complicated, difficult to maintain, etc. But it may map more closely to how Github intends its checks API to work. There are also nice UX benefits. For example, when someone opens a PR you could fetch the user's codechecks.yml and immediately trigger 'pending' notifications in their GH UI for whatever they've listed there.

Apologies in advance if I've misunderstood something basic and everything above is nonsense. As in #5 - if you have any interest in volunteer help with something like this, please just let me know. Codechecks is super useful/powerful and there's lots of stuff I'd like to build on top of it.

Github Actions `isFork` False Positive

I am observing an error running codechecks in Github Actions - Provider should never be in fork mode and not in PR mode!
It appears as if there is a false positive as the PR is not coming from a fork. The code is here:

monorepo/packages/client/src/getExecutionContext.ts

Lines 20 to 25 in 13986fe

	const isFork = await ciProvider.isFork();
	const pr = await ciProvider.getPullRequestID();
	const projectSlug = await ciProvider.getProjectSlug();
	if (!pr && isFork) {
	throw new Error("Provider should never be in fork mode and not in PR mode!");
	}

For Github actions, !pr is always true

monorepo/packages/client/src/ci-providers/GithubActions.ts

Lines 14 to 16 in ba52ad8

	getPullRequestID(): number | undefined {
	return undefined;
	}

And isFork is always true because GITHUB_HEAD_REF is set for PRs submitted from the same repo. See the docs here: https://docs.github.com/en/actions/reference/environment-variables

monorepo/packages/client/src/ci-providers/GithubActions.ts

Lines 27 to 31 in ba52ad8

	isFork(): boolean {
	// This is only set for forked repositories
	// @see https://help.github.com/en/articles/virtual-environments-for-github-actions#default-environment-variables
	return !!this.env["GITHUB_HEAD_REF"];
	}

GITHUB_HEAD_REF	Only set for pull request events. The name of the head branch.

Perhaps there is a better way to determine there the PR is coming from a fork, will update if I determine a good solution

Fail if any check has been failures

Currently if you run codechecks it always exited with code 0, even if any check is failed. I'm not sure, but I believe CLI should fail in that case as well.

Migrate from mocha to jest

It was a long time coming. Private monorepo is already halfway migrated to jest.

Main reasons for the migration are (inline) snapshots, they are a pleasure to work with. I might even write a full blog post about it soon.

Feedback request: codechecks wrapper for dependency-cruiser

I'm working on dependency-cruiser integration. It will allow for a custom set of rules but I wanted to provide a value for people unfamiliar with the dependency-cruiser by providing some predefined rules. I welcome any feedback.

https://github.com/MichalZalecki/codechecks-dependency-cruiser

Show graph of the check's metrics

I'm not sure that it's right place to fill this issue, but it looks like codechecks' server isn't open-sourced right now...

Some checks (for instance type coverage or file size) have numeric metric values, which can be shown on the graph and then users can compare them as time goes on (for example see how much every commit adds in size to the library or something).

I believe it might be very useful to track changes into historical perspective.

Track all contributors to the whole codechecks ecosystem

This is an issue to track all contributions to all CC packages.

Publish @codechecks/prettier-config

@krzkaczor It would be nice if plugin creators could reuse the config files for prettier (and also for linter, #38?).

Support Github Actions

Are you on board with adding Github as a service provider? I forked the repo and started the work already (just need to finish the tests). If you are okay want to move forward with it, let me know and I will create a PR. It seems to be pretty straight forward.

Thanks!

Jenkins Support?

Jenkins Support

Switch from TSLint to ESLint

@krzkaczor I would like to know your thoughts on switching from TSLint to ESLint due to the planned depreciation.

Cleanup on error

https://stackoverflow.com/questions/14031763/doing-a-cleanup-action-just-before-node-js-exits

Track unused code

Hello and thank you for your work.

I was wondering if you're interested in making Code Checks track how much dead code is introduced by a PR.

The problem

When making changes to a codebase, it's common to leave code that is exported but never used. It was useful once, but it's not anymore. Unless there are good reasons to leave it in the project, it should be deleted and restored from Git history if need be.

Solutions

There are tools capable of spotting such scenarios:

• @dsherret's no-unused-exports
• https://github.com/plantain-00/no-unused-export
• https://github.com/nadeesha/ts-prune
• https://github.com/pzavolinsky/ts-unused-exports

The problem with some of these tools is that they find too many false positives or they don't provide an escape hatch to mark something as not used now, but let's leave it there. For this reason, using them to prune dead code in a precommit/prepush task may be too aggressive.

I believe there is a use case for them though. If a CI check detected the amount of dead code has increased (and presented the Δ), then this information alone is already useful. I'm not sure if the build should fail or not, but merely being presented with such a warning could help keep our codebases cleaner.

Please let me know what you think, and thanks again for your wonderful tool.

Be nicer to local repositories with not fully setup git

Currently it will fail if:

• git repo doesn't exist,
• doesnt have any commits,
• doesnt have remote

all of them can be dealt with by ussing dummy defaults

Cleanup client's npm bundle

Remove tests and other garbage from NPM package. Original issue by @timocov

Add support for Semaphore CI

CC: @marktran

Expose debug for plugins

Better error messages

Currently, client will always print out only error.message which looks nice but is totally unpractical for unexpected errors. You need to use DEBUG=codechecks:* to get those.

What would be desired is to have our subclass of errors something like UserFacingError which would print out a human-friendly error message. On unexpected errors, we should print out "ugly" error with full stack trace for debugging purposes.

CC: @MichalZalecki

Add support for BuildKite CI

Per your docs, I am putting this request in to add support for the BuildKite CI system. Thanks!

Docs say you can use `codechecks.ts`, but settings are not read from it.

Howdy 👋

See

monorepo/packages/client/src/file-handling/settings.ts

Line 9 in ba52ad8

const CODECHECKS_SETTINGS_FILES_NAMES = ["codechecks.yml", "codechecks.yaml", "codechecks.json"];

for settings implementation and https://github.com/codechecks/docs/blob/master/configuration.md#codecheckstsjs.

I'm interested in PRing codechecks.ts if it's still planned and not supported yet.

Outsourcing reading config to cosmiconfig might be a fair approach.

After reading more of the code and implementing a bad hack I realize I might have misunderstood how codechecks.ts file is supposed to work — I assumed it returns a declarative config from main and you use it exactly the same way as codechecks.yml, but with autocomplete.

Project name does not update

I renamed a project in GitHub, but the codechecks overview page still shows the old name.

Make local-mode work offline

Currently, local-mode requires API connection and tries to get base artifacts for public repos. This is great but when project is not yet setup it fails when if fact it could just run in full offline mode.

Remove node-git dependency to simplify client requirements

Message is:

[nodegit] WARN - Could not finish postinstall
/usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /home/travis/.npm/_npx/9492/lib/node_modules/cgewecke-cc-testing/node_modules/nodegit/build/Release/nodegit.node)

It's a known node-git issue and can be resolved by adding the following to travis.yml:

addons:
  apt:
    sources:
      - ubuntu-toolchain-r-test
    packages:
      - libstdc++-4.9-dev

Bug: Expects "master" to be main branch

This doesn't seem to work with repos that use a different name for the main branch like "main".

➜  snoo-wrapped git:(main) ✗ npx codechecks
CodeChecks Client v.0.1.11
https://codechecks.io

Executing 1 codechecks files
Error: Command failed: /bin/sh -c git rev-parse master
fatal: ambiguous argument 'master': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

master

    at makeError (/Users/xo/code/OmgImAlexis/snoo-wrapped/node_modules/@codechecks/client/node_modules/execa/index.js:174:9)
    at /Users/xo/code/OmgImAlexis/snoo-wrapped/node_modules/@codechecks/client/node_modules/execa/index.js:278:16
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async LocalProvider.getShaForRef (/Users/xo/code/OmgImAlexis/snoo-wrapped/node_modules/@codechecks/client/dist/ci-providers/Local.js:60:17)
    at async LocalProvider.getPrInfo (/Users/xo/code/OmgImAlexis/snoo-wrapped/node_modules/@codechecks/client/dist/ci-providers/Local.js:55:22)
    at async Object.getConstExecutionContext (/Users/xo/code/OmgImAlexis/snoo-wrapped/node_modules/@codechecks/client/dist/getExecutionContext.js:41:18)
    at async main (/Users/xo/code/OmgImAlexis/snoo-wrapped/node_modules/@codechecks/client/dist/runner.js:37:32) {
  code: 128,
  stdout: 'master\n',
  stderr: "fatal: ambiguous argument 'master': unknown revision or path not in the working tree.\n" +
    "Use '--' to separate paths from revisions, like this:\n" +
    "'git <command> [<revision>...] -- [<file>...]'\n",
  failed: true,
  signal: null,
  cmd: '/bin/sh -c git rev-parse master',
  timedOut: false,
  killed: false
}

Support custom scopes

Currently, every artifact is scoped by commit SHA.

It should be possible to provide custom scopes like branch name or totally arbitrary key. This is required to inspect "history" of changes (for example needed for graphs).

Initial work done in: #64

Support Travis

I know this is on your roadmap - if you're open to volunteers working on the client I'd love to help.

Support patterns for speculative branch execution

There are scenarios where a branch is behind master (or its equivalent) and shouldn't be compared against it, or else codechecks is very likely to return false positives/regressions for things like build sizes, type coverage, etc.

Examples of types of branches affected include but are not limited to:

• Maintenance branches
• Hotfix branches
• Release branches

Currently if it's desired to support these branches they need to be manually listed in codechecks.yml under settings > branches and no patterns are supported for variable things like release branches.