heroku / heroku-ci

this code is now in https://github.com/heroku/cli

Heroku CI CLI

The CLI plugin for Heroku CI

Installation

This plugin is part of the official Heroku CLI, so there is no need to install it to run their commands.

Development

First, please read Developing CLI Plugins on Heroku's DevCenter.

Run Tests

$ npm test

Deploy

1. Release a new version of this npm package. e.g. Using np

› np

Publish a new version of heroku-ci (1.6.0)

? Select semver increment or specify new version minor 	1.7.0
? Will bump from 1.6.0 to 1.7.0. Continue? Yes
 ✔ Prerequisite check
 ✔ Git
 ✔ Cleanup
 ✔ Installing dependencies using Yarn
 ✔ Running tests
 ✔ Bumping version using Yarn
 ✔ Publishing package
 ✔ Pushing tags

 heroku-ci 1.7.0 published 🎉

2. Open a new pr in https://github.com/heroku/cli/blob/master/package.json, updating to the appropriate heroku-ci version.

Commands

All commands accept a -a flag. This should be the name of an app in the pipeline you have connected to Heroku CI. The -a flag can be omitted if you are in the directory of an app already.

heroku ci:list, aliased as heroku ci

Show the statuses of the most recent test runs. Add the -w or --watch flag to keep the session running and listen for updates.

heroku ci:info NUMBER

Show the status of a specific test run. If no run number is provided the most recent will be shown.

heroku ci:last

Show the status of the most recent test run

heroku ci:run

Run this from within your repo directory to trigger a test agains the current branch and commit.

heroku ci:rerun NUMBER

Re-run a previous test run. If no test run number is provided, the most recent test run will be re-run.

heroku ci:migrate-manifest

Migrate the current directory from the deprecated app-ci.json file format to the new app.json + environments. See the docs for more details.

heroku ci:debug

Run this from within your repo directory to start and attach to a debug run. Useful for debugging failing test setup or a test which fails in CI but passes locally.

heroku-ci's Issues

Improve CLI error message "Not found."

When running a CI command if there is a problem this is the only error output:

$ heroku ci:debug
 ▸    Not found.

You can get this error if the app you're running ci:debug on does not have a pipeline attached to it. Also if that pipeline is not linked to a github account.

We should explicitly account for both of those cases and give better error messages

Proposal

$ heroku ci:debug
 ▸    Error: This application `issuetriage` does not have a connected pipeline
      To add a pipeline run `$ heroku pipelines:create issuetriage`, for more 
      info see https://devcenter.heroku.com/link-to-something-explaining-why-pipelines-are-needed

and

$ heroku ci:debug
 ▸    Error: This pipeline for app `issuetriage` is not connected to a GitHub Repo
      To connect this application to a GitHub Repo and enable CI please visit
      https://dashboard.heroku.com/pipelines/374207ae-37c1-4907-bfd0-d6eef4c46a02
      And select "connect to GitHub" button.

      For more info on why this is needed please 
      info see https://devcenter.heroku.com/link-to-something-explaining-why-github-is-needed

As a side note, do we have a CLI command that allows you to connect to GitHub? That would be really nice.

Feature request: open dashboard to test screen from CLI

Something like:

heroku ci:open <<optional run number>>

and that opens to the pipelines test page.

Error running watch

$h --version
heroku-cli/5.6.26-e6aeac4 (darwin-amd64) go1.7.5

$h plugins
heroku-ci 1.3.0

$h ci --watch -r staging
=== Watching latest test runs for the blog pipeline
 ▸    cli.color[STATUS_COLORS[status]] is not a function

If it helps, I cancelled a build and restarted it before this happened.

Error during install: cannot find module 'bluebird'

I'm running into the following error when trying to install that plugin:

$ heroku plugins:install heroku-ci
Installing plugin heroku-ci...module.js:442
    throw err;
    ^

Error: Cannot find module 'bluebird'
    at Function.Module._resolveFilename (module.js:440:15)
    at Function.Module._load (module.js:388:25)
    at Module.require (module.js:468:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/Users/steve/.local/share/heroku/plugins/node_modules/heroku-ci/lib/git.js:2:17)
    at Module._compile (module.js:541:32)
    at Object.Module._extensions..js (module.js:550:10)
    at Module.load (module.js:458:32)
    at tryModuleLoad (module.js:417:12)
    at Function.Module._load (module.js:409:3)
 !
 ▸    Error installing plugin heroku-ci
 !    error installing plugin heroku-ci

Any idea of what I'm doing wrong here?

Be able to run a test from a different branch without doing checkout

Currently, we're only running tests in a current branch. i.e. If you wanted to run a test in a branch different than master, we require doing git checkout before.

It'd be nice if we could do the same without requiring that step.

e.g.:

Now:

$ git checkout branch-name
$ heroku ci:test

Proposal:

$ heroku ci:test --branch branch-name

Limit ci:list to a specific branch

We recently switched to Heroku CI, and we've been using successful CI build output on the master as a prerequisite for a production deploy. We can use heroku ci or heroku ci:list to get a list of recent build results, and grep that list for master and check whether the latest is successful, but that only works if master happens to be in the last 15 test runs. If we've had a lot of development activity without a recent merge, it's entirely possible that we won't have a master build result in the list at all.

It'd be ideal if we could supply a branch name to the command, perhaps --branch master, and retrieve the latest results for only that branch. That way, we can be sure to have something valid to check before deploying.

require disambiguatePipeline returns undefined

heroku-ci/lib/utils.js

Line 3 in 0713e43

const disambiguatePipeline = require('@heroku-cli/plugin-pipelines').disambiguatePipeline

heroku ci not available when install via MacOS installer

I installed heroku cli via the MacOS installer, and heroku ci is not available.

Tried totally uninstalling and re-installing, but still not available.

Issues with CI

I'm not sure where to report that;

Anyone from CI team, please take a look into codeclimate/ruby-test-reporter#183 respond and close this issue.

thanks!

Enable Heroku CI through the Heroku API

Currently, there is no way to enable Heroku CI on a pipeline via the Heroku API.

Heroku's GUI sits ontop of the Heroku API(s), which means there must be an API beheind the scenes making the call (just a private one not documented or exposed).

Is there any chance we could please expose an API endpoint to enable Heroku CI on the pipeline? I currently see no docuentation on your public API.

Command to toggle Heroku CI

We support creating test runs, listing test runs, debugging test runs, but that is all predicated by the feature being turned on. This plugin would be additionally useful with something like:

heroku ci:enable --pipeline my-pipeline

Sudo support with `ci:debug`

Currently if you try to sudo into a ci:debug instance, it will not sudo you onto a pipeline as well:

⛄  2.4.1 🚀  ~/documents/projects/tmp/drawbotics-staging (support)
$ heroku sudo ci:debug
 ▸    You do not have access to this pipeline.

Then if you do sudo yourself into the app, it can still fail

$ heroku sudo access:add [email protected] -a drawbotics-staging
Adding [email protected] access to the app drawbotics-staging... done
$ heroku sudo ci:debug
Preparing source... done
Creating test run... done
 ▸    Test run creation failed while undefined with message "Unable to create test app: your role collab on the team drawbotics-dev is not allowed to perform that action."

If you try to do this manually you can run into the issue of hitting limits:

timestamps with ci runs

=== Showing latest test runs for the testing pipeline
!  828  i2523       886a6f3  errored
!  827  testing-new     93c3e03  errored

It'd be handy if that logged the timestamp that the job ran at...

Watch doesn't update with new runs?

v 1.3.1, macOs

heroku ci:list -w

I would expect that to show a new run when it gets created via automatic test runs. It does not update with the new test run.

Heroku Generated Env vars not present on `ci:debug`

Test runs on ci:debug do not get the same environment variables set as test runs from a pipeline. This means if you have code that depends on one of these env vars you will get different behavior in test and in ci:debug

• HEROKU_TEST_RUN_BRANCH
• HEROKU_TEST_RUN_ID
• HEROKU_ENV

We should either include these in ci:debug or add an extra environment variable such as HEROKU_IS_RUNNING_CI and document it.

$HEROKU_TEST_RUN_BRANCH not accurate when running `ci:debug`

It appears that if I run heroku ci:debug from a non-master branch I will be dumped into a dyno with my master branch, but the $HEROKU_TEST_RUN_BRANCH will tell me that I am on the local branch.

Link to slack discussion: https://heroku.slack.com/archives/C1P0609FC/p1498764480135177

I expect HEROKU_TEST_RUN_BRANCH to be the branch that I see when I run heroku ci:debug, but that is not the branch that I see.
Let me try to make it clearer: try-npm5 does not have a yarn.lock file, instead it has a package-lock.json

❯ ls
Procfile          app.json          config.js         flow-typed        node_modules      package.json
__tests__         bin               dev               lib               package-lock.json readme.md

But when I run debug I get what looks like my master branch: yarn.lock exists and package-lock.json is nowhere to be found:

❯ heroku ci:debug --pipeline nodebin
[ lots of output]
~ $ git status
fatal: Not a git repository (or any parent up to mount point /app)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
~ $ ls
Procfile  __tests__  app.json  bin  config.js  dev  flow-typed  lib  node_modules  package.json  readme.md  yarn.lock
~ $ eval $(ci env)
~ $ env
[ elided ]
HEROKU_TEST_RUN_BRANCH=try-npm5```

Link to code picking up the local branch in the CLI: https://github.com/heroku/heroku-ci/blob/master/commands/ci/debug.js#L50

Reporting to external coverage tools requires additional git info

All of the tools I've checked (Codacy, Code Climate, and Codecov) require at least the SHA for the commit under test, in addition to the branch name (which we have access to as an environment variables). Code Climate at least also requires a value for "committed at", which is the result of git log -1 --pretty=format:'%ct'.

Any possibility of getting these added into the environment of a CI run?

Better debug of App.json parsing

Got this error

-----> Reading manifest...
-----> Parsing manifest failed with `json: cannot unmarshal string into Go value of type sprettur.AppManifestBuildpack`...

When I had an invalid schema.

I wrote this:

  "environments": {
    "test": {
      "buildpacks": ["https://github.com/heroku/heroku-buildpack-ruby#schneems/debug-bad-ruby"]
    }
  }

When I needed this:

  "environments": {
    "test": {
      "buildpacks": [
       {"url": "https://github.com/heroku/heroku-buildpack-ruby#schneems/debug-bad-ruby"}
      ]
    }
  }

Would be nice to lint as well as expose what part exactly doesn't conform to the valid schema.

Key all commands on pipeline

All commands currently only accept an app as the base, from which we look up the pipeline. This is a little odd since CI belongs to a pipeline.

We should allow all commands to accept a -p flag to specify the pipeline directly, but continue to support app lookup.

Cannot read property 'setup_stream_url' of undefined

MacOS 10.12.4
heroku-ci v 1.5.1

When running h ci:run -r production I get the following output

$ h ci:run -r production
Preparing source... done
Starting test run... done
 ▸    Cannot read property 'setup_stream_url' of undefined

I'm guessing that's coming from test-run.js. I'm happy to share more details in a non-public forum

`heroku ci:list` should be `heroku ci`

the default flag is deprecated in v6. Using a :list command is not encouraged in the style guide in favor of using the topic name only. We will be adding better support for aliases in v6 but for now there is a more manual way to fix this.

see heroku/heroku-pipelines#55 for a similar change

ci:debug builds

If a part of the build fails before it is ready to run the tests then ci:debug fails. Could we add a flag to drop me into the environment right when where it failed maybe heroku ci:debug --allow-failure. Even better if we could set this via a local environment variable CI_DEBUG_ALLOW_FAILURE=true or something.

I'm trying to debug a customer's app install failing on CI and it takes a good 5 minutes for each iteration.

Do not honor `.slugignore`

Had a customer who had their entire test directory ignored by .slugignore and the directory was silently deleting it. I imagine more people have done this.

At bare minimum we should be outputting something like what we see in build

-----> Deleting 2 files matching .slugignore patterns.

I think though that since these slugs are not intended to live as long (and boot up time is slightly important) that we ignore the .slugignore.

If we decide that it's not acceptable to ignore .slugignore than we'll need to create a new paradigm for ignoring files in production, but not in test. Something like .slugignore-except-in-test.