percy / percy-client Goto Github PK

I have a PR at ampproject/amphtml#11827, for which Travis CI ran Percy visual diff tests that passed. However, Percy <-> github integration is showing a red cross mark against the PR that links to an older failed build even though the latest Travis test run's Percy build happened after the failed build.

Screenshot from PR:

This links to Percy build # 6421

However, the latest Travis build from the PR generated Percy build # 6423, which was successful.

We enforce Percy checks in our repository, so this issue is blocking some of our PRs from being merged.

@timhaines @fotinakis is this a known problem? Or could I be missing something?

I have added Percy as an integration on Github where I am an owner.
On Github, I have allowed all requested privileges and the integration looks complete on Github.

On Percy, the screen hangs with the following message:
GitHub App processing...
Waiting for GitHub to complete the installation.

This is a feature request stemming from the discussion in ampproject/amphtml#11385, and from 1:1 conversations with @fotinakis.

The ampproject/amphtml project uses forked repos for development, due to which github's API has no way of knowing a PR's branch point off of the mainline master branch. This is messing with Percy's automatic baseline picking logic, and is proving to be problematic in a bunch of ways.

If we have a way to provide Percy with a branch point SHA which can be used during baseline picking, we'd be able to mitigate all related problems.

/to @fotinakis @timhaines

Need to migrate calls and variables away from the legacy /repos API endpoint to the modern /projects endpoint.

Percy-client should wrap Faraday::TimeoutError into own base error class so clients can rescue all errors coming from percy-client.

A relatively new way to build git projects is with the Pipeline Plugin, which includes the Multi-Branch Plugin. These set different environmental variables than the GitHub PR Builder Plugin, which Percy natively supports.

This causes problems like all builds being listed as detached heads instead of the actual branches, unless the optional variables are set manually.

Here are the variables I see that expect the PR Builder Plugin, and what the Pipeline Plugin provides:

• ghprbPullId -> CHANGE_ID
• ghprbTargetBranch -> BRANCH_NAME
• ghprbPullId -> CHANGE_ID

We're seeing this on ampproject/amphtml, which uses forked branches for development. If a developer inadvertently pushes a PR from their local master branch, and if that build contains visual diffs, those snapshots are used as the baseline for all subsequent Percy builds until a new commit is merged to the real master.

/cc @fotinakis @timhaines

In some odd CI builds I get:

An error occurred in a `before(:suite)` hook.
--
  | Failure/Error: build = Percy::Capybara.initialize_build
  |  
  | ArgumentError:
  | invalid byte sequence in US-ASCII
  | # ./vendor/bundle/ruby/2.4.0/gems/percy-client-1.11.0/lib/percy/client/environment.rb:39:in `match'
  | # ./vendor/bundle/ruby/2.4.0/gems/percy-client-1.11.0/lib/percy/client/environment.rb:39:in `match'
  | # ./vendor/bundle/ruby/2.4.0/gems/percy-client-1.11.0/lib/percy/client/environment.rb:39:in `block in commit'
  | # ./vendor/bundle/ruby/2.4.0/gems/percy-client-1.11.0/lib/percy/client/environment.rb:47:in `commit'
  | # ./vendor/bundle/ruby/2.4.0/gems/percy-client-1.11.0/lib/percy/client/builds.rb:7:in `create_build'
  | # ./vendor/bundle/ruby/2.4.0/gems/percy-capybara-2.5.1/lib/percy/capybara/client/builds.rb:19:in `block in initialize_build'
  | # ./vendor/bundle/ruby/2.4.0/gems/percy-capybara-2.5.1/lib/percy/capybara/client.rb:68:in `rescue_connection_failures'
  | # ./vendor/bundle/ruby/2.4.0/gems/percy-capybara-2.5.1/lib/percy/capybara/client/builds.rb:18:in `initialize_build'
  | # ./vendor/bundle/ruby/2.4.0/gems/percy-capybara-2.5.1/lib/percy/capybara.rb:29:in `initialize_build'
  | # ./spec/spec_helper.rb:36:in `block (2 levels) in <top (required)>'
  |  
  | An error occurred in an `after(:suite)` hook.
  | Failure/Error: $build_url = build['data']['attributes']['web-url']
  |  
  | NoMethodError:
  | undefined method `[]' for nil:NilClass
  | # ./spec/spec_helper.rb:40:in `block (2 levels) in <top (required)>'

Not sure, yet, when or why exactly this happens.

What would you suggest is the best option name?
What about this? https://gist.github.com/christophemaximin/81f7c2d8f4bc5263a626c3f713925b8a