rykus0 / nightwatch-visual-regression Goto Github PK

View Code? Open in Web Editor NEW

16.0 16.0 1.0 13 KB

Custom nightwatch assertion and commands for capturing, cropping, and comparing screenshots

License: MIT License

JavaScript 100.00%

nightwatch-visual-regression's People

Stargazers

Watchers

Forkers

drptbl

nightwatch-visual-regression's Issues

cropping screen is limited to visible portion of the page

Hi Rykus0,

Is there any way to take the whole page screenshot instead of cropping only to the visible portion of the page in the browser?

Thanks.

Sporadic error while saving error file

Receiving a system error that the file does not exist while writing the error file. Seems to be trying to open the file before it's created.

This is the same as the diff file, which hasn't had this problem.

To avoid this, ensure the file exists before writing.

Documentation - Extend Description

Include some info about NightwatchJS, and make any other improvements to the readme, including better installation instructions.

Generating error screenshots every time

Every run is generating error screenshots, and outputting the error message, even though the test actually passes.

Not scrolling into view before screenshot

Specifically for chrome, need to scroll the target element into view before taking the screenshot

Error while using browser.assert.visualRegression('.selector'); - allElements.shift is not a function

Hello,

Using newest Nightwatch.js:
browser.assert.visualRegression('.site-nav'); command throws an error, while browser.assert.visualRegression(); works properly.

Running:  Main menu tests should check that site navigation has header with proper styles and url
Error while running getLocation command: allElements.shift is not a function
    at Object.cropElement (/js/assertions/visualRegression.js:17:10)
    at Object.<anonymous> (/js/assertions/visualRegression.js:186:37)
    at /Users/jm/repos/kiln/entities-manager-e2e-tests/node_modules/nightwatch/lib/api/client-commands.js:106:22
    at /Users/jm/repos/kiln/entities-manager-e2e-tests/node_modules/nightwatch/lib/index.js:422:11
    at /Users/jm/repos/kiln/entities-manager-e2e-tests/node_modules/graceful-fs/graceful-fs.js:43:10
    at FSReqWrap.oncomplete (fs.js:82:15)
Error while running getLocation command: allElements.shift is not a function
    at Object.cropElement (/js/assertions/visualRegression.js:17:10)
    at Object.<anonymous> (/js/assertions/visualRegression.js:186:37)
    at /Users/jm/repos/kiln/entities-manager-e2e-tests/node_modules/nightwatch/lib/api/client-commands.js:106:22
    at /Users/jm/repos/kiln/entities-manager-e2e-tests/node_modules/nightwatch/lib/index.js:422:11
    at /Users/jm/repos/kiln/entities-manager-e2e-tests/node_modules/graceful-fs/graceful-fs.js:43:10
    at FSReqWrap.oncomplete (fs.js:82:15)

Problem is exactly there:

    this.getLocation(selector, function(results){
        var origin = results.value;

getLocation throws an error.

Any suggestions? Looks like an issue with cropping or something related to this, because it occurs only when I want to screenshot chosen element.

Thanks in advance,
drptbl.

Error screenshot is empty

Error screenshot is an empty file. Probably because pack() is being called again.

`expected "0.1" but got: null` while screen differs.

Hey @Rykus0,

Getting this error while screen differs from baseline:

✖ Visual Regression: Screen differs by 0.41% (see: visualregression/screenshots/failures/homePage/mainMenuSpec/chrome_51.0.2704.103_MAC/1680x1050/Check that menu exists on home page__body.diff.png)  - expected "0.1" but got: null
    at Object.module.exports.Check that menu exists on home page (/tests/homePage/mainMenuSpec.js:56:22)
    at Module.call (/tests/node_modules/nightwatch/lib/runner/module.js:63:34)
    at /tests/node_modules/nightwatch/lib/runner/testcase.js:97:29
    at _fulfilled (/tests/node_modules/q/q.js:834:54)
    at self.promiseDispatch.done (/tests/node_modules/q/q.js:863:30)
    at Promise.promise.promiseDispatch (/tests/node_modules/q/q.js:796:13)
    at /tests/node_modules/q/q.js:556:49
    at runSingle (/tests/node_modules/q/q.js:137:13)
    at flush (/tests/node_modules/q/q.js:125:13)
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickCallback (internal/process/next_tick.js:98:9)

expected "0.1" but got: null
instead of null, there should be a differ percentage, right?
so properly it should look like:
expected "0.1" but got: 0.41
in this case.

Tested on MAC OS, Nightwatch.js v0.9.5.

Cheers!

Baseline getting created every time

Due to a misunderstanding of fs-extra's ensureFile method, I am overwriting the baseline screenshot each time, meaning tests will never fail. Fix is coming momentarily.

visualRegression() assertion is ran async(?) instead of sync.

Hey @Rykus0, thanks for your great work last days.

There is one more issues which is the most "disturbing" one - I would say.

It looks like visualRegression() assertion is ran async. It's weird because all nightwatch.js assertions are ran sync. Is there any way to fix this?

For example:

[Home Page / Main Menu Spec] Test Suite
===========================================

Running:  Main menu tests should check that menu exists on home page
ⓘ  Finished setting up everything before testing..
ⓘ  Finished visual regression checks!
No assertions ran.

Running:  another one
 ✖ Visual Regression: Screen differs by 4.19% (see: /diff/Main menu tests should check that menu exists on home page__chrome__v51.0.2704.103__MAC__1680x1050__body.diff.png)  - expected "0.1" but got: "null"
    at Object.module.exports.Main menu tests should check that menu exists on home page (/tests/homePage/mainMenuSpec.js:59:22)
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickCallback (internal/process/next_tick.js:98:9)

ⓘ  Finished visual regression checks!

FAILED:  1 assertions failed (2.669s)
 ✖ Visual Regression: Screen differs by 12.55% (see: /diff/another one__chrome__v51.0.2704.103__MAC__1680x1050__body.diff.png)  - expected "0.1" but got: "null"
    at Object.module.exports.another one (/tests/homePage/mainMenuSpec.js:92:24)
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickCallback (internal/process/next_tick.js:98:9)

ⓘ  Closing down everything after testing..

 _________________________________________________

 TEST FAILURE:  1 assertions failed, 0 passed. (12.124s)

 ✖ homePage/mainMenuSpec

   - another one (2.669s)
   Visual Regression: Screen differs by 4.19% (see: /diff/Main menu tests should check that menu exists on home page__chrome__v51.0.2704.103__MAC__1680x1050__body.diff.png)  - expected "0.1" but got: "null"
       at Object.module.exports.Main menu tests should check that menu exists on home page (/tests/homePage/mainMenuSpec.js:59:22)
       at _combinedTickCallback (internal/process/next_tick.js:67:7)
       at process._tickCallback (internal/process/next_tick.js:98:9)

In example above:
each Finished visual regression checks! log is done after visualRegression(), so it's kind of "weird" that there is no assertion after first log, it's done while running second test case (so it's not sync for sure). Also it says "No assertions ran." after first test case, while visualRegression() was ran there (still issue with async I believe). And summary prints only 1 failure while there was 2 failures actually.

I think that most of this bugs are related to this issues with async issues in visualRegression(); assertion.

Thanks for great work, Cheers!

Enable the "enabled" flag

There is a flag in the configuration to allow disabling the assertion from a single point - useful if you need to run tests without visual comparison for any reason and don't want to rewrite all of your tests!

Currently this flag is not used. Opening the issue for the addition of this feature.

Provide a shortcut to crop to viewport

Not exactly sure what the syntax would be, but I was just thinking it could be useful to crop to the viewport.

rykus0 / nightwatch-visual-regression Goto Github PK

nightwatch-visual-regression's People

Stargazers

Watchers

Forkers

nightwatch-visual-regression's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs