ckeditor / ckeditor4-benderjs-runner Goto Github PK

As mentioned in README:

I have tested it one day and it seems to be doable AFAIR. The only issue could be with IE8 which displays alert about long executing script for bender and I'm not sure if it can be somehow disabled.

It should be doable but it's not really a small task. OTOH launchpad used here for launching browsers already have some API for using BrowserStack so it should work in a similar manner as launching local browsers. Additionally, tunneling will be needed here so BS can access locally launched bender instance.

See https://github.com/microsoft/playwright.

I noticed when running locally, that browser instances are not correctly closed sometimes. Since we would like to run this on CI, it shouldn't matter much, but could be improved.

While generating the difference between branch and target branch, the travis task fails.

More details:

• It was between t/4866 and major branch.
• Chrome and Firefox jobs are failed.
• https://app.travis-ci.com/github/ckeditor/ckeditor4/jobs/536589721

Frontend runner uses fetch API to send tests results to backend server. This means it will not work on any IE browser. We could go with feature detection here and use other method if fetch is not available.

At the moment, logging is here and gives enough information. However, it could look slightly better and could use some colors and better formatting.

It would be nice to have Edge support. It can be done via Windows image (see #13) or trying to run Edge on Linux image (should be doable too).

If not going with BS straightaway (#4) should also solve this issue.

As mentioned in README:

There are Linux, macOS and Windows images available so I assume at least Chrome, Firefox, Edge, Safari and IE11 can be covered here. For IE8-IE10 and mobiles (so Chrome@Android and Safari@iOS) we will need integration with BrowserStack (or similar service).

Since GH Actions allows matrix runs it will be easier to define - I'm not only sure if such steps run simultaneously (that would be faster) or one after another.

We can try with forked ckeditor/ckeditor4 repo. I guess the workflow could be triggered on commit and pullrequest events in ckeditor4 repo, checkout ckeditor4-benderjs-runner and try to run tests.

We can use either Windows machine or BrowserStack here. Tickets which may need to be fixed along the way:

• #6
• #13

The code works but could be slightly improved to be more readable and us Bender API whenever possible. Also it should work on older IEs so no ES6 here unfortunately.

Related issues: #6.

Right now - changes in package.json produce an empty test query.

Due to the nature of such changes - it should be rather a full run. I know that our deps are only the 'dev' one - but still - passed full run indicates that nothing breaks. Also, invalid changes should be visible at least at installing deps level.

There are some runs where browser is launched and then nothing happens, for example:

• https://travis-ci.org/github/ckeditor/ckeditor4-benderjs-runner-tests/jobs/750377536#L204
• https://travis-ci.org/github/ckeditor/ckeditor4-benderjs-runner-tests/jobs/750377535#L321

I'm not sure why it happens. I was not able to reproduce it locally so far too. I suspect some issue with frontened runner and browser focus maybe? But the cause may be totally different ofc 🤔

There is some issue with JSON parsing. See https://travis-ci.org/github/ckeditor/ckeditor4-benderjs-runner-tests/builds/750367960#L251.

Travis (or rather GitHub itself 🤔) supports [skip ci] flag which skips CI runs when added to specific commit for this commit. We could have something similar (for commit title/message) which allows:

• Skipping all tests, e.g. [tests:skip]
• Running full test run e.g. [tests:all]
• Running specific tests, e.g. [tests:testsPath] ([tests:is:unit,path:/tests/plugins/image2])

This can help us run more/less tests if we see there is some mismatch in tests diff thing (see #1). Or do a full run if we see that changes introduced are really broad.

Related to #1.

As mentioned in README:

It can greatly decrease CI run times (as we don't want to run all tests each time). For PRs it should be easy since GH provides a way to get PR target branch (so diffing all changes is easy), for branches (not yet PRs) we could compare against main branches (so master and major - should be defined in config) and assume that target branch is the one which has less changes against. For commits pushed directly to main branches (so master and major - again should be configurable) we could only get changes from latest commit (which will basically cover all PR changes for merge commits) and also do full runs only once in a while (e.g. on a daily basis).

So the general idea is to diff with target branch, see all JS file which were changed and then decide which tests should to be run,  few examples:

Changes in tests/plugins/autogrow/autogrow.js

If something is changed in test file we may simply run only this file or all test for a given plugin (because it might be some helper which have changed).

Changes in plugins/widget/plugin.js

If plugin code is changed we should run all tests for this plugin and all plugins which depends on it (in this case embed, image2,...)

Changes in some core files

Not 100% sure here, but it will be good to run most/all core tests and tests for most crucial plugins (and it's dependants) like widget, dialog, clipboard, etc

Changes in CHANGES.md file

No tests should be run.

We had a PR from community ckeditor/ckeditor4#4837 which was set up on the same branch name as the target branch but from the fork.

It ends with an empty test query, so we have no test run: https://app.travis-ci.com/github/ckeditor/ckeditor4/jobs/532205765

We should be able to detect differences in another origin.

The code works but could be slightly improved to be more readable and split better into more reasonable parts. There are also some "not really nice" quick hacks here and there which could be ironed too.

At the moment it gest stuck on launching bender NPM subtask - https://travis-ci.org/github/ckeditor/ckeditor4-benderjs-runner-tests/jobs/750407154#L142.

I did some adjustments (which fixed error which was thrown), but it still doesn't run correctly. Maybe there is an issue with paths 🤔

It would be nice to have Safari support. When we worked in December on the runner it worked with Travis CI osx image just fine (apart from #10) - see for example ckeditor/ckeditor4-benderjs-runner-tests@fb69346 and https://travis-ci.org/github/ckeditor/ckeditor4-benderjs-runner-tests/builds/750370942.

After getting back to this in May, Travis was not able to trigger any job containing osx image config. The job simply didn't run and when triggering manual Travis CI was giving were vague error (see related commits in ckeditor/ckeditor4-benderjs-runner-tests#3).

We can try with osx image once again and dig deeper or go with BS straightaway (#4) which should also solve this issue.