Multiple Scenarios and Browserstack about serenity-js

[ttretau]

Hey @jan-molak,
Ok, I updated to latest dependencies (like the tutorial-project). Now I got
[09:18:38] E/launcher - BUG: launcher exited with 1 tasks remaining
I will try to build a minimal example project, reproducing that..

[ttretau]

Cool, now it works! Great work!

[ttretau]

Ah, think I fixed this by changing protractor.conf, like here:
https://github.com/serenity-js/tutorial-from-scripts-to-serenity/blob/master/protractor.conf.js#L9
I had 'serenity-js' before there..

[jan-molak]

Hey @ttretau, Serenity/JS itself is tested with BrowserStack, so yes, those two should work together.

Could you please provide a minimal test case demonstrating the issue you're seeing?

Thanks,
Jan

[ttretau]

I created this Example:
more or less running, with not so well session behaviour
After updating dependencies it stopped after first scenario with
[11:58:44] E/launcher - BUG: launcher exited with 1 tasks remaining

[jan-molak]

Thanks @ttretau, I managed to reproduce the issue.

The [11:58:44] E/launcher - BUG: launcher exited with 1 tasks remaining is caused by the protractor-cucumber-framework, which both the examples are using instead of serenity-js.

Having said that, when the example is changed to use serenity-js, the real issue is triggered: function timed out after 10000 milliseconds.
It seems like the 10 seconds allowed for Serenity/JS cleanup tasks, which include pulling the screenshots down from BrowserStack, is not long enough. The default timeout should probably get extended and ideally made configurable so that you can tweak it yourself.

Thanks for reporting this problem, a fix is on the way.

Jan

[ttretau]

Hey @jan-molak, great news! Thanks for the prompt answer and the hint with wrong configuration. Let me know if I can help you fixing/testing the bug..
Torben

[jan-molak]

Hey @ttretau, upgrading to Serenity/JS 1.2.4 makes the example pass. I'm looking into a nicer way of managing timeouts now.

[ttretau]

Hi @jan-molak,
I tried your new config with the newest serenity-js version (1.3.0).
In practice and with this simple example it shows some problems handling multiple scenarios and their according sessions in browserstack.
The first scenario is handled fine with the corresponding successful browserstack session. After that it starts another short lived session, doing "nothing" except setting a timeout.
The next scenario does not finish and is running in a timeout. After that another "empty" browserstack session is started.

Looks like it also happening running locally with this example. leaving open windows until timeout..
Let me know if/how I can support you in finding/fixing this issue.
Kind regards,
Torben

[jan-molak]

Hey @ttretau!

I cloned the example repository and run it against browserstack with the following result:

[22:59:10] I/hosted - Using the selenium server at http://hub-cloud.browserstack.com/wd/hub
Feature: Add new items to the todo list

    In order to avoid having to remember things that need doing
    As a forgetful person
    I want to be able to record what I need to do in a place where I won't forget about them

  Scenario: Adding an item to a list with other items
  ✔ Given that James has a todo list containing Buy some cookies, Walk the dog
  ✔ When he adds Buy some cereal to his list
  ✔ And he adds Buy some cereal to his list
  ✔ And he adds Buy some cereal to his list
  ✔ Then his todo list should contain Buy some cookies, Walk the dog, Buy some cereal, Buy some cereal, Buy some cereal

  Scenario: Adding an item to a list with other items with Umlauts
  ✔ Given that Klaus Jäger has a todo list containing Kaufe zwei Mehlsäcke, geh das Auto waschen
  ✔ When he adds Kaufe vier Bananen to his list
  ✔ And he adds Kaufe fünf Gurken to his list
  ✔ And he adds Kaufe fünf Gurken to his list
  ✔ Then his todo list should contain Kaufe zwei Mehlsäcke, geh das Auto waschen, Kaufe vier Bananen, Kaufe fünf Gurken, Kaufe fünf Gurken

2 scenarios (2 passed)
10 steps (10 passed)
0m37.711s
[23:00:27] I/launcher - 0 instance(s) of WebDriver still running
[23:00:27] I/launcher - Chrome #01 passed

What exact parameters are you running the tests with, please?
I tried:

npm start --BROWSERSTACK_USERNAME=XXXX --BROWSERSTACK_ACCESS_KEY=XXXX

J

[ttretau]

Hi,
I am running the tests exactly this way. My result

[08:26:08] I/launcher - Running 1 instances of WebDriver
[08:26:08] I/hosted - Using the selenium server at http://hub-cloud.browserstack.com/wd/hub
Feature: Add new items to the todo list
[...] 
2 scenarios (2 passed)
10 steps (10 passed)
0m54.357s
[..here it stays for 4 minutes..]
[08:30:59] I/launcher - 0 instance(s) of WebDriver still running
[08:30:59] I/launcher - Chrome #01 passed

Same as yours, except 4 minutes pause until the WebDriver output..
Leading to this Browserstack Sessions:

Duration: 4 mins 40 secs
Session	OS	Browser / Device	Duration	Finished
 single_test	 Win 10	 Chrome 57.0	21s	a min ago
 single_test	 Win 10	 Chrome 57.0	1m 41s	1m ago
 single_test	 Win 10	 Chrome 57.0	1m 56s	2m ago
 single_test	 Win 10	 Chrome 57.0	10s	4m ago
 single_test	 Win 10	 Chrome 57.0	29s	4m ago

As it works fine on your side I think I have to look for a Browserstack or Network problem..

[ttretau]

Ah, perhaps queuing on Browserstack - we have only one parallel tests available - so although maxInstances and count are set to one on protractor side something is put into a queue on Browserstack-side.. Do you have more than one parallel tests available in your Browserstack-Account?

[jan-molak]

Hey @ttretau, it does sound like queueing on BrowserStack; I have 2 parallel tests available, so running with two browsers works fine. I suppose it's sensible to set maxInstances to a number of browsers the server can actually spawn.

Does that answer your question?

[ttretau]

Hi @jan-molak, i debugged a little into browser starts and it looks wired to me - this is my debug output:

BLUB enter the stage [object Object]
Feature: Add new items to the todo list

    In order to avoid having to remember things that need doing
    As a forgetful person
    I want to be able to record what I need to do in a place where I won't forget about them

  Scenario: Adding an item to a list with other items
BLUB browse the web http://todomvc.com
BLUB browse the web http://todomvc.com
  ✔ Given that James has a todo list containing Buy some cookies, Walk the dog
  ✔ When he adds Buy some cereal to his list
  ✔ And he adds Buy some cereal to his list
  ✔ And he adds Buy some cereal to his list
  ✔ Then his todo list should contain Buy some cookies, Walk the dog, Buy some cereal, Buy some cereal, Buy some cereal

  Scenario: Adding an item to a list with other items with Umlauts
BLUB browse the web http://todomvc.com
BLUB browse the web http://todomvc.com
  ✔ Given that Klaus Jäger has a todo list containing Kaufe zwei Mehlsäcke, geh das Auto waschen

The "browse the web debug" output is in BrowseTheWeb.using(), instatiating the BrowseTheWeb Ability. Does it mean that it is starting too much browser-windows for a scenario?

[jan-molak]

Interesting;

The output seems weird; BrowseTheWeb is not itself responsible for starting the browser, it receives an instantiated browser object from Protractor, which is also responsible for sharding the tests and running them in parallel...

Can you push your debug version to github so that I can try to reproduce?

[ttretau]

Ah, I now saw, that if I run this sample locally it flickers after the actual tested browser window opens, so I think there is also more than one window opened if you run it without Browserstack.
My debug version contains only the console.log in BrowseTheWeb.using() to show that how often it is called..
I am looking into how protractor browser is instantiated..

[jan-molak]

Cool, please keep me posted!

[ttretau]

Hey @jan-molak ,
today I found out that this could depend on the "default" crew configuration in serenity. If I comment out these lines it works. If they are in, after the actual session another 3 browser sessions are started.. I am trying to get into how that should work..

[jan-molak]

Hmm, the ProtractorReporter and ProtractorNotifier use the instance of Protractor's Runner. Is it one of those objects that cause trouble? I don't think the StandInd should affect anything as they don't interact with the Runner ...

Could there be some additional Runner cleanup task that we're missing there? 🤔

[ttretau]

That's right. StandIns only leads to 'stand-in' actor entering the stage.
So the Runner instances look hot. The Notifier/Reporter are spawning new sessions?

[jan-molak]

No, they shouldn't. The idea here is that Protractor does all the heavy lifting and session management and Serenity/JS provides an abstraction layer on top of that.

Notifier invokes afterEach to let Protractor know when the test is finished (as per the requirements).

Reporter notifies protractor of test passes/failures and the final result (protractor spec).

[ttretau]

Ah, thanks. That explanation brings some light.. So I think getting deeper into protractor debugging makes sense..

[ttretau]

Nothing new.. :-( Did not found out anything regarding this problem.. Looks like protractor integration is working fine.. After scenario is finished a short browser session is popping up. Could that also be a problem with Cucumber?

[jan-molak]

Hmm, and what if you don't instruct Protractor to restart the browser between the tests?
Maybe the short session is caused by the restart? 🤔

[ttretau]

Ah, you nailed it! 👍 That is a workaround for to the browserstack problem..
Could this be related to this protractor problem? angular/protractor#2001

[jan-molak]

@ttretau it looks like the problem is not related to BrowserStack, but more to how Protractor handles browser restarts. I've created a new ticket so we can have a more focused conversation there. #56