yahooarchive / arrow Goto Github PK

It would be good to not be tied to the default wait timeout for an html element being present on a web page, because often the timeout is too short for larger data-driven requests on test systems where resources to make things go faster are constrained. The end result is false fails.

To set the implicit wait under the Selenium webdriverjs, the static 'webdriver' object used in test creation is needed (I tried importing it as static code, but '.schedule' is undefined in that case, causing an error).

At least for those drivers that have a significant static component (like Selenium's webdriverjs), I suggest passing it in after the driver parameter below, or finding some other suitable way so that developers are not stuck with the default timeout. Passing in the static webdriver object will also allow access to many other features.

From Arrow's controller/default.js :

Controller.call(this, testConfig, testParams, driver);

From Selenium's selenium/javascript/webdriver/webdriver.js :

webdriver.WebDriver.Timeouts.prototype.implicitlyWait = function(ms) {
  return this.driver_.schedule(
      new webdriver.Command(webdriver.CommandName.IMPLICITLY_WAIT).
          setParameter('ms', ms < 0 ? 0 : ms),
      'WebDriver.manage().timeouts().implicitlyWait(' + ms + ')');
};

See: http://selenium.googlecode.com/git/docs/api/javascript/class_webdriver_WebDriver_Timeouts.html

Could you please remove them and/or add a lines to your .gitignore so they do
not get copied by the repo.?

The best I can come up with so far is barely acceptable:

cp /Users/test/arrow-foo/foo-page-load/test-foo-load-controller.js /usr/local/lib/node_modules/yahoo-arrow/lib/controller/test-foo-load-controller.js

That is the only way I can get my custom controller to execute.

If I simply put the path to the controller in my test-descriptor.json, then the console log output tells me that it has loaded the controller, but it never executes it. The other controllers, such as Locator, always have console output log lines that show both loading and execution.

At the least, the "loaded" console log line for the custom controller is incorrect, or there should be some hint of an error message as to why the custom controller doesn't execute.

I think that also, there is a bug where local custom controllers are not read.

Please fix and advise.

https://github.com/yahoo/arrow/blob/master/lib/util/arrowsetup.js#L27

cb(null, ...) ignores any error that happens.

on https://github.com/yahoo/arrow/blob/master/docs/arrow_cookbook/arrow_in-depth.rst the link for Arrow Tutorial points to https://github.com/yahoo/arrow/blob/master/docs/arrow_cookbook/arrow_tutorial.html and goes to a 404 page

It is in the sentence: Consider the following scenario: You have just finished creating a suite of tests that validate the application we discussed in the Arrow Tutorial chapter.

According to the doc here, arrow / docs / arrow_cookbook / arrow_in-depth.rst , starting the proxy server is required for following redirects. I have done that, and can see no other Arrow requirements for following redirects. However, as soon as my test page redirects, the test exits completely. But I need to do some dom interaction on the destination redirect page, and then initiate a client-side redirect there (it's a log-in page that I need to fill in, and then hit 'submit' and move to my next test page).

I have the functional/integration test in the cookbook example working fine on my box. Although my test is simple, it involves a redirect. The redirect is server-initiated, but that seems to not matter in this case, as clicking on a link causes the issue as well. Also https vs. http seems to not matter.

The Command

arrow --startProxyServer=true test_descriptor.json

~ or ~

arrow test_descriptor.json

test_decriptor.json :

[
{
"settings": [ "master" ],

    "name" : "page-load",
    "startProxyServer" : true,
    "commonlib" : "./test-lib.js",

    "config" :{
        "baseUrl" : "https://foo.com"
    },

    "dataprovider" : {

        "int" : {
    "params" : {
                "test" : "test-page-load-int.js",
                "page" : "$$config.baseUrl$$/bar"
    },
            "group" : "smoke"
        }
}

},

{
    "settings": [ "environment:int" ]
}

]

The Output

Foos-MacBook-Pro:bat-page-load test$ arrow --startProxyServer=true test_descriptor.json
[2013-10-04 18:27:16.140] [INFO] ArrowSetup - Glob result: test_descriptor.json
[2013-10-04 18:27:16.487] [INFO] TestExecutor - Total test sessions: 1
[2013-10-04 18:27:16.487] [INFO] TestSession - Running test: int
[2013-10-04 18:27:16.488] [INFO] TestSession - Using selenium driver
[2013-10-04 18:27:16.506] [INFO] SeleniumDriver - Connecting to selenium: http://localhost:4444/wd/hub, browser: firefox, version: undefined
[2013-10-04 18:27:16.560] [INFO] Arrow - Loading controller: /usr/local/lib/node_modules/yahoo-arrow/lib/controller/default
[2013-10-04 18:27:16.562] [INFO] Arrow - Executing controller: /usr/local/lib/node_modules/yahoo-arrow/lib/controller/default
[2013-10-04 18:27:18.648] [INFO] SeleniumDriver - Loading dependency: ./test-lib.js
[2013-10-04 18:27:18.652] [INFO] SeleniumDriver - Loading dependency: /usr/local/lib/node_modules/yahoo-arrow/lib/common/yui-arrow.js
[2013-10-04 18:27:18.653] [INFO] SeleniumDriver - Loading test: test-page-load-int.js from
[2013-10-04 18:27:18.654] [INFO] SeleniumDriver - Loading page: https://foo.corp.bar.com/bat
[2013-10-04 18:27:22.302] [INFO] TestExecutor - Time taken to run - int is::5.82
[2013-10-04 18:27:22.303] [INFO] TestExecutor - Time taken to run descriptor-test_descriptor.json = 5.82
[2013-10-04 18:27:22.303] [INFO] TestExecutor - All tests over for descriptor:test_descriptor.json
Test Report Summary for the descriptor::test_descriptor.json

Total Number of Executed Tests: 1
Total Number of Passed Tests: 0
Total Number of Failed Tests: 1
Total Number of Skipped Tests: 0

Total Test Execution Time Inside Browser: 0.00 seconds
Total Test Execution Time:5.82 seconds

                       List of failed descriptors

Failed Descriptor Path : test_descriptor.json

Total Number of Failed Tests : 1

Foos-MacBook-Pro:bat-page-load test$

Should we link to the Arrow tutorial https://github.com/arrowfmk/arrow_tutorial or include it in the package?

If you don't have Selenium Server running on http://localhost:4444/wd/hub and run:
arrow_selenium --list
you get a Javascript error:

$ arrow_selenium --list
[2012-10-01 12:00:20.726] [INFO] selenium - Selenium host: http://localhost:4444/wd/hub
[2012-10-01 12:00:20.740] [FATAL] WebDriverSession - Error while connecting to Selenium Server: http://localhost:4444/wd/hub
[2012-10-01 12:00:20.740] [FATAL] WebDriverSession - Error Message : connect ECONNREFUSED

/Users/dmitris/.nvm/v0.8.11/lib/node_modules/yahoo-arrow/arrow_selenium/selenium.js:39
    if (0 === arrSessions.length) {
                         ^
TypeError: Cannot read property 'length' of null
    at listSessions (/Users/dmitris/.nvm/v0.8.11/lib/node_modules/yahoo-arrow/arrow_selenium/selenium.js:39:26)
    at ClientRequest.WebDriverSession.getSessions (/Users/dmitris/.nvm/v0.8.11/lib/node_modules/yahoo-arrow/lib/session/wdsession.js:58:9)
    at ClientRequest.EventEmitter.emit (events.js:93:17)
    at Socket.socketErrorListener (http.js:1330:9)
    at Socket.EventEmitter.emit (events.js:93:17)
    at Socket._destroy.self.errorEmitted (net.js:328:14)
    at process.startup.processNextTick.process._tickCallback (node.js:244:9)

If you do have a server running but no browser instances created, the error is:

$ arrow_selenium --list
[2012-10-01 11:55:37.488] [INFO] selenium - Selenium host: http://localhost:4444/wd/hub
[2012-10-01 11:55:37.505] [FATAL] WebDriverSession - No active selenium session found
[2012-10-01 11:55:37.505] [INFO] WebDriverSession - You can also create sessions here : http://localhost:4444/wd/hub

/Users/dmitris/.nvm/v0.8.11/lib/node_modules/yahoo-arrow/arrow_selenium/selenium.js:39
    if (0 === arrSessions.length) {
                         ^
TypeError: Cannot read property 'length' of null
    at listSessions (/Users/dmitris/.nvm/v0.8.11/lib/node_modules/yahoo-arrow/arrow_selenium/selenium.js:39:26)
    at IncomingMessage.WebDriverSession.getSessions (/Users/dmitris/.nvm/v0.8.11/lib/node_modules/yahoo-arrow/lib/session/wdsession.js:46:17)
    at IncomingMessage.EventEmitter.emit (events.js:123:20)
    at IncomingMessage._emitEnd (http.js:366:10)
    at HTTPParser.parserOnMessageComplete [as onMessageComplete] (http.js:149:23)
    at Socket.socketOnData [as ondata] (http.js:1366:20)
    at TCP.onread (net.js:403:27)

It would be good to add some error checking and user-friendly message - for example, a reminder that you can download the selenium-server-standalone JAR from http://code.google.com/p/selenium/downloads/list and start the server with:
java -jar path/to/jar/selenium-server-standalone-2.<minor_version>.jar

The jar linked to on the Tutorial, https://github.com/yahoo/arrow/blob/master/docs/arrow_cookbook/arrow_getting_started.rst#start-selenium , in this text:

Download the Selenium JAR Executable

did not work for me on Mac. I don't recall the error, but it was flagged on Google Code and fixed in later versions.

The latest version of Selenium works fine for me.

The version that didn't work (linked to at the tutorial), is: http://selenium.googlecode.com/files/selenium-server-standalone-2.25.0.jar

I'd like arrow to provide better error messages when an error occur. It wasn't vocal enough and it was hard to understand the cause for the failure.

For example, I wrote a yui test but accidentally passed --engine mocha.
Arrow timed out after a minute and showed the following text from the console:

[LOG]
[LOG]

It took me a long time to understand the mistake.
Arrow should be more vocal when users pass command line parameters that result in bad configuration.

I think a similar problem occurs if you pass an invalid path to --engineConfig for example.

@pranavparikh can elaborate implementation details.

The doc at:

https://github.com/yahoo/arrow/blob/master/docs/arrow_cookbook/arrow_in-depth.rst

reads, under heading "Using Session-Reuse":

"However, one should keep in mind that this approach means your test will have a sterile environment as session and cookie information will be reused"

but it seems it should read:

"However, one should keep in mind that this approach means your test will have a non-sterile environment as session and cookie information will be reused."

One thing I want to add to the arrow command line is the ability to execute a smoke test scenario repeatedly for a specified number of times, say 100. We collect test result each time and report on how many times it fails. It would be useful to see if the feature is stable or the test setup itself is stable. Sometimes I write a test and it pass locally for once or twice but after checking in the code, it would pass and fail intermittently. If I can simulate the repeated testing in the pipeline environment locally, it would increase my confidence that my test and the feature is stable. What do you think about this? It would be something like "--repeat=100".

Hi Arrow team, I would say this is a high impact pain point that impedes adoption. To replicate, add any syntax error to your YUI Test file, or descriptor. The error log will convey vague errors such as:

[2014-03-31 13:04:58.606] [ERROR] TestSession - Error while running controller :Scenario failed at atom: 1
Error :UnknownError: unknown error: Runtime.evaluate threw exception: SyntaxError: Unexpected token .
(Session info: chrome=33.0.1750.152)
(Driver info: chromedriver=2.9.248307,platform=Mac OS X 10.9.2 x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 13 milliseconds

Or:

[2014-03-31 13:04:58.606] [ERROR] TestSession - Error while running controller :Scenario failed at atom: 1

(respectively).

This is particularly problematic when front-loading markup testing to a descriptor file. A few solutions off the top of my head:

1. Identify wrapper points for try and catch statements within Arrow layers.
2. Json Lint / JSHint descriptor file come runtime.

For some reason, in our build environment, phantomjs has started failing randomly. Arrow should attempt to retry starting phantomjs, but instead it fails on the first standard error message. Here is the relevant line of code

https://github.com/yahoo/arrow/blob/master/lib/util/phantomJsSetup.js#L54

For a test MyTest.js with its 'it' block takes longer than 30 seconds:

var expect = require("chai").expect,
waitSeconds = 40; 

describe('My Test -> ', function() {
    it('should pass after ' + waitSeconds + ' seconds', function(done) {
        // explicitly set a longer timeout to make sure test can finish
        this.timeout((waitSeconds + 1) * 1000);
        // takes more than 30 seconds
        setTimeout(function() {
            expect(1).to.equal(1);
            done();
        },  
        waitSeconds * 1000);
    }); 
});

The test passes if run with mocha:
mocha MyTest.js -R spec

My Test -> 
    ✓ should pass after 40 seconds (40006ms)


  1 test complete (40 seconds)

But it breaks when run with arrow:
arrow MyTest.js --engine=mocha

 My Test -> 
    ◦ should pass after 40 seconds: [2013-12-29 21:12:36.375] [INFO] runNodejsTest - Waiting for the test report
[2013-12-29 21:12:36.877] [INFO] runNodejsTest - Waiting for the test report
[2013-12-29 21:12:37.377] [INFO] runNodejsTest - Waiting for the test report
...
[2013-12-29 21:13:05.922] [INFO] runNodejsTest - Waiting for the test report
    1) should pass after 40 seconds
{
  "title": "should pass after 40 seconds",
  "async": 1,
  "sync": false,
  "timedOut": false,
  "pending": false,
  "type": "test",
  "parent": "#<Suite>",
  "ctx": "#<Context>",
  "timer": {
    "ontimeout": null
  },
  "state": "failed",
  "err": {
    "uncaught": true
  }
}

  ✖ 1 of 1 test failed:

  1) My Test ->  should pass after 40 seconds:
     TypeError: Cannot read property 'length' of undefined
      at onReportReady (/home/kevin/dev/tlswe/aws-mock/node_modules/yahoo-arrow/nodejs/node.js:61:63)
      at null.<anonymous> (/home/kevin/dev/tlswe/aws-mock/node_modules/yahoo-arrow/nodejs/node.js:94:17)
      at wrapper [as _onTimeout] (timers.js:252:14)
      at Timer.listOnTimeout [as ontimeout] (timers.js:110:15)

Environments:
arrow: v0.0.88
mocha: 1.7.4 (as dependency in arrow)
node.js: v0.10.22

Is that a bug, or is there any switch or solution for enlarging arrows' default timeout (especially by using engine=mocha)?

Thanks very much!

If some assertion fails , Arrow should output the actual assertion which
failed.

For e.g, in the test below,

var module = Y.one('#mediasportsathletefilter'),
suite = new Y.Test.Suite("MediaSportsAthleteFilter Test Suite");

suite.add(new Y.Test.Case({
  "#1 - test Module Presence": function() {
    Y.Assert.isNotNull(module);
    Y.Assert.isTrue(module.one('.hd h3').get('text') !== '');
    Y.Assert.areEqual(module.all('.bd select').size(), 2);
    Y.Assert.areEqual(module.all('.bd input').size(), 4);
  }
}));

If Y.Assert.isTrue(module.one('.hd h3').get('text') !== ''); failed, output the
line

Y.Assert.isTrue(module.one('.hd h3').get('text') !== ''); alongwith the failed
assertion in the error message.

I am getting a "YUITest is not defined" when trying to run my tests. Right now I am consistently getting this error in my 'arrow_travis' branch (https://github.com/mridgway/mojito/tree/arrow_travis) when I run npm test. This script is running through unit tests for client-side JS with nodejs driver and then unit tests for server-side JS with selenium driver. The error only occurs on the nodejs driver.

Travis job with full run with debug log level: http://travis-ci.org/#!/mridgway/mojito/jobs/2280094

Previously, I thought this was caused by a YUI version mismatch between Mojito and Arrow, but that doesn't seem to be the case.

If you have a debugger statement in a client-side test the next test will start and cause your debugger to reset. There's no way to tell arrow not to time out.

$: arrow sports --func // checks out what you need
Use current / setup VM session? [Y/N]
Test a specific module? [Y/N to list modules, default tests all] // Etc..

Arrow hangs randomly while creating screenshots.

Similar issue reported on Webdriver forum -webdriver stops responding when
getting screenshot. -https://code.google.com/p/selenium/issues/detail?id=3239

To investigate.

Based on recommendation in #206 , users shall be able to specify the parameters like commonLib as

"commonLib" : ["lib1","lib2"] ( as well as "commonLib":"lib1, lib2" for backward compatibility)

If user is passing --capabilities from command line, Arrow should given an option to merge them with the default capabilities supplied by Arrow. So that user doesn't have to provide the default capabilities explicitly.

This fails:

suite.add( new Y.Test.Case({
        "foo-bar.bat_baz": function() {

but this succeeds:

suite.add( new Y.Test.Case({
        "foo bar bat baz": function() {

Making the issue very difficult to fix, there is no descriptive error message that helps to isolate the cause of the issue. I was able to fix by trial and error, finally hitting upon this key name. It is totally confusing because as a quoted js object key, it just should simply always work.

Not sure if the fail happens under plain YUITest without Arrow or not.

Note that in an object, this quoted key works fine:

foo = { "foo-bar.bat_baz": function() { console.log( 'No fail.' ); } };
Object {foo-bar.bat_baz: function}
foo[ 'foo-bar.bat_baz' ]();
No fail. 

Some users may want to use identifiers such as the fail-case one here because they wish to key their tests to json schema names, which are often named after the json paths used to the json instances that they validate.

Since the method identifier is simply a valid quoted js object key, as show just above, it just should simply always work. As a stop-gap measure, it would be acceptable to log a descriptive error message about the fail whenever the fail happens.

I have a use case where I want to run the same test with multiple window sizes to test my responsive design. I have created my own controller which successfully sets the window to the size I want, but when I call executeTest the window is maximized before the test is executed. Please provide a parameter to executeTest with which I can override this behavior. Thanks!

commonLib: "foo.js, bar.js", fails where commonLib: "foo.js,bar.js", succeeds.

However, it is common practice to allow a space in a comma-delimited list, for human readability. The fail is a failure to load the files specified.

commonLib: "foo.js, bar.js", should succeed.

Hi Arrow Team,

As a user debugging a major production issue module, caused by a module I have never worked on, one of the common questions is "what changed?" Therefore, I think there would be a lot of value in showing a 'last successful build' screenshot as well as the current 'after failure' screenshots (as well as html grabs). Maybe even a a markup diff feature at some point. All the above would be configurable in the event an operation's infrastructure cannot handle the extra file load. I am working on the Y! mod-dna project internally that would leverage the above, I'll showcase to you guys soon, which will take full advantage of these types of features as well.

Regards,

-A

Instead of wildcards we should use version numbers for our dependencies. This causes new versions of modules that we might not want, like YUI 3.7.0pr1 which was just pushed. I believe it also may have cause tests from breaking because of the new version of express. cc @ktslwy

Arrows internal debugging logs should be output at a "SYSTEM" level not "DEBUG" to keep the logs specific to those messages output by framework consumers.

Currently, when Arrow takes a "screenshot" of a failure, the HTML that it
outputs comes from this code:

webdriver.getPageSource().then(function(src) {
fs.writeFileSync(testName + "-" + count + ".html", src);
callback();
});

However, this always represents the state of the DOM when the page first
loaded, not the current state (which would be more useful). I'm not certain
how to get at the exact full current DOM, but I did get this far:

webdriver.findElement(webdriver.By.tagName("html")).getAttribute("innerHTML").then(function
(src) {
fs.writeFileSync(testName + "-" + count + ".html", src);
callback()
});

This at least gets the innerHTML of the element, which should contain at
least the part of the page you care about.

Am I completely off-base here, or is this something worth looking into?

-Andy

I am using yahoo-arrow as dependency and for the same I need request module > 2.16. [email protected] is having [email protected] and it fails for me with error [Error: UNABLE_TO_VERIFY_LEAF_SIGNATURE] for https with strictssl false.

I upgraded [email protected] (which pulls [email protected] )- as dependency for arrow manually and it works fine for me.

Here is the reference: yui/yui3#887

A nice feature would be to define globs within your descriptors for sets of tests that will be run with exactly the same settings. For instance:

"dataprovider": {

            "controllers":{
                "params": {
                    "test": "./tests/controllers/**/*.js",
                    "engine": "mocha",
                    "driver": "nodejs"
                },
                "group": "unit"
            }
        }

Arrow seems to spawn node processes which don't carry along debugging flags, meaning debugger statements on server-side tests are ignored.

Right now, if Selenium server is not up, Arrow fails with this message
"Error while capturing HTML screenshot: Error: EHOSTUNREACH connect EHOSTUNREACH" error

This shall be handled gracefully. Arrow should quit with error message "Selenium Server is not up".

Let's use this to track windows support. At the moment, when trying to test mojito in windows by using npm test, I'm getting some weird errors, I'm not so sure it is related, but in any case, I will add it here.

> [email protected] test C:\Users\caridy\Dropbox\Public\mojito-windows
> cd tests && node run.js test --browser phantomjs

node C:\Users\caridy\Dropbox\Public\mojito-windows\tests\run.js test --browser phantomjs

---Run both unit and functinal tests with phantomjs---
---Starting Arrow Server---
node C:\Users\caridy\Dropbox\Public\mojito-windows\tests/../node_modules/yahoo-arrow/arrow_server/server.js
Server running at: http://10.0.2.15:10000
Server running at: http://10.0.2.15:10001
Server running at: http://10.0.2.15:10002
Server running at: http://10.0.2.15:10003
Server running at: http://10.0.2.15:10004
Server running at: http://10.0.2.15:10005
Server running at: http://10.0.2.15:10006
Server running at: http://10.0.2.15:10007
Server running at: http://10.0.2.15:10008
---Starting Phantomjs---
node C:\Users\caridy\Dropbox\Public\mojito-windows\tests/../node_modules/phantomjs/bin/phantomjs --webdriver=4445
PhantomJS is launching GhostDriver...
[ERROR - 2013-05-16T23:03:29.382Z] GhostDriver - Main - Could not start Ghost Driver => {
  "message": "Could not start Ghost Driver",
  "line": 79,
  "sourceId": 66540992,
  "sourceURL": ":/ghostdriver/main.js",
  "stack": "Error: Could not start Ghost Driver\n    at :/ghostdriver/main.js:79",
  "stackArray": [
    {
      "sourceURL": ":/ghostdriver/main.js",
      "line": 79
    }
  ]
}

Running this throws the error

] [INFO] DataProvider - [ { uri: 'urn:uuid:5e6ecb36-72a3-4c3b-97ce-a9e367a597b6#/config',
schemaUri: 'urn:uuid:098fc000-6568-4665-a274-253a6813937a#/properties/config',
attribute: 'type',
message: 'Instance is not a required type',
details: [ 'object' ] },
{ uri: 'urn:uuid:5e6ecb36-72a3-4c3b-97ce-a9e367a597b6#/dataprovider/Test%20Data%20Driven%20Descriptor/params/page',
schemaUri: 'urn:uuid:098fc000-6568-4665-a274-253a6813937a#/properties/dataprovider/additionalProperties/properties/params/properties/page',
attribute: 'type',
message: 'Instance is not a required type',
details: [ 'string' ] } ]

[
    {
        "settings": [ "master" ],

        "name": "controllers",

        "config": [
            {
                "finance": {
                    "baseUrl": "http://finance.yahoo.com",
                    "searchBtnId": "#mnp-search_box"
                }
            },
            {
                "yahoo": {
                    "baseUrl": "http://yahoo.com",
                    "searchBtnId": "#p_13838465-p"
                }
            }
        ],

    "dataprovider" : {

            "Test Data Driven Descriptor" : {
                "group" : "func",

                "params" :{

                    "page": "$$config.baseUrl$$",
                    "test": "dataDriventest.js",
                    "searchBtnId":"$$config.searchBtnId$$"

                }

            }
        }
    },
    {
        "settings": [ "environment:preprod" ],
        "config": [
                    {
                        "finance": {
                            "baseUrl": "http://mail.yahoo.com",
                            "searchBtnId": "#mnp-search_box"
                        }
                    },
                    {
                        "yahoo": {
                            "baseUrl": "http://gma.yahoo.com",
                            "searchBtnId": "#p_13838465-p"
                        }
                    }
                ]

    }

]

Even though arrow supports all node version 0.6 and above, arrow has express2.5.2 as dependent package and which doesn't support node08.
Please update express to a version support node08.

npm info [email protected] engines
npm http GET https://registry.npmjs.org/express
npm http 304 https://registry.npmjs.org/express
{ node: '>= 0.4.1 < 0.7.0' }

npm info [email protected] engines
node >=0.6

Currently Arrow supports the --testTimeOut parameter, but this is global across
all descriptors. We have a few tests which take longer to run, but we don't
want to increase the global time out, as this would cause full failures to take
much longer to fail.

Ideally, we could have a JSON configuration option for the default controller
that could change this value to suit the specific case. Alternatively, setting
it in the descriptor config would get us most of the way, if that's more
convenient.

I find writing tests with the camme/webdriverjs to be much easier than with Selenium's webdriver. Does Arrow support it or can it be easily switched to support it?

If not, the team may want to consider adding support for https://github.com/camme/webdriverjs .

There are several bad links in the FAQ https://github.com/yahoo/arrow/blob/master/docs/arrow_cookbook/arrow_FAQs.rst:

1. "here" in "Linux box has arrow installed and it has all the packages listed here" points to http://devel-stage.corp.yahoo.com/ynodejs_arrow/guide/arrow_getting_started.html#linux-installation which is a Yahoo internal URL and is not reachable from outside Internet.

2. same in the next sentence - "selenium server" points to http://devel-stage.corp.yahoo.com/ynodejs_arrow/guide/arrow_getting_started.html#selenium-server

3. in https://github.com/yahoo/arrow/blob/master/docs/arrow_cookbook/arrow_FAQs.rst#how-do-i-point-arrow-to-a-specific-selenium-server, "config file's" link in "Update the config file's seleniumHost value" goes to 404 page
   https://github.com/yahoo/arrow/blob/master/docs/arrow_cookbook/arrow_in-depth.html#configuration

4. In https://github.com/yahoo/arrow/blob/master/docs/arrow_cookbook/arrow_FAQs.rst#error-arrowserver---buffer-65-78-65, "installed PhantomJS" link goes to a 404 page https://github.com/yahoo/arrow/blob/master/docs/arrow_cookbook/arrow_getting_started.html#mac-installation

5. In https://github.com/yahoo/arrow/blob/master/docs/arrow_cookbook/arrow_FAQs.rst#how-can-i-use-the-locator-controller-to-login, "locator" link in "Built-in, Arrow comes with two controllers, default and locator" goes to a 404 page https://github.com/yahoo/arrow/blob/master/docs/arrow_cookbook/arrow_in-depth.html#the-locator-controller

6. In https://github.com/yahoo/arrow/blob/master/docs/arrow_cookbook/arrow_FAQs.rst#how-can-i-install-a-specific-arrow-version "[email protected]" in "sudo npm install --registry=http:// [email protected] -g" becomes a mailto link which opens a mail client when clicked - will be confusing to users.

(there are also a couple of "TODO... needs to be updated" notes on that page)

Opening the file each time for writing to proxy.log , we run into this issue

Error: EMFILE, too many open files 'node/proxy.log'
at Object.fs.openSync (fs.js:427:18)

at ProxyManager.writeLog (node_modules/yahoo-arrow/lib/proxy/proxymanager.js:537:13)
at Object.userCallback (proxy/proxymanager.js:451:34)
at ClientRequest.<anonymous> (node_modules/yahoo-arrow/ext-lib/follow_redirect_http.js:97:41)
at ClientRequest.g (events.js:175:14)
at ClientRequest.EventEmitter.emit (events.js:95:17)
at HTTPParser.parserOnIncomingClient [as onIncoming] (http.js:1689:21)
at HTTPParser.parserOnHeadersComplete [as onHeadersComplete] (http.js:120:23)
at Socket.socketOnData [as ondata] (http.js:1584:20)

If startPhantomJs is set to true, look for local copy of PhantomJs. If it doesn't exist, look for globally installed phantomjs.

With Arrow versions greater than 0.0.85, including the very latest version of Arrow, working with the latest Selenium driver (also tried 2.40.0) and no global arrow install and no global phantom install, running with the Firefox web browser (latest: version 28.0), the yui js test files (test-lib.js, and the actual test) do not load, even though the test loads the page under test just fine (the page shows up, loads well under 30 seconds, and so forth).

It is evident that they do not load as no console logs are delivered under logLevel "debug".

The fact that the js test files do not load on the browser causes the test to timeout, as shown below. An error message about the timeout is shown, but doesn't contain any information pertaining to the actual problem.

This happens on two different Macs for me.

I can always work-around by reverting to yahoo-arrow version 0.0.85 .

Checking the docs for any changes in the convention for test-descriptor.json does not show any relevant changes.

I'd like to upgrade from 0.0.85, but this is blocking.

Log snippets:

$  node_modules/.bin/arrow test-descriptor.json --driver=selenium --logLevel=debug 
...
[2014-04-01 13:28:47.475] [INFO] SeleniumDriver - Loading dependency: ./test-lib.js
[2014-04-01 13:28:47.479] [INFO] SeleniumDriver - Loading dependency: /Users/foo/bar/bat/test/arrow/node_modules/yahoo-arrow/lib/common/yui-arrow.js
[2014-04-01 13:28:47.480] [INFO] SeleniumDriver - Loading test: ./test-page-load-int.js from .
[2014-04-01 13:28:47.483] [DEBUG] SeleniumDriver - {"platform":"MAC","javascriptEnabled":true,"acceptSslCerts":true,"browserName":"firefox","rotatable":false,"locationContextEnabled":true,"webdriver.remote.sessionid":"ff8aa474-5573-4891-8550-3c8c27f23849","version":"28.0","databaseEnabled":true,"cssSelectorsEnabled":true,"handlesAlerts":true,"browserConnectionEnabled":true,"webStorageEnabled":true,"nativeEvents":false,"applicationCacheEnabled":true,"takesScreenshot":true}
[2014-04-01 13:28:48.058] [DEBUG] SeleniumDriver - Waiting for the test report, attempt: 1
...
[2014-04-01 13:29:19.056] [DEBUG] SeleniumDriver - Waiting for the test report, attempt: 60
[2014-04-01 13:29:19.089] [DEBUG] SeleniumDriver - No debug log found in browser.
[2014-04-01 13:29:19.090] [ERROR] SeleniumDriver - Test Timed Out. It can happen due to one of these three reasons :
1) Your Test Script or page had a JS error, if you had --logLevel=debug in your command, you can see debug messages from browser console printed just before this error message.
2) The page got redirected before completing the test, this happens if your page has auto-redirects, or your tests perform some UI action resulting into page change event. Please use a custom controller for these kind of issues. 
3) Your test took more than 30 Seconds.```

Pure mocha works with istanbul in my project, but arrow does not.

Statements: 82.93% (34 / 41)      Branches: 53.57% (15 / 28)      Functions: 100% (13 / 13)      Lines: 82.93% (34 / 41)      Ignored: none 

File        Statements      Branches        Functions       Lines
full/       83.33%  (20 / 24)   50.00%  (8 / 16)    100.00% (8 / 8) 83.33%  (20 / 24)
simple/     82.35%  (14 / 17)   58.33%  (7 / 12)    100.00% (5 / 5) 82.35%  (14 / 17)

The above js code coverage report could be successfully generated by attaching to a pure mocha command:

istanbul cover _mocha example/nodejs/test/TestExamples.js -- -R spec

But unfortunately, when using arrow:

istanbul cover arrow example/nodejs/test/TestExamples.js -- --engine=mocha --testTimeOut=180000

we would get nothing covered at all:

Statements: 100% (0 / 0)      Branches: 100% (0 / 0)      Functions: 100% (0 / 0)      Lines: 100% (0 / 0)     

Using the "--coverage" switch for arrow, per its documentation too (in https://github.com/yahoo/arrow/tree/master/docs/arrow_cookbook), also reports "0 / 0" coverage at last (printed after test done, appended to arrow's report). Not much more clue found in arrow's doc.

What is the correct way to use istanbul with arrow (or output coverage data with arrow)?

Is there any handy usage example of arrow with code coverage report generated by istanbul? Or any known project on github that uses arrow for testing while generates its code coverage data with istanbul?

Here is some example code for reproducing the Arrow zero-coverage issue:

tar xzvf foo.tar.gz
cd foo
npm install && npm test

Look at the three repeated tests as written in the npm test scripts in package.json. Only the first test (pure mocha + istanbul way) outputs the right coverage report:

./node_modules/yahoo-arrow/node_modules/.bin/istanbul cover ./node_modules/yahoo-arrow/node_modules/.bin/_mocha test/MyTest.js -- -R spec

But the following two other ways with Arrow do not.

./node_modules/.bin/arrow test/MyTest.js --engine=mocha --coverage

./node_modules/yahoo-arrow/node_modules/.bin/istanbul cover ./node_modules/.bin/arrow test/MyTest.js -- --engine=mocha

They report nothing covered at all:

Statements: 100% (0 / 0)      Branches: 100% (0 / 0)      Functions: 100% (0 / 0)      Lines: 100% (0 / 0)     

Example code.

if I runt this command from arrow command line,
./node_modules/.bin/arrow ./arrow_files/basic_test/test-yahoo-title.js
--baseUrl=http://sports.yahoo.com --browser=firefox

It gives me error

[2013-04-08 12:03:23.997] [ERROR] SeleniumDriver - Cannot load a local file
without arrow_server running:
[2013-04-08 12:03:23.997] [ERROR] Arrow - Cannot load a local file without
arrow_server running:
/but with this

./node_modules/.bin/arrow ./arrow_files/basic_test/test-yahoo-title.js
--baseUrl=http://sports.yahoo.com --browser=firefox
--page=http://sports.yahoo.com

runs fine..

Basically, arrow should treat --baseUrl as --page in this case..

For a new module owner, it is often hard to decipher the purpose of each scenario when reading through a long scenario chain within a descriptor file. This could probably be baked into the insertion point of a JsonLint solution proposed here:
#222

In some testing, we make several get requests for beacons on the same host, and they all get redirected. The proxy is using http.request to make these calls, and for some reason the sockets are getting kept alive, which is blocking subsequent beacon firings after the first 5. This is due to the default max request setting for http.request for the same host:

http://stackoverflow.com/questions/16472497/nodejs-max-socket-pooling-settings

This can be fixed by:

options.agent = false;
http.request(options, func() { ...

I will create a pull request for this.

--capabilities takes file path as argument . Also allow JSON blob to be passed as capabilities