salsita / shishito

Python module for selenium webdriver test execution

Shishito

Shishito is module for web and mobile application functional testing using Selenium Webdriver & Python. It runs tests using included libraries and generates nice test results output.

Documentation - http://shishito.readthedocs.org/en/latest/index.html (hosted on Read the Docs)

Features

• runs python Selenium Webdriver tests via PyTest
• easy configuration for local and remote (BrowserStack, Appium, ..) test execution
• contains useful test libraries
• generates HTML test results report (with screenshots for failed tests)
• designed to be used as a module (by multiple projects if needed)

Pre-requisities

Install Python moodules from requirements.txt

pip install -r requirements.txt

Webdriver drivers need to be setup (ChromeDriver, InternetExplorerDriver etc.)

Quick Start

1. clone Shishito repository. git clone [email protected]:salsita/shishito.git
2. add shishito directory into PYTHONPATH environment variable
3. clone sample test project repository https://github.com/salsita/shishito-sample-project git clone [email protected]:salsita/shishito-sample-project.git
4. if you want to use BrowserStack for running your tests, replace "bs_username", "bs_password" values with your credentials in shishito-sample-project/config/server_config.properties or pass it to runner python file as command line argument using flag --browserstack username:token
5. if you want to use Saucelabs for running your tests, add your credentials to saucelabs variable in shishito-sample-project/config/server_config.properties or pass it to runner python file as command line argument using flag --saucelabs username:token
6. set your preferred browser settings in shishito-sample-project/config/web/(browserstack|local).properties or for mobile apps in shishito-sample-project/config/mobile/appium.properties
7. run google_test_runner.py in sample project folder!

If you use local driver, you should now observe browser being started and tests running. There are information about progress shown in console output. Once testing is finished, HTML report can be found in:

shishito-sample-project/results folder # HTML report
shishito-sample-project/results_archive folder # zipped HTML report

Continuous Integration

Using Shishito with Continuous Integration solution, such as Jenkins, is easy! All you need to do is clone Shishito repo and add it into the PYTHONPATH.

Example script below (Jenkins "execute shell" build step):

#!/bin/bash
######################
# clone Shishito  #
######################

cd $WORKSPACE
git clone [email protected]:salsita/shishito.git

######################
# VARIABLES          #
######################

export PYTHONPATH=${PYTHONPATH}:/$WORKSPACE/shishito

######################
# SCRIPT             #
######################

python google_test_runner.py

Command line options

--platform web         # define platform on which run tests (currently supported: web, mobile, generic)
--environmnet local    # define environment in which run tests (currently supported: local, browserstack, appium, remote)
--test_directory tests # define directory where to lookup for tests (project_root + test_directory)

# supported platform/environment combinations:
#   generic/local
#   generic/remote
#   web/local
#   web/browserstack
#   web/remote
#   mobile/appium (can run on local/remote appium server or on saucelabs)
#   node_webkit/node_webkit

--smoke # runs only tests with fixture "@pytest.mark.smoke"

--browserstack testuser1:p84asd21d15asd454 # authenticate on BrowserStack using user "testuser1" and token "p84asd21d15asd454"
--saucelabs testuser1:p84asd21d15asd454 # authenticate on Saucelabs using user "testuser1" and token "p84asd21d15asd454"
--test_rail [e-mail]:1AVFS51AS # authenticate on TestRail using user email "[email protected]" and password "1AVFS51AS"

If no arguments are specified, Shishito, by default, searches for settings combinations in (server|local).properties files and runs tests according to them.

Configuration files

server_config.properties

• default configuration file with test variables
• changes to variables should be maintained in VCS; so that configuration can be reused for automated test execution

# modules
test_platform=web
test_environment=local

# test dir
test_directory=tests

# General
base_url=http://www.google.com
environment_configuration=Chrome

• test_platform - on which platform run tests (web, mobile)
• test_environment - in which environment run tests (local, browserstack, appium)
• test_directory - in which directory lookup for tests
• base_url - url that will be loaded by default upon start of each test
• environment_configuration - which configuration use from .properties file (used when tests are run without runner)
• remote_driver_url - remote driver hub. Selenium server needs to be running on this url.

local_config.properties

• if variable local_execution=True, script will look first search local config for test variables
• in case variables are not found, it will fall back to values in default server_config.properties
• changes to this file should not be maintained in VCS (they serve only for local test execution)

<platform>/<environment>.properties

• contains combinations, for which the tests should be executed
• e.g. browser and resolution for local web browser

conftest.py

• helper file that defines command line arguments, provides fixtures and other information for Shishito runner

Configuration

Shishito can be configured with command lines arguments and config files. Some configuration values are also added as arguments to PyTest (depends on test environment). Configuration values are looked up according to these priorities:

1. pytest.config
2. command line arguments
3. local configuration file (if enabled: local_execution=True)
4. server cofiguration file

Node-webkit configuration

Shishito is able to run tests against node-webkit applications. Current implementation does not allow tester to specify based URL, just to run application from URL directly specified within application. Creating of webdriver driver object is done by specific chromedriver which has to be placed in same directory as node-webkit application. Chromedriver will search for node-webkit binaries and start the application. Binaries have to have specific names otherwise chromedriver won't find them.
Node-webkit binary must have name:

• For Linux: nw
• For Windows: nw.exe
• For OS X: node-webkit.app

TEMPORARY SCREENSHOT ON FAILURE FUNCTIONALITY

Due to issue in Node-webkit chromedriver; there is added temporary screenshot on failure functionality using pyscreenshot module. This functionality takes screenshot of whole desktop not only node-webkit application window. This issue should be fixed in chromedriver 2.15. There is alpha version of node-webkit chromedriver v2.15. This functionality is going to be removed once issue is fixed.

Note: Ubuntu: It is necessary to install also python-imaging sudo apt-get install python-imaging

Troubleshooting for Node-webkit platform

If you see exception similar to one below

   raise WebDriverException(
       \"\'\" + os.path.basename(self.path) + \"\' executable needs to be \
           available in the path. Please look at \
           http://docs.seleniumhq.org/download/#thirdPartyDrivers \
           and read up at \
\>                   http://code.google.com/p/selenium/wiki/ChromeDriver")
E    WebDriverException: Message: 'chromedriver' executable needs to be available in the path. Please look at http://docs.seleniumhq.org/download/#thirdPartyDrivers and read up at http://code.google.com/p/selenium/wiki/ChromeDriver

/usr/local/lib/python2.7/dist-packages/selenium/webdriver/chrome/service.py:70: WebDriverException

You need to check Chromedriver file access rights mainly in Linux or OS X, Windows should be ok.

Test Management Support

Shishito support upload of test results to TestRail test management app. Following properties in server/local config have to be filled:

• test_rail - credentials for test rail. Can be also left empty and passed via cmd argument (see above)
• test_rail_url - URL of test rail instance. (example: https://mycompany.testrail.net)
• test_rail_project_id - ID of TestRail project (example: 1)
• test_rail_section_id - ID of TestRail test section (example: 2)
• test_rail_test_plan_id - ID of TestRail test plan (example: 5)
• test_rail_suite_id - ID of TestRail test suite (example: 1)

For further information, see TestRail API documentation http://docs.gurock.com/testrail-api2/start.

shishito's Issues

Allow to run test on any kind of Remote driver

Right now we allow tests to run locally or on remote BrowserStack driver.

We should also allow them to run on any kind of remote driver = for instance selenium grid server.
This should be great to use it with https://github.com/yveshwang/selenium-2step = for running test locally without disturbing local session.

Sould not be too difficult. Basically it seems we just take call_browserstack_browser() method and split the part when Remote Driver is called into separate method.

Big gap between browsers reports in Combined report.

Big gap between browsers reports in Combined report. See screenshot for better clearness.

Steps to reproduce:
Prerequisites: Set up Shishito framework

1. Clone Shishito Sample project
2. Setup dependencies to Shishito framework
3. Execute Sample project
4. Open created CombineReport.html from results folder within the project

Expected result: Result table for each browser are one below another without any big gap between.
Actual result: Result table for each browser are one below another but with big gap in between.

Add full support for browser extensions

• Add full support for starting browser with more extensions at once.
• Figure out how to setup extension for Firefox - i.e. set_preference option

Misiing links in html report to Browserstack job

Look at html report not all tests have html links to Browserstack

Add support for running tests on mobile devices

We haven't support for running tests on mobiles.
Need to add option:
device, browserName and platform.
Simple mobile capabilities for Browserstack look like this:
desired_cap = {'platform': 'MAC', 'browserName': 'iPhone', 'device': 'iPhone 5' }

Allow selenium to accept untrusted certificates

Tests running on sites using untrusted certificates will now crash. Support for acceptance of untrusted certificates needs to be added.

Make availability to create reports use salsa_runner for non-selenium tests

For now if I try to use salsa runner with non-selenium tests - it's shows error dependent on driver settings. So it would be good to have runner works for non-selenium tests as well:
Errors on running:
File "E:\workspace\salsa-webqa\salsa_webqa\salsa_runner.py", line 40, in init
self.driver_name = self.tc.gid('driver')
File "E:\workspace\salsa-webqa\salsa_webqa\library\control_test.py", line 77, in gid
if local_execution.lower() == 'true':
AttributeError: 'NoneType' object has no attribute 'lower'

Add details into test rail results

Currently, we only sync to TestRail that test-case either passed or failed.
We should add failure message (stack-trace) for failed test-cases
This means simply passing tag content (shishito xml xUnit report) into test-case description.

Additionally, it would be useful to show link to Circle CI job in Test Run entry.

Write documentation for Browser Extension support.

Move browserstack credentials to jenkins configuration parameters.

Move browserstack credentials to jenkins configuration parameters. Set those credentials as command line parameters.

Shishito sample project is not fully updated according new Shishito structure

There are still some part of Shishito sample project which are not either updated or removed to fully comply with new Shishito structure. "Salsa_webqa" is still mentioned in several parts of sample project like in /library/google_lib.py

Setup proper logging for Shishito

Some print statements are useful, some can be spamming console output with unnecessary information (that is why gid() print statements were commented out.

I suggest using python logging functionality such as:

import logging

logging.basicConfig(
    format='%(levelname) 10s %(asctime)s %(message)s',
    level=logging.ERROR
)
log = logging.getLogger('error_log')

log.error('something is wrong man.')
log.warning('something is funny man')

Code snippet above will print only "error" messages

Use testreail API key instead of password

Now when I add support in shishito for testrail I need to specify password instead of API key. This need to be changed

Create pypi package for Shishito

Add support for extracting variables from non-selenium config

#25 brings support for running non-selenium tests.

While there is an option for having non-selenium config file to store test variables, method gid from ControlTest was not updated to be able to retrieve those variables yet.

Support need to be added, such that those variables can be used in tests.

Add support for appium execution (remote driver)

Add support for running tests using Appium.

Basically we need to introduce new config for appium, add new appium capabilities to conftest & add new methods for handling those capabilities into test config_test.py

Library names causing PyTest to think they're tests

Libraries such as "TestSelenium" and "TestControl" are automatically loaded by PyTest in test discovery (because of prefix Test). PyTest thinks those files contains testcases but when it finds out they do not, it marked those files as skipped in report.

This does not affect test runtime in any negative way. Just reports some additional skipped tests in PyTest console which is confusing.

Proposed convention = [library name]Test
This change will require updates on affected test suites.

Test which ends with error is marked as passed in Test rail.

Test which with error is marked as passed in Test rail. If test ends in error during the initialization, test itself has no result - success, fail, blocked, no-run but error and this is not recognized as fail in Testrail. Test is marked as successful in Testrail.

This issue can be reproduced in way that test initialization is broken, test is executed and result of test is checked in Testrail. Test should be marked as failed/blocked in Testrail not successful.

Add support for testing extension on BrowserStack

Title is self-explanatory.
Snippet below already works for Chrome, need to add proper support to SalsaWebQA now.

# Start Chrome
options = webdriver.ChromeOptions()
options.add_extension(os.path.join(self.extension_path, 'test_extension.crx'))

capabilities = {'os': 'Windows',
                'os_version': '7',
                'browser': 'Chrome',
                'browser_version': '35.0',
                'resolution': '1280x1024',
                'project': 'Extension Test',
                'build': 'v0.1'}

command_executor = 'http://[username]:[password]@hub.browserstack.com:80/wd/hub'
desired_capabilities = options.to_capabilities()
desired_capabilities.update(capabilities)
self.driver = webdriver.Remote(
    command_executor=command_executor,
    desired_capabilities=desired_capabilities)

Move common test fixtures from project to library

There are common fixture functions in conftest.py (in sample project). They are picked up by tests during runtime. Problem is that conftest.py is directory-specific file and therefore needs to be located in the project folder. Having it in salsa_webqa folder will not work.

There are possible solutions to this:

• create custom plugin with those common fixtures http://pytest.org/latest/plugins.html#setuptools-entry-points
• automatically copy the file from salsa_webqa to project folder before tests start

Wrong test name passed to browserstack

There is currently problem with passing testcase name to browserstack over the API in "test_control.py" file. Test name is automatically set as 2nd file in call stack (example: A called B that called C. If C is the "test_control.get_capabilities()" then A is the "test").

However, now when test control methods can be overridden, call stack can have various depth and such solution is no longer reliable.

See - "test_control.py", line 96

Error when running test project containing only none selenium tests

When running test project containing only none selenium tests, test project ends with error during test initialization.

Error message:

usage: voicekick_runner.py [options] [file_or_dir] [file_or_dir] [...]
voicekick_runner.py: error: unrecognized arguments: --html=....../integration_tests/results/2015-02-02_13-39-20/Firefox.html

Process finished with exit code 2

Steps to reproduce:

1. Create test project using latest shishito framework
2. Crete none selenium folder
3. Create none-selenium test
4. Run project (using default configuration)

Expected result: Tests are executed.
Actual result: Test ends in error during initialization

Note: Using --reporting simple parameter is workaround how to run none-selenium only test project.

Add support for node-webkit application test automation

Add support for node-webkit application test automation

• incorporate Chromedriver to be able to start node-webkit application (use: https://github.com/nwjs/nw.js/wiki/Chromedriver)
• extend start_browser method to start appropriate browser
• check whether name of node-webkit application is correct according to Chromedriver requirement. If it is not correct rename it accordingly.
• check whether Chromedriver is located in the same folder as node-webkit application
• add path to folder with node-webklit application to all property files

Use pytest config instead of environment variables for BrowserStack credentials

Browserstack credentials should be ideally passed as pytest config variables, not as environment variables = this is not very safe.

This means: passing it for instance as cmd argument to salsa runner, where you set it as pytest argument:

pytest_arguments.extend([ '--jirausername=' + jirausername])

and you collect it in control library, for instance

jirauser = pytest.config.getoption('jirausername')

• it needs to be added to conftest.py

ControlTest.gid() function behavior needs to be changed to fall back to pytest config variables instead of environment variables

Merge various browserstack combination results into one report

Currently there is one report generated for each browserstack combination (specified either by OS env variable or browserstack.properties file)

Those reports should be merged into one HTML file with some possibility to filter, based on combination

Unable to set more browserstack combination when launching specific tests manually

User is currently not able to set specific browserstack combination (= use existing configurations defined in config/browserstack.properties file) when manually launching tests directly - not using the test runner "salsa_runner.py".

In this case, if "BrowserStack" property is specified in appropriate config file, the default configuration will be used. This default configuration is set within conftest.py.

"test_control.py" must be updated in such way that browserstack.properties combinations are used even when running test over browserstack without using the runner script.

Remove temporary screenshot on failure solution for node-webkit platform.

Remove temporary screenshot on failure solution for node-webkit platform.

• Remove temporary method: node_webkit/control_test > stop_test
• Remove pyscreenshot from requirements.txt

Screenshots can be overridden if tests are run for multiple browsers

Screenshots are now overridden if same test fail for another BS combination. Need to append filename with combination string

Separate issue created (was bundled together with combined report generation)