18f / testing-cookbook

The 18F testing cookbook contains recipes and best practices for automated and manual testing in lots of different environments, languages, stacks and platforms.

layout	permalink
default	/

Testing Cookbook

The 18F testing cookbook contains recipes and best practices for automated and manual testing in lots of different environments, languages, stacks and platforms. Check 'em out:

• JavaScript testing in Node and the browser
• Python testing in pure Python, Django and Flask, etc.
• Ruby testing with RSpec, minitest, etc.
• Browser testing and VirtualBox
• Load testing with Locust and wrk

Public domain

This project is in the worldwide public domain. As stated in CONTRIBUTING:

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.

testing-cookbook's Issues

Add a load testing section

Can probably start stealing from 18F/college-choice-data#7, maybe even lure @dnesting into drafting, or at least contributing or reviewing.

cc: @shawnbot

Document linting and auto-formatting tools.

Every developer should use a text editor with the relevant linting plugins turned on, unless maybe they're using gofmt or yapf.

Add repo to 18F org and set primary branch to gh-pages

Once this is done, http://18f.github.io/gh-pages/ should build, which will provide a nicer link from http://18f.github.io/guides/.

@shawnbot, I'll ask about getting the repo added; but since you created the repo, you could tweak the primary branch right now.

Display on 18F pages is not quite right

There are some sections and some subsections in the Python portion, at least, that may not be displaying at all. There is also some stray boilerplate that should be removed. See:

https://pages.18f.gov/testing-cookbook/python/

But the doctest sub-pages are not showing up.

https://github.com/18F/testing-cookbook/tree/18f-pages/pages/python/doctest

There are some broken links, as well.

Proposed change in description of repo.

Propose changing description of repo to: The 18F testing cookbook contains recipes and best practices for automated and manual testing in lots of different environments, languages, stacks and platforms.

That way, anyone who comes across it here will know exactly what the repo contains.

Identify good browser testing tools and provide guidance for different use cases

So, I've been researching this topic on and off for months now, and have basically gotten nowhere. What we desperately need at 18F is a good browser ("functional") testing tool that we can easily add to our projects. My ideal would be one that does the following:

1. Uses Selenium (both locally and remotely)
2. Supports cloud services like Sauce Labs or BrowserStack out of the box
3. Runs easily on CI services like Travis and Circle

The CI requirement is tricky, as I've documented in this pull request, especially when your app needs to run alongside the test suite with a language other than Node. So, in addition to just identifying a tool, we should also provide some recipes and/or boilerplate for the following matrix of environments:

1. Running tests on different Selenium environments:
   • with local browsers (including PhantomJS) using Selenium Server
   • with virtual browsers on cloud services like Sauce Labs
2. Running tests against different app/site URLs:
   • a static, public URL
   • a local URL over a tunnel, e.g. with Sauce Connect
   • an "ephemeral" local server URL, e.g. when you run your app server alongside the test suite on CI services
3. A series of target browsers, hopefully identified via analytics and/or research, on one or more of the above environments, for instance:
   • Chrome and Firefox (theoretically the same on all platforms, locally or virtually)
   • Internet Explorer (9, 10, 11, Edge, etc.) on Windows, virtually
   • Safari on iOS (virtually)
   • Chrome on Android (virtually)

Also, I think it would be good for this cookbook to help people avoid writing tons of browser tests by writing better unit tests to cover "core" JavaScript, and only browser test the parts of their code that interface with browser APIs. This should help us avoid writing bloated, difficult-to-maintain browser test suites and save a bunch of time waiting for the tests to run on (potentially) lots of different browsers.

It feels to me like I'm the wrong person for this job, not only because I'm relatively new to testing, but because I also just don't have any time for it right now. So I'm tapping @DavidEBest, @yozlet, @jmcarp, @msecret and @vzvenyach because they have some interest and/or experience with this stuff, and I wholeheartedly defer to their judgement. Feel free to rope in other folks who know about this, and share thoughts or resources here.

I mentioned to Yoz that I've had a hard time finding any good examples of browser testing tools like Nightwatch and WebDriverIO in practice (as used on an actual site, as opposed to just usage examples), so that might be a good place to start. It's also probably worth listing the testing needs of projects that we've all worked on so that we can get a better sense of what our needs as an organization are. I'm happy to contribute those for the projects I've worked on so far.

Document cross-browser & CI testing setup on college-choice

The setup is documented in this pull request, but it needs to be summed up and explained in plain English. Some things to touch on specifically:

• Configuring Travis to run tests against the live Jekyll server was tricky.
• Using Sauce Connect can produce false positives. At one point I wasn't even running the Jekyll server on Travis, but the tests were succeeding because I still had a tunnel open, and Sauce was just running the tests against the site running on my laptop.
• Explain the Nightwatch configuration, and how having the config parsed as JavaScript (rather than JSON) made things much easier to manage.
• Explain the multi-browser testing setup and the scripts that call Nightwatch once per browser.

Document code coverage tools and configuration

Document codecov, coveralls, and other relevant services. Link to resources on getting your test runner to output compatible output.