mozilla-services / services-test

Tools and test scripts used by the Mozilla Cloud Services team

services-test

Description

This repo is intended for: testing tools, test manifests and scripts used by the Mozilla Cloud Services team as part of the Mozilla Cloud Services automated test pipeline.

See: services-qa-jenkins for how these tests are used in Jenkins jobs. This is a private repo that requires permission to view.

Get Involved

You can read about our team and how to get involved at Read the Docs.

Here is a list of contributors to the repo.

Repo Structure

To contribute a new automated test to the services-test repo, please adhere to the following guidelines. This following file structure is required by Jenkins to execute automated tests.

• {project}/ (directory)
• One directory per project. For example:
  • services-test/absearch
  • services-test/autopush, etc.
• {project}/README.md (file)
• Add README file in project folder with links to: project repo (github), readthedocs, testplan, etc.
• {project}/{test-type}/ (directory)
• One project child directory per test-type. For example:
• services-test/absearch/e2e-test
• services-test/absearch/schema-check, etc.
• {project}/{test-type}/run (file)
• Add a "run" file in each test-type directory which should install all dependencies and kick off a test of the type indicated by the parent directory
• example: example run file
• {project}/{test-type}/manifest.ini (file)
• Add a manifest file in each test-type directory which should specify any environment-specific parameters
• example: example manifest.ini
• {project}/{test-type}/misc (files)
• Any additional files needed by that test type should be self-contained in that directory.

Test Environments

Environments where tests should be run for a given project

TEST ENV	DESCRIPTION
stage	where most testing happens before prod deployment
stage-loadtest	this is a special env for msisnd where we need a different setup for loadtesting. In general loadtesting can take place in stage.
pre-prod	final verification in an actual prod env (before DNS switch)
prod	final verification to make sure prod deploy was successful. Also, continuous prod testing can be executed as a kind of prod "health check"

Test Types

Specify what kinds of tests should be run for any given environment

TEST TYPE	DESCRIPTION
stack-check	verify stack procs, urls, etc. are running
config-check	verify application configuration
e2e-test	client-side test
loadtest	verify application scalability
schema-check	API test
security	ZAP test (TBD)
cdn-test	check existence/delivery of remote files

services-test's Issues

[pep8] Fix imports in msisdn-gateway e2e fabfile

/msisdn-gateway/e2e-test/fabfile.py:1-2 has the following:

from fabric.api import *
import fabric

Flake8 doesn't like dis, and complains:

• ./msisdn-gateway/e2e-test/fabfile.py:1:1: F403 'from fabric.api import *' used; unable to detect undefined names
• ./msisdn-gateway/e2e-test/fabfile.py:2:1: F401 'fabric' imported but unused

We should try and improve the glob import to avoid flake8 errors and avoid having to do weird hacks like adding # noqa to get flake8 to ignore our code.

[mozdownload] replace _utils/firefox_download.py

mozilla/mozdownload#345

Mozdownload now fixed for OSX. we should look at replacing our custom one:
https://github.com/mozilla-services/services-test/blob/dev/_utils/firefox_download.py

Add shavar stack-check verification

[refactor] remove ff-tool (first attempt) files from services-test

We don't need any of the files that are now in ff-tool and should remove them from here
https://github.com/mozilla-services/services-test/tree/dev/_utils

[feat] add sync_e2e-test manifest.ini

Fix utility scripts to inject global env vars

Migrate tests and page objects back to services-test

move services-marionette dir to _shared or separate repo

feat: add loop prefs.ini file

Add autopush loadtest config file

moved to:
https://github.com/rpappalax/autopush-loadtests

Abort job when mozilla-central is unavailable (i.e. during maintanence job)

• cd /home/jenkins-slave/workspace/sync_e2e-test_prod/mozilla-central/testing/tps
  ././sync/e2e-test/run: line 30: cd: /home/jenkins-slave/workspace/sync_e2e-test_prod/mozilla-central/testing/tps: No such file or directory
• exit 1

Add run file: kinto tests

[_utils/firefox_download] add args

right now firefox_downloads.py will download all new version of FF for the Jenkins slave env.
This tool would be more useful for general testing if we could specify specific version via args.

Add 'schema-check' tests to Normandy

Add a directory to store tests that are checking the Normandy API responses for valid schema and properties

add run file: loop-client tests

Change test-type of loop-client "deployment-tests" to standard test-type

The deployments-tests look good, but should prob be named consistently per the services-test README to: "schema check":
https://github.com/mozilla-services/services-test/blob/dev/README.md

@chartjes on the other hand, would it make more sense to create a more descriptive test type for tests like these called "config-check"?

[docs] update README to reflect standardized file structure

Add manifest.ini schema linting

Ref: https://github.com/mozilla-services/services-test/blob/dev/sync/e2e-test/manifest.ini

True vs true
cron formatting
prod vs production
job_timeout must be integer
required fields present
manifest file must define at least 3 envs (dev, stage, prod)

[firefox_profile_handler] test code on win, osx, linux

https://bugzilla.mozilla.org/show_bug.cgi?id=1206254

Update README with team info, links, communication, contributor info, etc.

Remove deprecated projects: loop, msisdn-gateway, etc.

@chartjes looks like no further deploys for loop? If so, can archive the tests in a release and or tag?

Add shavar e2e-test prefs files

We can now use ff-tool to e2e-test shavar, but will need to first convert and add shavar prefs files.

TODO:
figure out how to handle multiple pref sets

[test-manifests] Add a "security" test type

Simplify Firefox downloaden

Looking at ./_shared/install_ff_nightly.sh, it may be slightly simpler if we use the lovely firefox-get npm module (assumes the host images all have Node.js installed).

$ curl $(firefox-get -v beta) | tar xz --directory ff_beta

NOTE: This assumes you already have an ./ff_beta/ directory.

We could also possibly abstract this one step further by creating some UNIX-y function where we pass a target "version" of Firefox that we want to download and it will ensure the directory is created for us.

Then we can do one/all of the following:

• $ download_firefox release
• $ download_firefox beta
• $ download_firefox nightly

From there we can create some ENV variables to point to the different Firefox versions, where $FF_BETA points to ./ff_beta/firefox/firefox-bin.

Remove inert run files from all projects

Various projects have inert run(.sh) files that were added as a "first attempt" at scoping out how services-test would operate. Now that we have a working architecture, these should all be removed.

Add opengraph test

This is the Opengraph test I have finished and have been requested to merge here.

[pep8] fix line length errors

flag lines with legit >80 length, then remove E501

Add sync_e2e-test_prod

Enable mozplugin files for shavar

[DEFAULT]
browser.safebrowsing.blockedURIs.enabled = true

and add: mozplugin2-block-digest256" to urlclassifier.blockedTable

chore: remove loop/e2e-test

This test was a proof-of-concept that no longer does anything.

Update shavar/e2e-test/prefs.ini to reflect new shavar lists

docs: Add to README note on best practices: i.e. - using virtualenv for python projects

https://github.com/mozilla-services/services-test/search?utf8=%E2%9C%93&q=virtualenv&type=Code

We're doing virtualenv . in a couple places, which can make things a bit messy and hard to exclude when trying to grep or use globs in certain tools.

Merge /_temp with /_utils/_temp?

Currently we have a couple "floating" _temp folders

1. the root /_temp/ folder is used for temporary Firefox Profiles (which we can clean up in the nightly refresh job).
2. We have a /_utils/_temp/ folder is used for temporary Firefox binaries (added by #55 in /_utils/firefox_download.py:41)

We should maybe consider using /_temp/profiles/ and /_temp/firefoxes/, or something.
This will keep everything close, and not have a bunch of random temporary folders peppered throughout the repo.

Add daily maintenance script for ubuntu-desktop mozilla-central downloads

script downloads to maintenance workspace and setups up symlink

[firefox-tool] create a separate package

For ease-of-use, we should probably consider moving the download, install, uninstall and profile manager mods bundled into their own package w/ 1 or 2 console entry points. It would be easiest if we start with 1 menu of args and see if that looks practical. I looked through quite a few other github repos around firefox and found some other partial attempts at this, but I think if we put our tools together, it might be fairly useful or others outside of our team as well.

INSTALLERS
https://github.com/chef-cookbooks/firefox
https://github.com/bellpeterm/firefox
https://github.com/zanardo-ansible/firefox
https://github.com/boxen/puppet-firefox

HACKS
https://github.com/omgmog/install-all-firefox/blob/master/install-all-firefox.sh

PROFILE MANAGERS
https://github.com/Puppet-Finland/firefox

Add test-catalog.ini

We need to add a test-catalog.ini to services-test.

The test-catalog.ini in dev would "activate" the jenkins-job-manager to create a corresponding job for every test listed in the catalog in the jenkins-dev.
Likewise, a test-catalog.ini file in the master branch, would activate corresponding jobs for every test listed in the master jenkins.
Forcing contributors to explicitly add their tests to this list would permit contributors to retain tests within services-test that still can be housed in github, but would not have a corresponding Jenkins job.

Example:
https://github.com/rpappalax/jenkins-job-manager/blob/dev/test_catalog.ini

[feat] refactor sync_e2e-test job to run script

also add more to manifest.ini for job definition

Remove normandy "study_tests"

@chartjes would these be schema-checks?

https://github.com/mozilla-services/services-test/tree/dev/normandy/study_tests

Change autopush test prefs to point at ewong's test page

add run file: fennec-dlc-catalog tests

[travis] fix tests for python3.4 compatibility

seems like largely converting print to print()

[firefox_install] test code on win, osx, linux

We have some dummy code for this in print statements.
Need to flesh it out

Move manifest.json files (for old stack check) to separate branch

Run pep8 and ESLint on commit?

Me thinks we can use Travis-CI to do this, although not sure of the optimal way. I can try experimenting with this.

Was also looking at Tox since I've seen that used in other projects, but not sure if that is overkill.

As for checking JavaScript files, we could use ESLint, but that may get tricky if we plan on excluding oodles of files/folders (in which case we may need Gulp or Grunt, and we're starting to add a few dependencies to our Travis env).

Probably best to just dive in and start doing a bunch of PRs and see what falls out.

Remove MSISDN-Gateway tests

MSISDN-Gateway tests no longer needed

https://bugzilla.mozilla.org/show_bug.cgi?id=1262454

fix: Modify all test run.sh / run.py files to drop file extension

please include:

#!/usr/bin/env python
#!/usr/bin/env node
#!/usr/bin/env cobol

so then we can also leave off the file extension.
Since all our jenkins jobs are generated dynamically (and we can't assume one language), let's just drop the extension altogether.

For treeherder integration, we will probably need to go with either python or node:
see: https://bugzilla.mozilla.org/show_bug.cgi?id=1213542#c8

README should also be updated accordingly

feat: add daily maintenance script for ubuntu-desktop firefox updates

Example file links in README are broken. :(

STR:

1. Go to https://github.com/mozilla-services/services-test
2. Click 'example run file' or 'example manifest.ini'

Expected results:
See docs

Actual results:
404

Add autopush pytest for URL checks

[firefox_uninstall] test code on win, osx, linux

We have some dummy code for this in print statements.
Need to flesh it out