jquery / testswarm

Distributed continuous integration testing for JavaScript.

TestSwarm - Distributed Continuous Integration for JavaScript

TestSwarm provides distributed continuous integration testing for JavaScript.

The main instance monitoring jQuery core and related projects runs at swarm.jquery.org.

Project Status

TestSwarm is used in projects of the jQuery Foundation, but it isn't under active development anymore. Although critical issues may be patched in the future, most open issues will remain unaddressed.

Within the jQuery Foundation, we're experimenting with alternative projects, to eventually shut down our own instance of TestSwarm:

• airtap
• Karma
• Intern
• browserstack-runner

We recommend reviewing those and other alternatives.

Quick start

Clone the repo, git clone --recursive git://github.com/jquery/testswarm.git.

Bug tracker

Found a bug? Please report it using our issue tracker!

Installation

Browser compatibility

• Chrome 58+ (2017)
• Edge 15+ (2017, both legacy MSEdge and Chromium-based)
• Firefox 45+ (2016)
• Internet Explorer 9+
• Opera 36+ (2016)
• Safari 9+ (2015)
• Android 4.3+ (2013)
• iOS Mobile Safari 7+ (2013)

Environmental compatibility

To run TestSwarm you will need a web server, a database server and PHP. At the moment TestSwarm supports the following, but other configurations may work as well.

• Apache 2.0+, NGINX 1.10+
• PHP 5.4+ (or PHP-FPM for NGINX)
• MySQL 5.6+
• cURL (for the cleanup action; see step 8)

Steps

1. Create an empty MySQL database and create a user with read and write access to it.

2. Copy config/sample-localSettings.php to config/localSettings.php
   Copy config/sample-localSettings.json to config/localSettings.json.
   Edit localSettings.json and replace the sample settings with your own.
   Refer to the Settings page for more information.

3. For Apache:
   Copy config/sample-.htaccess to .htaccess.
   To run TestSwarm from a non-root directory, set web.contextpath in localSettings.json to the correct path from the web root and update RewriteBase in .htaccess. Verify that .htaccess is working properly by opening a page other than the HomePage (e.g. /testswarm/projects) in your browser.
   Required Apache configuration:
   

   • AllowOverride is set to All (or ensure FileInfo is included).
   • mod_rewrite installed and loaded.

   For NGINX:
   Copy config/sample-nginx.conf to /etc/nginx/sites-available.
   The file name should match your domain e.g. for swarm.example.org:
   cp config/sample-nginx.conf /etc/nginx/sites-available/swarm.example.org.conf
   Open this conf file in your editor and replace the "example" values with the correct values.
   Make sure your install is located at /var/www/testswarm (otherwise update the file to match the correct location).
   Now you need to link the sites-available config to the sites-enabled config:
   (replace the "swarm.example.org" with your own file name):
   ln -s /etc/nginx/sites-available/swarm.example.org.conf /etc/nginx/sites-enabled/swarm.example.org.conf
   Now make sure that php-fpm is running: /etc/init.d/php-fpm status
   if is not running start it: /etc/init.d/php-fpm start

4. Copy config/sample-robots.txt to robots.txt
   Or, if TestSwarm is not in the root directory, add similar rules to your root robots.txt.

5. Set storage.cacheDir to a writable directory that is not readable from the web. Either set it to a custom path outside the document root, or use the default cache directory (protected with .htaccess).
   Chmod it: chmod 777 cache.

6. Install dependencies composer install --no-dev

7. Install the TestSwarm database by running: php scripts/install.php

8. Create an entry in your crontab for action=cleanup. This performs various cleaning duties such as making timed-out runs available again.
   * * * * * curl -s https://swarm.example.org/api.php?action=cleanup > /dev/null

9. Create a project and submit jobs.

Get involved

You're welcome to use the GitHub issue tracker to start discussions.

Some of us are also on Gitter at jquery/dev.

Documentation

• TestSwarm wiki
• Submit jobs README

Copyright and license

See LICENSE.txt.

Versioning

TestSwarm uses the Semantic Versioning guidelines as much as possible.

Releases will be numbered in the following format:

<major>.<minor>.<patch>

The -alpha suffix is used to indicate unreleased versions in development.

For more information on SemVer, please visit https://semver.org/.

History

TestSwarm was originally created by John Resig as a basic tool to support unit testing of the jQuery JavaScript library. It later become a Mozilla Labs project, and has since moved again to become a jQuery Foundation project.

testswarm's Issues

invalid default values for datetime fields in MySQL schema

The testswarm.sql script tries to create datetime columns with default value '0000-00-00 00:00:00'. However this is not a valid date. MySQL 5.0 which is configured in NO_ZERO_IN_DATE mode (or TRADITIONAL, for example), will not accept a zero date.

A workaround so far was to remove the entire "NOT NULL default '0000-00-00 00:00:00'" constraint for all datetime columns. Need more testing to see if this causes some surprises later on though.

Text overflow for IE for Windows Vista, XP active client list item

Screenshot here: http://skitch.com/miketaylr/b7yae/test-swarm-miketaylr

In Firefox 3.5, Safari 4.0, and Opera 10b3 (all on Mac), the "connected" information overflows (partially, or completely) out of its containing list item, when looking at a Windows OS and IE client.

Time connected

When I first registered today (account nickooms) what appeared on the page was :

nickooms
Active Clients:

Chrome 3.0b1 Windows 7
Connected 8 hours ago

I suspect that the 8 hours difference comes from the timezone because I'm located in Belgium

chrome 4.0.202.0 recognized as chrome 3.0b1

see title !

Firefox 3.6 Not Recognized

Firefox 3.6 is not recognized as being needed:

Browser is not needed for testing. Browser: gecko Version: 1.9.2

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (.NET CLR 3.5.30729)

Drop OS from useragents table, but save OS for runs

We don't want to enumerate each browser-version-os combination in the useragents table anymore. To reduce the list, drop the OS column and remove all the resulting duplicates.

In order to still see which OS a run has, move the column to the run_useragent table.

SRWare Iron Browser detected as safari

Actually sometimes im using SRWare Iron.
http://www.srware.net/en/software_srware_iron.php
It is based on the "Chromium" Sourcecode. So its actually pretty much Chrome and might want to be detected as such.

Add Opera Mobile

operamobile 2.5.28
Opera/9.80 (S60; SymbOS; Opera Mobi/751; U; en-GB) Presto/2.5.28 Version/10.1

Please add support to this one too. Thank you!

/config/testswarm.sql#105 auto_increment is incorrect

There are more useragents in the table than the auto_increment. You aren't inserting into the table yet, so this isn't a problem, but will be if you add an administration interface. It should be updated to 70 at the time of this writing and kept up to date when adding more useragents. I'd suggest a comment line on line #176 to remind people to do it, or better yet, a test.

Add Opera 11 and IE9 support

the title says it all

Status on running jobs does not parse HTML

The status does not parse the html when running a test

Didn't get a screen shot or was able to copy in time but I saw html like #6300 in the status.

Option to change the browserset via the job page

When an row is added to "useragents," jobs don't detect it because of the way that run_useragents works, which is reasonable. However, whenever I want test a new version of a browser, I have to rewrite the job, which can be a more work. I would recommend either having browsers automatically added to run_useragents if they fit under the criteria for a run, or create a clone button that creates a new copy of the job with the new user agent added, or create an update job button that adds the new user agent to the job without creating a new job or deleting an old.

I am most in favor of the last option.

Chrome Dev Channel v4 is not recognized

The badge for Chrome says 3b1 when I am using dev channel 4.0.203.2.

wiki, testswarm site, john's blog all point to the wrong github repository, wiki

This repository is located here:
https://github.com/jquery/testswarm

The swarm website is located here now: (if that's the only one...)
http://swarm.jquery.org/

And the links to these pages are wrong at http://swarm.jquery.org/, http://ejohn.org/blog/test-swarm-alpha-open/ and even on the wiki right next here on this repository.

Test Frame Sometimes Visible

My useragent for Firefox 3.5.3: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (.NET CLR 3.5.30729)

The testing frame is sometimes visible when the jQuery UI tests are run. The status contains escaped html:

Status: Running jQuery UI Commit #3268 Droppable tests...

And the frame appears below the history section. Here's a screenshot.
http://twitpic.com/ih8xk

Don't group all Chrome versions as one user agent entry

Currently we're ignoring all Chrome versions, which is fine on the assumption that everyone runs latest stable. Lots of web devs run Chrome dev channels, we should consider taking those into account.

User signup doesnt check for erroneous return result, users not created

With the local build Im running, the testswarm.sql doesn't have a default value for request in the users table, and the logic/signup.php insert statement doesnt pass in a request parameter.

The user ends up thinking their account is created successfully because they are redirected accordingly, but the insert never occurs. The return result of mysql_queryf needs to be checked.

If I run the insert statement manually (from line 23 of signup.php) I get the following error: ERROR 1364 (HY000): Field 'request' doesn't have a default value

Test overview page incorrect after adding tests

The test overview page (where it shows all the commits on a grid) displays incorrectly after adding a new entry.

I activated firefox 2.0 testing which added an extra column on the page. However, the old tests did not have extra entries for ff 2.0, which made all the cells be one off after where ff 2.0 should have been.

Hopefully that makes sense.

Tests that were run prior to the inclusion of the new test had a missing cell in the row, causing test cells to be displayed under the wrong column.

Detect database failure early and abort with friendly notice

While fixing MySQL and optimizing the code is more important, when the site can't connect to MySQL it needs to display an error message instead of attempting to load the page. When MySQL is down, I get the message that my browser is not supported for testing, and all the browsers listed on the home page are missing.

recognise Chrome on Linux

Chrome 3.0/4.0 for Linux is out there (in beta at least). Please add a user agent entry for it.

Chrome 4.0.266.0 on Linux gives me the following user agent triplet: "chrome 532.6 linux", and the user agent string is:

Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/532.6 (KHTML, like Gecko) Chrome/4.0.266.0 Safari/532.6

Document how to add jobs

i was searching for continues integration web gui tool and found this.
From reading all the articles and descriptions I overlooked how actually to create the tests.
Is there a recorder?
Does it utilizes selenium tests.
Should I write the tests ?
Or there is a gui when I install it.

content/home.php:89 is missing a question mark for the anchor title

for the title attribute, <php should be <?php
Fixed in http://github.com/takamori/testswarm
Although it's so trivial that you might just rather edit it directly.

Opera 10.20 wrong useragent

In config/useragents.sql there is an entry for Opera 10.20, I think this should be renamed to Opera 10.10 because there where no Opera stable releases with a 10.20 number (only one beta and one labs release).

The last stable release before Opera 10.50 was Opera 10.10 (check Opera version history).

The regex for the engine version can be left as is ('^2.2.15$')

Optionally allow any browser to run a test

I hate the idea of turning away someone who is willing to run the tests. And sometimes I don't have the specific browser/os combo that the test wants.

I'd like to come up with something a bit more flexible.
Track & report all the data people are willing to collect for us.

Allow the test creator to, instead of specifying which are the only browsers they're interested in, specify which browsers are required in order for the whole suite to pass.

Then in the report view, only show boxes for the few required browsers plus whatever other browsers have actually run the tests.

php config short_open_tag deprecated

Hi,

As seen in the discussion group, short open tags are used in php scripts.
As this property is now deprecated in php, it would be nice to avoid using it.

Thanks

Patch to enable testswarm in https environment

--- a/js/run.js
+++ b/js/run.js
@@ -54,7 +54,7 @@ function runTests( txt ) {
iframe.className = "test";
iframe.src = run_url + (run_url.indexOf("?") > -1 ? "&" : "?") +
"_=" + (new Date).getTime() + "&swarmURL=" +

•                   encodeURIComponent("http://" + window.location.host + "?" + params + "&state=");
  

•                   encodeURIComponent(window.location.protocol + "//" + window.location.host + "?" + params + "&state=");
          jQuery("#iframes").append( iframe );
  
          // Timeout after a period of time
  

Recognizes Chrome 3.0 as Safari 4.0

Google Chrome 3.0.195.32 is being recognized as Safari 4.0 as shown in images below:

http://i71.photobucket.com/albums/i157/mikecanfield/GoogleAquiresApple_1.png
http://i71.photobucket.com/albums/i157/mikecanfield/GoogleAquiresApple.png

Remove access to .gitignore in the .htaccess file

For web security, the following probably should be inserted into the root .htaccess

<Files ~ ".gitignore$">
Order allow,deny
Deny from all

Camino 1.5 UA not being recognized

I opened up a run page in Camino 1.5 on my Intel Mac running 10.5.8 but the icon that shows up is FF2.0.

-David

Recent jobs view incorrect after adding/removing tests

If you add/remove a test and then view the recent jobs, tasks that were previously run are no longer under the correct column.

Lock wait timeout exceeded

When trying to test I sometimes receive the following error:

Invalid query: Lock wait timeout exceeded; try restarting transaction

At the same time, other browsers may still work correctly.

This doesn't go away with a page refresh and stays broken for quite some time (sometimes days).

Job status and tinder should not assume first row is equal to others

When running testswarm and only having one test-module the browser icons does not appear above the results-table. Adding an empty dummy-module makes them appear.

Opera Mini incorrectly identified as Opera Desktop versions

Opera Mini browsers are incorrectly identified as Opera Desktop Browsers e.g.
Opera/9.80 (J2ME/MIDP; Opera Mini/5.0.19693/20.2463; U; en) Presto/2.5.255

This gets identified as Opera 10.5x. There should be a check added to inc/browser.php which sorts Opera Mini versions out (until supported) because they neither are Opera Mobile nor Opera Desktop and could behave differently

More info in Opera's User Agent String Documentation

Testswarm cannot run in a site subdirectory

When testswarm is used, it must be used as it's own domain. It would be nice if it could be used a sub-folder. The primary thing that prevents this is the relative links to css files, scripts, etc., which expect everything to be in a site's root.

Querying for more tests never finishes

Usually after leaving the test run page open for awhile, it will eventually query for a new test but not get any response back, so the page sits there doing nothing until it is manually refreshed. I'm sure this is related to site load but the page should either force refresh or open a new ajax call if the ajax query doesn't respond after a certain amount of time.

This has happened to me in every browser I've used including Firefox, IE, Safari and Chrome.

UA detection: Firefox 4 on Windows 7

(gecko 2.0 win7)

UA: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0) Gecko/20100101 Firefox/
4.0

IE 6 is stuck at the "Loading..." Phase

When I connect an instance of IE 6 to the test swarm system I have set up at my office, log-in, and "Join the Swarm", the page sits stuck at the "Loading" phase and never runs any tests.

Additionally, when I enter a username into the field on the home page for test swarm and click the button - it takes me to http://whatevermydomainis/?state=run&user=myusername but I am presented with a 404 page instead of the page from the above paragraph. (Testswarm is sitting behind an HTTP Digest authentication .htaccess, if this could have something to do with it)

user creation fails due to missing default value for "request" column

When signing up with the login form, or just entering a new user name on the front page, I get an error message:
Invalid query: Field 'request' doesn't have a default value

A workaround was to change the "users" table to allow NULL in the "request" column.

Adding jobs manually is broken

The logic scripts checks $_REQUEST['state'] instead of $_POST['state'], because of this it's impossible to access the form outputted by content/addjobs.php because logic/addjobs.php things it was submitted.

line 5 should instead be:

if ( $_POST['state'] == "addjob" ) {

Require log-in to run tests

If logged in, visiting http://testswarm.com/run/somelogin/ starts testing with my username no matter what's in the url.
But if I'm not logged in, visiting http://testswarm.com/run/somelogin/ also starts testing but with username "somelogin".
I was expecting not being able to run tests if not logged in.

Use $contextpath whenever URLS are used

Yet another place where $contextpath is not supported in logic/addjob.php, Location-header doesn't prefix $contextpath

I suggest creating a function that returns the path, and not hardcode it anywhere.
That way, you can't forget $contextpath so that I'm redirected to a 404-error, also, it'll make it future proof.

Another one:

Missing $contextpath in logic/run.php on line 7:
header("Location: /run/$user/");

EDIT:
More in logic/wipejob.php:

Please add {$GLOBALS['contextpath']} (or create a function like suggested) for line 34 and line 37
header("Location: /user/" . $_SESSION['username'] . "/");

header("Location: /job/$job_id/");

Cannot view scores for users with underscore in name

from logic/tinder.php
$search_user = preg_replace("/[^a-zA-Z0-9]/", "", $_REQUEST['user']);
only allows alphanumeric usernames, but signup does not prevent this.

Make Opera 10 Current

Replacing Opera 9.6.

QUnit.moduleDone logs bogus module at start

If module() is used from the beginning, there is one bogus log entry with that name of the last module and stats all at zero. If module() isn't used before the first test, those tests get grouped into a log entry, again with the name of the last module().

The issue lies in the fact that test() accesses config.currentModule for determining whether to run the current test or not. Therefore module() has to assign config.currentModule twice: Once before and once within the synchronizer. The check within the synchronized-module() code wrongly assumes that config.currentModule is undefined for the first module().

username href missing $contextpath variable

The top menu "username" link is missing the $contextpath variable

Server/client timezone difference should be accounted for

When I look at my user page, I see a history of runs. All these say: "Connected 8 hours ago" which is not true, since I connected less than 5 minutes "ago".

Android 2.3.2 matched as 2.2

Nexus S with Android 2.3.2 is matched as Android 2.2.

Here's the user agent:

Mozilla/5.0 (Linux; U; Android 2.3.2; en-us; Nexus S Build/GRH78C) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1

Opera 10.53

presto 2.5.24 xp

Opera/9.80 (Windows NT 5.1; U; en) Presto/2.5.24 Version/10.53

Option to replace previous job on addjob

Sometimes when adding a job I want to also delete the previous job for the same project or whatever.
e.g. I don't need 5 jobs for mootools trunk, when I add a new one, I want to delete the old one.

Documentation for crontab

Please update the crontab instructions to append to the crontab and not to blow it away.

From:
crontab < config/cronjob.txt

to:
crontab << config/cronjob.txt