mozilla / mozmill-ci

[OBSOLETE] Mozmill CI system driven by Mozilla Pulse and Jenkins

Mozmill CI

With the Mozmill CI project the Firefox Automation team aims to keep running a continuous integration system for Mozmill tests written for Firefox. As core system Jenkins is used, and gets enhanced by adding configurations, plugins, and what-ever necessary.

Installation and startup

Mozmill CI can easily be set-up. Please consult our installation instructions for details.

Once done you can follow the instructions to start Jenkins and if necessary the build listeners.

Development

In case you want to help us in further developing Mozmill CI, you might wanna be interested in the details for developers.

Troubleshooting

In case of problems running Mozmill-CI our Troubleshooting page gives necessary hints to get problems solved.

General information

For any other topic in how to work with Jenkins please have a look at our wiki. If you have questions please get in contact with us.

mozmill-ci's Issues

Cache job requests if Jenkins or nodes are not available

Whenever Jenkins is not available (outtage, maintanance...) we should cache job requests until Jenkins is back. We should define a maximum threshold until data keeps valid. Same applies to nodes with the given label.

ESR version numbers are stripped from on-demand configuration files

It appears that the 'esr' suffix from version numbers is being stripped from the configuration files when triggering on-demand jobs.

Add email notifications for Releases

Now that the automation runs automagically when Beta and RC candidates are available (via Pulse), it would be great to get notification of what is happening so we don't have to continually spotcheck Jenkins and Brasstacks. Here is what I am suggesting:

1. email notification to the QA Desktop team when a functional testrun has been initiated
   "Mozmill-CI functional testrun has been started for Firefox 12.0b6#1"

2. email notification to the QA Desktop team when a functional testrun has been completed
   "Mozmill-CI functional testrun has finished for Firefox 12.0b6#1 with N failures"

I think a dependency for this might be getting a "[email protected]" email alias set up for the team. If I have support for this idea then I will go ahead and file the appropriate bugzilla request.

Update to xshell 0.8

Once this version has been made available we should update:
https://wiki.jenkins-ci.org/display/JENKINS/XShell+Plugin

Add mozilla-beta builds to daily testruns

In the foreseeable future we will get daily builds on the beta channel. That means we should also add those to our matrix of builds to test.

Clone workspaces via the "Clone Workspace SCM" setting

Right now we are using the copy artifact plugin together with the clone workspace plugin to copy workspaces for other jobs. We should only make use of the clone workspace SCM setting in the future. But as long as the following issues in the plugin haven't been fixed, we will have to wait.

https://issues.jenkins-ci.org/browse/JENKINS-6890
https://issues.jenkins-ci.org/browse/JENKINS-9250

mozilla-aurora_functional is trying to copy artifacts from tools

The mozill-aurora_functional jobs is still trying to copy artifacts from the tools job, which no longer creates artifacts.

get_mozmill-environments should use temporary files for download

Given that we run jobs across platforms on nodes, the execution can fail if we download new environments on the master. To prevent this as best as possible get_mozmill-environments should download the env to a temporary file and rename it right after. So there will be only some milliseconds overlap instead of seconds.

Include release-mozilla-release for testing

We should include the branch as given by Pulse messages to test the candidate builds of releases. We have to wait for issue #40 to be fixed first.

Create docs based on Sphinx

I have read this article (http://blog.johnford.org/generating-an-epub-copy-of-the-python-docs/) yesterday and I think we should create our documentation based on Sphinx. See also the documentation from MozTrap: http://moztrap.readthedocs.org/en/latest/

Remove queued items if older than a day

We are not interested in running tests against builds which are more than a day old. It would even be good to not cancel by time but when the next request of the same type comes in. Not sure if that can be done but we should check that.

Bind Jenkins to a specifc version

Right now we haven't specified a version of Jenkins we support for the CI project. We should do so to be able to:

• Download the correct build
• Config settings apply correctly
• Installed plugins working correctly

Builds are being triggered when values are empty

It appears that we have had some builds triggered by pulse when certain values such as version and buildid are empty. See log/mozilla-central/firefox_win32_en-US_2012-05-18T09:40:15+01:00.log and http://release3.qa.mtv1.mozilla.com:8080/job/mozilla-central_update/1232/console as an example.

Make latest version of hg repositories available as artifacts

Cloning the hg repository for the automation scripts and tests each time is not a solution we should continue. It's way broken due to the flakiness of this website. I don't want to see that our system cannot run Mozmill tests when hg.mozilla.org is down. So we should cache the latest version of both repositories locally and make them available as artifacts for all nodes.

We could use a changegroup hook (http://mercurial.selenic.com/wiki/Hook#The_changegroup_hook), use Pulse notifications from the HGConsumer, or simply pull the repository for changes in intervals like 5 minutes.

@davehunt what would you suggest? I'm up for polling solution.

Pin the dependency versions in setup/requirements.txt

We should specify the versions in the requirements.txt so that we can assure compatibility.

Create branch specific testrun jobs

We need the testrun jobs to be branch specific because different versions of Firefox are really different products. Those can throw a different list of failures which should not be combined with other versions.

Add platform version to email notification

Right now we can only see 'win32' in teh email notification for all windows platforms. We should include the platform version so we can see more detailed failures. Same applies to Mac and Linux. Shall we simply include the node labels instead of the platform?

Dave can you work on this?

Fix job descriptions

The jobs currently have descriptions like:

Execute addon tests for Nighly and Aurora builds.

The testrun type differs, but Nighly (sp) and Aurora are not appropriate to all builds. It would be good to have a sensible description for all jobs.

Pulse consumer for release builds is broken

See the following exception thrown:

INFO:automation:2012-04-18T23:15:36+01:00 - Product: None, Branch: releases/comm-beta, Platform: None, Locale: en-US
Traceback (most recent call last):
File "./pulse.py", line 266, in
main()
File "./pulse.py", line 262, in main
message=options.message)
File "./pulse.py", line 234, in init
Automation.init(self, _args, *_kwargs)
File "./pulse.py", line 84, in init
pulse.listen()
File "/Users/mozilla/mozmill-ci/jenkins-env/lib/python2.6/site-packages/mozillapulse/consumers.py", line 136, in listen
self.consumer.wait()
File "/Users/mozilla/mozmill-ci/jenkins-env/lib/python2.6/site-packages/carrot/messaging.py", line 446, in wait
it.next()
File "/Users/mozilla/mozmill-ci/jenkins-env/lib/python2.6/site-packages/carrot/backends/pyamqplib.py", line 300, in consume
self.channel.wait()
File "/Users/mozilla/mozmill-ci/jenkins-env/lib/python2.6/site-packages/amqplib/client_0_8/abstract_channel.py", line 97, in wait
return self.dispatch_method(method_sig, args, content)
File "/Users/mozilla/mozmill-ci/jenkins-env/lib/python2.6/site-packages/amqplib/client_0_8/abstract_channel.py", line 117, in dispatch_method
return amqp_method(self, args, content)
File "/Users/mozilla/mozmill-ci/jenkins-env/lib/python2.6/site-packages/amqplib/client_0_8/channel.py", line 2060, in _basic_deliver
func(msg)
File "/Users/mozilla/mozmill-ci/jenkins-env/lib/python2.6/site-packages/carrot/messaging.py", line 291, in _receive_callback
self.receive(message.payload, message)
File "/Users/mozilla/mozmill-ci/jenkins-env/lib/python2.6/site-packages/carrot/messaging.py", line 344, in receive
callback(message_data, message)
File "./pulse.py", line 184, in on_build
filename = os.path.join(self.log_folder, branch, basename)
File "/Users/mozilla/mozmill-ci/jenkins-env/bin/../lib/python2.6/posixpath.py", line 65, in join
if b.startswith('/'):
AttributeError: 'NoneType' object has no attribute 'startswith'

I will have to analyze the saved Pulse notifications why this happens.

Jobs are hanging in "Recording test results"

The tests are working fine but most of the nodes are hanging here:

*** Removing old installation at /var/folders/yc/yc-yt4kPHweMwoUftOHQgE+++TQ/-Tmp-/tmplvW9wc.binary/FirefoxAurora.app
*** Removing repository '/var/folders/yc/yc-yt4kPHweMwoUftOHQgE+++TQ/-Tmp-/tmpzjIyBo.mozmill-tests'
Build step 'Invoke XShell command' marked build as failure
Recording test results

They can't finish the recording of test results. I could imagine because that's of the change to '**/results_??.xml. So for now I will revert that. But we should figure out why.

@davehunt have you ever seen that?

Make type of build configurable by parameter

To allow us to run the jobs for different kind of builds the type has to be configurable as parameter.

All win32 Pulse triggered 13.0b4 testruns failed

Console Output:
Started by user anonymous
Building remotely on windows-7_64_01 in workspace c:\jenkins\workspace\release-mozilla-beta_functional

Deleting project workspace... done

Unable to access upstream artifacts area. Does source project archive artifacts?
Build step 'Copy artifacts from another project' marked build as failure
Recording test results
Finished: FAILURE

---

I'm not sure what happened but all the Linux and Mac 13.0b4#1 builds ran okay automatically. For some reason all the win32 builds fell down. I'm now triggering an ondemand testrun to see if it reproduces outside of CI.

If Jenkins is not running script should handle the exception and not exit

Traceback (most recent call last):
File "./pulse.py", line 229, in
main()
File "./pulse.py", line 217, in main
handle_notification(data, None)
File "./pulse.py", line 158, in handle_notification
'REPORT_URL': config['tests']['report_url']})
File "/Volumes/data/code/mozmill-ci/jenkins-env/lib/python2.7/site-packages/jenkins/init.py", line 332, in build_job
if not self.job_exists(name):
File "/Volumes/data/code/mozmill-ci/jenkins-env/lib/python2.7/site-packages/jenkins/init.py", line 267, in job_exists
self.get_job_info(name)
File "/Volumes/data/code/mozmill-ci/jenkins-env/lib/python2.7/site-packages/jenkins/init.py", line 150, in get_job_info
response = self.jenkins_open(urllib2.Request(self.server + JOB_INFO%locals()))
File "/Volumes/data/code/mozmill-ci/jenkins-env/lib/python2.7/site-packages/jenkins/init.py", line 174, in jenkins_open
return urllib2.urlopen(req).read()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 394, in open
response = self._open(req, data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 412, in _open
'_open', req)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 372, in _call_chain
result = func(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1199, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1174, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno 61] Connection refused>

Treat ondemand builds as a higher priority than pulse triggered builds

We will need to run the ondemand jobs as a higher priority than pulse triggered jobs.

Add testrun coverage per branch

Not on all branches we want to run all of the available testruns. We have to make it configurable.

Pulse consumer timed out in amqplib

Probably an external issue but filing for now to have it listed.

Traceback (most recent call last):
File "./pulse.py", line 265, in
main()
File "./pulse.py", line 261, in main
message=options.message)
File "./pulse.py", line 233, in init
Automation.init(self, _args, *_kwargs)
File "./pulse.py", line 85, in init
pulse.listen()
File "/Users/mozilla/mozmill-ci/jenkins-env/lib/python2.6/site-packages/mozillapulse/consumers.py", line 136, in listen
self.consumer.wait()
File "/Users/mozilla/mozmill-ci/jenkins-env/lib/python2.6/site-packages/carrot/messaging.py", line 446, in wait
it.next()
File "/Users/mozilla/mozmill-ci/jenkins-env/lib/python2.6/site-packages/carrot/backends/pyamqplib.py", line 300, in consume
self.channel.wait()
File "/Users/mozilla/mozmill-ci/jenkins-env/lib/python2.6/site-packages/amqplib/client_0_8/abstract_channel.py", line 95, in wait
self.channel_id, allowed_methods)
File "/Users/mozilla/mozmill-ci/jenkins-env/lib/python2.6/site-packages/amqplib/client_0_8/connection.py", line 202, in _wait_method
self.method_reader.read_method()
File "/Users/mozilla/mozmill-ci/jenkins-env/lib/python2.6/site-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method
raise m
socket.error: [Errno 60] Operation timed out

Add platform coverage per branch

Each listed branch should have its own list of supported platforms. It's necessary because Nightly builds of Firefox 13 no longer run on Windows 2000.

Update Readme with Release Automation instructions

Now that we have mozmill-ondemand working in Jenkins, we have a simpler process for testing releases with the automation. I am going to update the Readme.md file to include instructions about running the ondemand tests.

Add details to the execution reason for jobs

Currently jobs simply state 'Started by anonymous user'. It would be great if we could include details of how and why it was triggered.

Include release-mozilla-beta for testing

We should include the branch as given by Pulse messages to test the candidate builds of beta releases.

Add documentation for initial setup

Lets write some stuff how to setup the system. I will cover it today.

Hard-kill a running process after a given amount of time

I had an update test which was trying to download the build for over 1 week now and has not been killed. We should force a hard-kill after a given amount of time.

Disable Windows 2000 Jenkins node

Firefox 12 is our last release where we will support Windows 2000. As a result, any bugs we happen to find on Windows 2000 will be resolved WONTFIX. Therefore, I see no point to keep the Windows 2000 node alive either for CI or Ondemand testing.

Disable [Started] emails notifications

The new email notifications are great but are bit "noisy". I don't think we need emails for [Started], only for "finished" conditions. Thanks.

Integrate DTC automation with CI

Product teams are looking to be able to have nightly Beta builds instead of weekly builds (target is May 17th). One blindspot that creates in QA testing is with add-ons and plugins. It seems like the Default to Compatible automation would fit the bill for filling that blindspot. Can we set up the DTC automation to run in Mozmill-CI on a daily basis?

Here is what we would want:

• Grab the previous Beta
• Start with any number of top add-ons installed
• Run some set of automated tests
• Update to the latest Beta
• Re-run the test to make sure nothing has broken (in particular we would look for crashes)

It seems like DTC automation would lend itself well to this type of testing and if we can run this in CI, even better.

Now I know that the DTC automation takes a long time to run, partly because it runs against all locales. Could we restrict the CI integration to en-US for the time being?

Update Clone Workspace plugin once JENKINS-13165 is fixed

Once pull #60 is merged we will be on Jenkins version 1.455 (because a change in 1.456 causes JENKINS-13165). Once this pull for the Clone Workspace plugin has been merged and released, we need to update to the latest version of the plugin, which will require Jenkins 1.463 or later.

Jobs are broken on Windows (VirtualAlloc pointer is null)

Not sure why but we fail to run Cygwin on the PoC. Could be due to lesser memory but I will have to check. Here the failure message:

[mozilla-central_update] $ cmd.exe /C '"mozmill-env\run hg clone https://hg.mozilla.org/qa/mozmill-automation && exit %%ERRORLEVEL%%"'
0 [main] us 0 init_cheap: VirtualAlloc pointer is null, Win32 error 487
AllocationBase 0x0, BaseAddress 0x71110000, RegionSize 0x480000, State 0x10000
c:\jenkins\workspace\mozilla-central_update\mozmill-env\msys\bin\bash.exe: *** Couldn't reserve space for cygwin's heap, Win32 error 487

Only archive mozmill-automation workspace on successful builds

Currently the get-mozmill_automation workspace is configured to be archived on every completed build. We should update this so it is only archived on successful builds.

Add IRC notifications for failures

Notify #automation (irc.mozilla.org) when builds fail.

Mercurial version should be 2.2.1

I suspect there was a typo in the requirements file. It looks like Mercurial 2.1 causes a build to fail when pulling and there are no updates. Upgrading to 2.2.1 fixes this. It's already 2.2.1 locally and on qa-masterblaster.

--debug doesn't save messages anymore

Here the exception I get:

File "./pulse.py", line 116, in init
pulse.listen()
File "/Users/qa/mozmill-ci/jenkins-env/lib/python2.7/site-packages/mozillapulse/consumers.py", line 136, in listen
self.consumer.wait()
File "/Users/qa/mozmill-ci/jenkins-env/lib/python2.7/site-packages/carrot/messaging.py", line 446, in wait
it.next()
File "/Users/qa/mozmill-ci/jenkins-env/lib/python2.7/site-packages/carrot/backends/pyamqplib.py", line 300, in consume
self.channel.wait()
File "/Users/qa/mozmill-ci/jenkins-env/lib/python2.7/site-packages/amqplib/client_0_8/abstract_channel.py", line 97, in wait
return self.dispatch_method(method_sig, args, content)
File "/Users/qa/mozmill-ci/jenkins-env/lib/python2.7/site-packages/amqplib/client_0_8/abstract_channel.py", line 117, in dispatch_method
return amqp_method(self, args, content)
File "/Users/qa/mozmill-ci/jenkins-env/lib/python2.7/site-packages/amqplib/client_0_8/channel.py", line 2060, in _basic_deliver
func(msg)
File "/Users/qa/mozmill-ci/jenkins-env/lib/python2.7/site-packages/carrot/messaging.py", line 291, in _receive_callback
self.receive(message.payload, message)
File "/Users/qa/mozmill-ci/jenkins-env/lib/python2.7/site-packages/carrot/messaging.py", line 344, in receive
callback(message_data, message)
File "./pulse.py", line 203, in on_build
JSONFile(filename).write(data)
File "./pulse.py", line 65, in init
raise NotFoundException('Specified file cannot be found.', filename)

Could be related to my changes from pull request #5

Windows XP slave offline

Not sure why, but the Windows XP SP2 slave appears to be offline:
http://release3.qa.mtv1.mozilla.com:8080/computer/windows-xp_32_01/

Archive cfg file for ondemand triggered tests as artifact

To be able to retrospect a triggered ondemand testrun we will have to archive the uploaded config file. Otherwise there is no way to check the parameters which have been specified for further debugging.

@davehunt mind having a look at it?

Add timestamp to pulse log message

It's hard to identify the latest log files right now. So lets add a timestamp as prefix.

Improve body layout of email notifications

Right now the layout is a bit wonky and links hard to read. So we should improve it.

Here is what we have right now:

Mozmill mozilla-central_update testrun for Firefox Nightly 20120519030527 mac en-US completed with 0 failures.

View the build in Jenkins: http://qa-masterblaster.mv.mozilla.com:8080/job/mozilla-central_update/1260/
View the results in the Mozmill Dashboard: http://mozmill-ci.blargon7.com/#/update/report/f87375a634b1a5ba746e5f763a40383b

Something which would make it way better is:

Mozmill mozilla-central_update testrun for Firefox Nightly 20120519030527 mac en-US completed with 0 failures.

View the build in Jenkins:
    http://qa-masterblaster.mv.mozilla.com:8080/job/mozilla-central_update/1260/

View the results in the Mozmill Dashboard:
    http://mozmill-ci.blargon7.com/#/update/report/f87375a634b1a5ba746e5f763a40383b
    http://mozmill-ci.blargon7.com/#/update/report/f87375a634b1a5ba746e5f763a407f7f

Means each link should be in it's own line for better readability.

Implement notifications for when nodes go offline

There's a plugin that will notify by email when a node goes offline. We should implement this so that we can act faster when there's downtime. https://wiki.jenkins-ci.org/display/JENKINS/Node+Offline+Notification+Plugin

Allow jobs to specify a version of Mozmill

Currently only one version of Mozmill is supported (determined by a global variable). If we changed this so that version was an axis of the get_mozmill-environments job then we should be able to specify a version in the testrun job. We should use latest by default.

Include l10n testrun for localized tinderbox builds

The following pull will add the code which is necessary to test localized builds on mozilla-aurora which are getting triggered by check-ins of localizers.

Don't run tests if builds are invalid

If we don't obey this we will have a broken update path and wrong results:
http://mozmill-ci.blargon7.com/#/update/report/e438d6e3916b2b636037d77445b2e310

Unzip the mozmill-env archive on the slave

Right now we have to unzip the downloaded mozmill-environment archive on master because on Windows the unzip command is not available. It takes way more time to transfer all the files to slaves so we should get this fixed.