mozilla-b2g / fxos-certsuite

DEPRECATED - Tests and tools to verify the functionality and characteristics of Firefox OS on real devices.

Firefox OS Certification Testsuite

Tests and tools to verify the functionality and characteristics of Firefox OS on real devices.

Before running the tests, make sure you read the full documentation at http://fxos-certsuite.readthedocs.org/ or in documentation.pdf found in this directory.

fxos-certsuite's Issues

Need better user experience for running WebIDL tests

The WebIDL tests run a long time, and it can appear that the tests are hung. We should give the user better feedback about what's going on; at a minimum, print a statement that tests are running and might take a few minutes.

Perhaps we'll end up chunking the WebIDL tests, in which case we can generate a status message for each chunk.

[Documentation] Better Documentation

The document are too out-of-date. We need to put more up-to-date information for developers, QAs, or users.

[Request] Requesting of privilege in fxos-certsuite

@dminor @jonallengriffin @ShakoHo @Conjuror, I think I need more privilege to do more works. For example, I need the power to mark, open/close, and handling more things.

How can I get more privilege to do this repository? Any ideas?

Webserver doesn't recognize changes to files

The webserver used for the webidl tests doesn't recognize changes to files made after it was started; it seems to serve a copy of the file as it existed at the time the web server was started.

The WebIDL harness writes index.html from index.in and another file. If index.html doesn't exist at the time the webserver is started (as it won't if you run the tests from a fresh clone), you'll get a "file not found" error for index.html when running the WebIDL tests.

If you add a dummy index.html so that the file exists (to assist discovery), when you run the WebIDL tests it will fetch the dummy version of index.html rather than the generated version.

To work around this problem in the short term, I've checked in a copy of the generated index.html.

Text extends beyond the prompt/instruct/confirm overlay

The semiauto TestCase.prompt/confirm/instruct overlay is not large enough for some of the text that is required during some tests. It would be nice if the overlay could grow dynamically if need be.

For example, run the semiauto test_screen_orientation test, and notice some of the instructions overflow beneath the boundary of the prompt.

Check in omni.ja

We should check in the same omni.ja that's used to generate the expected results, so that omni_diff.py always produces consistent diffs.

TestCase.prompt should clear the browser text input field first

Use TestCase.prompt twice in a row, and notice on the second prompt, the input field on the browser contains the text that was entered by the user for the previous prompt.

[Report] Add send report link in report.html

Try to launch mail client with title, body, and attachment and user just need add comment and click send.

CODE_OF_CONDUCT.md file missing

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

1. Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
2. Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please see Mozilla-GitHub-Standards or email [email protected].

(Message COC001)

Meta-harness should check for rooted device

We have a requirement that these tests are run on a rooted device, but we should test that, and generate a useful error message if the test is run on a non-rooted device.

[New Suite] Compliance Test

JavascriptException: TypeError: mm is undefined

Getting this a lot when running the minimal Web API tests:

test_sms_incoming_get_message (test_incoming_get_message.TestSmsIncomingGetMessage) ... ERROR

======================================================================
ERROR: None
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/ato/Dropbox/Code/fxos-certsuite/webapi_tests/webapi_venv/lib/python2.7/site-packages/marionette_client-0.7.1-py2.7.egg/marionette/marionette_test.py", line 143, in run
    testMethod()
  File "/Users/ato/Dropbox/Code/fxos-certsuite/webapi_tests/webapi_tests/sms/test_incoming_get_message.py", line 13, in test_sms_incoming_get_message
    self.user_guided_incoming_sms()
  File "/Users/ato/Dropbox/Code/fxos-certsuite/webapi_tests/webapi_venv/lib/python2.7/site-packages/webapi_tests-0.1-py2.7.egg/webapi_tests/sms/sms_test.py", line 26, in user_guided_incoming_sms
    """, special_powers=True)
  File "/Users/ato/Dropbox/Code/fxos-certsuite/webapi_tests/webapi_venv/lib/python2.7/site-packages/marionette_client-0.7.1-py2.7.egg/marionette/marionette.py", line 1073, in execute_async_script
    filename=os.path.basename(frame[0]))
  File "/Users/ato/Dropbox/Code/fxos-certsuite/webapi_tests/webapi_venv/lib/python2.7/site-packages/marionette_client-0.7.1-py2.7.egg/marionette/marionette.py", line 577, in _send_message
    self._handle_error(response)
  File "/Users/ato/Dropbox/Code/fxos-certsuite/webapi_tests/webapi_venv/lib/python2.7/site-packages/marionette_client-0.7.1-py2.7.egg/marionette/marionette.py", line 612, in _handle_error
    raise JavascriptException(message=message, status=status, stacktrace=stacktrace)
JavascriptException: TypeError: mm is undefined
    stacktrace:
    execute_async_script @sms_test.py, line 26
    inline javascript, line 315
    src: "        mm.getThreads();"
TEST-UNEXPECTED-FAIL | test_incoming_get_message.

[Restruct] Webapi suite

V2.1 pulled conflicted version

python setup.py install

Installed /home/wachen/FirefoxOS/fxos-certsuite2/.mcts_v210/lib/python2.7/site-packages/plivo-0.9.6-py2.7.egg
error: Installed distribution marionette-client 0.7.10 conflicts with requirement marionette-client>=0.16

pip install marionette-client==0.16
runcertsuite

Traceback (most recent call last):
File "/home/wachen/FirefoxOS/fxos-certsuite2/.mcts_v210/bin/runcertsuite", line 5, in
from pkg_resources import load_entry_point
File "/home/wachen/FirefoxOS/fxos-certsuite2/.mcts_v210/local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 2711, in
parse_requirements(requires), Environment()
File "/home/wachen/FirefoxOS/fxos-certsuite2/.mcts_v210/local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 584, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: marionette-client==0.7.10

There is something wrong with the dependency. We need to figure that out before it crashes on partners.

[Restruct] cert suite

TestCase.prompt must support up to 160 characters

As 160 characters is the max length of an SMS, and the outgoing SMS test will request the user to enter the SMS message text.

Add send report link in report.html

Try to launch mail client with title, body, and attachment and user just need add comment and click send.

Marionette Wait Class not in the webapi_venv

I recently added a Wait (Marionette Wait class) to the semiauto SMS tests, and am using Wait for the telephony tests that are under development. If I run the SMS tests outside of the webapi_venv then the run and pass fine:

/fxos-certsuite/webapi_tests/$ python -m semiauto discover sms

However, if I enable the webapi_venv first (either via 'source setup_ven.sh', or by running the tests with './setup_and_run.sh', then the tests fail with this:

(, ImportError('Failed to import test module: test_sms_incoming_delete\nTraceback (most recent call last):\n File "/usr/lib/python2.7/unittest/loader.py", line 252, in _find_tests\n module = self._get_module_from_name(name)\n File "/usr/lib/python2.7/unittest/loader.py", line 230, in _get_module_from_name\n import(name)\n File "/home/rwood/fxos-certsuite/webapi_tests/sms/test_sms_incoming_delete.py", line 8, in \n from sms import SmsTestCommon\n File "sms/init.py", line 5, in \n from sms_test import *\n File "sms/sms_test.py", line 7, in \n from marionette.wait import Wait\nImportError: No module named wait\n',), )

So looks like the Wait class is not included in the webapi_venv. When the semiauto tests are hooked up to the cert meta harness, they will fail since the Wait class cannot be found.

Should check to see if device and host are on some network

We currently have a requirement that the device being tested and the host machine are on the same network, but we should explicitly test that this is the case and error out if not. Since multiple pieces of this tool have this requirement, I think this should likely be done in the meta-harness.

Allow user to set a different port to run the webservers on

It turns out that certsuite/cert.py has a hardcoded port it runs on (8080). If that port is used, then we can't assign it to something else.

I ran into this issue when running ./run.sh. It may be the case that the user will have 8080 already being used (as was the case with me, since Jenkins runs on this port by default). We should make this configurable, or scan for an open port and use it, so it'll be fool-proof.

[Webapi] In orientation test, merge first similar behaviour case to 1

The first 4 cases in webapi:orientation is similar behaviour.
It's convenient to merge the 4 cases to simplify the test flow.

[Overall] Make test more automatic by 2 phones

Some tests can be run with 2 different phone concurrently controlled by pc.

[Set Top Box] MCTS support requested

In orientation test, merge first similar behaviour case to 1

The first 4 cases in webapi:orientation is similar behaviour.
It's convenient to merge the 4 cases to simplify the test flow.

[Fixes] Web platform tests fixes

Starting suite through run.sh stops with error

It appears we are not packaging the bundles directory:

Traceback (most recent call last):
File "/home/dminor/src/2fxos-certsuite/certsuite_venv/bin/runcertsuite", line 8, in
load_entry_point('fxos-certsuite==0.1', 'console_scripts', 'runcertsuite')()
File "/home/dminor/src/2fxos-certsuite/certsuite_venv/local/lib/python2.7/site-packages/fxos_certsuite-0.1-py2.7.egg/certsuite/harness.py", line 116, in main
install_marionette(args.version)
File "/home/dminor/src/2fxos-certsuite/certsuite_venv/local/lib/python2.7/site-packages/fxos_certsuite-0.1-py2.7.egg/certsuite/harness.py", line 100, in install_marionette
cwd=script_dir)
File "/usr/lib/python2.7/subprocess.py", line 535, in check_call
retcode = call(_popenargs, *_kwargs)
File "/usr/lib/python2.7/subprocess.py", line 522, in call
return Popen(_popenargs, *_kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 709, in init
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1326, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory: '/home/dminor/src/2fxos-certsuite/certsuite_venv/local/lib/python2.7/site-packages/fxos_certsuite-0.1-py2.7.egg/bundles/1.3'

Add total executions count at bottom of the summary report table

Test output should contain the package version number

The test output should contain the package version number, e.g., 1.3.0.3. This will help us interpret results, and tell us what version an OEM is running when they submit results.

[New Suite] Performance Suite

[Restruct] security suite

[New Suite] MCTS Presentation API (W3C SPEC) for TV

As a user, I want to have presentation API in v2.5 TV released B2G verified. It should be before November.

[Overall] Virtual Environment or Docker solution

Need the ability for the user to provide info for some WebAPI tests

Need the ability to pass info into the WebAPI tests. For example, for testing outgoing SMS or outgoing phone calls, we need a destination mobile number from the test user. Perhaps we should have a testvars file similar to how the gaia-ui tests work, which is provided on the command line?

Move to using fxos_appgen

The app files added in #52 can be removed once we move to using fxos_appgen. By using the package to generate the app, we won't have to manually update any of these files, or add new ones when we move to 1.4 and higher.

Result file should contain the fxos-certsuite version number

...so we can determine which version was used when investigating bug reports, etc.