mozilla / esper-pioneer-shield-study

The ESPER experiment - Firefox Pioneer Study

The ESPER Experiment - Firefox Pioneer Study

Evaluating Similarity of Pioneers as Exemplars of Release

Assess the degree and sense in which users in opt-in cohort of Firefox Pioneer differ from the Firefox release channel population. This should be a one time collection of data that focuses on fields for which we have aggregate statistics pertaining to the firefox release population in Telemetry.

For more information, see the ESPER Product Hypothesis Doc

Getting started

See TESTPLAN.md for more details on how to get the add-on installed and tested.

Analysing Data

Telemetry ping payloads are encrypted and not decrypted until it is on a server that is not connected to the wider internet. See TELEMETRY.md for more details.

Bugzilla

• Initial launch
• Re-launch

esper-pioneer-shield-study's Issues

Gather feedback on possible reasons for the add-on failing to trigger a payload send event

Background:
We are seeing considerably fewer pings ending up in our study's parquet table than expected compared to the Normandy recipe's DAU estimates. Now we are exploring possible sources to this discrepancy.

This issue is created to gather feedback on possible reasons for the add-on failing to trigger a payload send event.

After the add-on is installed by Normandy, the bootstrap.js startup method is the first to fire up.

Possible reasons gathered so far:
a. The add-on yields fatal errors in the lines before the eligibility check, where Pioneer utils is initiated (of such nature that they were not seen during development nor QA)
b. The add-on's double-checking of Normandy's eligibility targets are not in sync with the Normandy recipe used: (normandy.addons[‘[email protected]’].isActive && normandy.version >= 59.0)
c. Normandy installs does not set the installation reason to one of install or upgrade (https://github.com/motin/esper-pioneer-shield-study/blob/1.0.4/addon/bootstrap.js#L62)

@pdehaan, @rhelmer, @rehandalal: can you spot any other potential reasons, stemming from the add-on implementation / pioneer-utils usage etc?

Thought experiment: What should be in a study-specific readme?

Claim:

• description of goal
• list of files
• list of variations
• links to TELEMETRY and TESTPLAN
• bug number? Maybe this should be in the description as well?
• build / run instructions?

Probably not:

• how to contribute?

Other thoughts welcome... we are trying to pave this path.

• claim should name / id of the study have both pioneer and shield in them? @esper-pioneer-shield-mozilla? This makes regex searching by id have a hope of succeeding :)

Implement ESPER-specific probing/pings

PHD: https://docs.google.com/document/d/1y3MSF_azfrAoKdj00_GEhC_o8sGetEuLLnloYBpVtNo/edit?ts=59ccadd2&pli=1

Relevant part:

Appendix (telemetry fields to be collected):
Meta
geo_country, geo_city, normalized_channel, app_version

Profile
profile_creation_date, profile_subsession_counter, subsession_start_date, timezone_offset

System
os, os_version, system.memory_mb, system_cpu.cores, system_cpu.speed_mhz, system_gfx.monitors[1].screen_width

Simple Measures
uptime, total_time, default_search_engine, locale

Histograms
places_bookmarks_count, places_pages_count, search_counts

Scalars
Scalar_parent_browser_engagement_max_concurrent_tab_count, scalar_parent_browser_engagement_max_concurrent_window_count, scalar_parent_browser_engagement_navigation_about_newtab, scalar_parent_browser_engagement_navigation_contextmenu, scalar_parent_browser_engagement_navigation_searchbar,
scalar_parent_browser_engagement_navigation_urlbar,
scalar_parent_browser_engagement_tab_open_event_count,
scalar_parent_browser_engagement_total_uri_count, scalar_parent_browser_engagement_unfiltered_uri_count,
Scalar_parent_browser_engagement_unique_domains_count,
scalar_parent_browser_engagement_window_open_event_count

Here is the WIP doc mapping required stmo.mozilla.org indicators and actual add-on probing: https://docs.google.com/spreadsheets/d/1COo7Cl0kMuEhAD_X25jQH2XP0owYX1gXdpbWvMmsnLw/edit#gid=0

It's more idiomatic to load/unload JSMs via resource:// not chrome://

From https://bugzilla.mozilla.org/show_bug.cgi?id=1450951#c17

FYI, chrome.manifest is going away in Firefox 61, so you'll want to move away from that and register the protocol handler at runtime instead (if you're doing webextension experiment APIs then you probably already know this, since it's the only way to do it)

Parameter 'aChannel' uses Hungarian Notation, consider using 'channel' instead (mozilla/no-aArgs)

$ npm run eslint

> [email protected] eslint /Users/pdehaan/dev/github/mozilla/esper-pioneer-shield-study
> eslint . --ext=jsm,js,json

/Users/pdehaan/dev/github/mozilla/esper-pioneer-shield-study/addon/lib/StudyTelemetryCollector.jsm
  218:38  warning  Parameter 'aChannel' uses Hungarian Notation, consider using 'channel' instead  mozilla/no-aArgs

Looks like the warning is coming from https://github.com/motin/esper-pioneer-shield-study/blob/552d2c7f6aef2a31c426e1c7e8082215936bca22/addon/lib/StudyTelemetryCollector.jsm#L218-L223

Looks like the easy fix is to add an // eslint-disable-next-line style comment to ignore that one rule for that one specific line (or, rename function arguments...):

// eslint-disable-next-line mozilla/no-aArgs
function getParentProcessScalars(aChannel, aKeyed = false, aClear = false) {
  ...
}

Suppress informative and debugging messages

From https://bugzilla.mozilla.org/show_bug.cgi?id=1450951#c17

Will be taken care of in conjunction with a conversion of this add-on to web extensions. Requires that mozilla/shield-studies-addon-utils#139 is resolved.

Default to TRUE when reading this pref and then test for preference FALSE states

https://github.com/motin/esper-pioneer-shield-study/blob/3f571c90faa8d76cb0eeb78d234e758cf0aedf64/addon/lib/StudyTelemetryCollector.jsm#L94-L108

Here the preference absence and preference set to false will similarly evaluate to false. This is an opt-out preference, but it is not present in every language of Firefox, so the current test will will return False for all non-english fx installs.

The same logic should be achieved testing for false states of the preference explicitly.

Values of the scalar_parent_browser_engagement_navigation fields do not change even if it is tested with different scenarios

Send an install event directly upon study add-on installation

(From: #26 (comment))

Send an install event directly upon study add-on installation so that we can confirm how many installed add-ons are able to send pioneer event telemetry (EVENTS.INSTALL) (Depends on mozilla/pioneer-utils#41)

Remove use of the duplicated ping attribute "uptime"

See https://bugzilla.mozilla.org/show_bug.cgi?id=1485695:

The main ping is reporting both "uptime" and "totalTime" (the are the same value, with different units) [1]. All the derived datasets are using "totalTime", and no derived dataset is using "uptime". "totalTime" was meant to replace "uptime".

Since we don't need "uptime", let's remove it.

In the study add-on, we already collect both "uptime" and "totalTime", so we can safely remove "uptime" from the add-on and adjust corresponding data queries as necessary, should we want to re-run the experiment in the future.

Change the id to end in @pioneer.mozilla.org

The official docs on how to ship a native shield study (mentioned and linked in mozilla/shield-studies-addon-utils#93 (comment)) states "Note: All Shield Study add-on ids must end in @shield.mozilla.org - Pioneer studies must end in @pioneer.mozilla.org"

Linter for Telemetry schema

Not sure what [if anything] the ./schemas/esper-study-telemetry.1.schema.json and ./schemas/esper-study-telemetry.1.sample.json files are for, but I wrote a quick ajv schema validator for the data to make sure everything was in sync:

const Ajv = require('ajv');
const metaSchema = require('ajv/lib/refs/json-schema-draft-04.json');

const data = require('../schemas/esper-study-telemetry.1.sample.json');
const schema = require('../schemas/esper-study-telemetry.1.schema.json');

const ajv = new Ajv({
  allErrors: true,
});
ajv.addMetaSchema(metaSchema);

const validate = ajv.compile(schema);

if (!validate(data)) {
  const txt = ajv.errorsText(validate.errors);
  console.error(txt.split(", ").join("\n"));
  process.exitCode = 1;
}

Feel free to close this. I was just posting it because it took me a bit of searching to figure out why the v4 schema was throwing the following error:

Error: no schema with key or ref "http://json-schema.org/draft-04/schema#"

CODE_OF_CONDUCT.md file missing

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

1. Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
2. Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please see Mozilla-GitHub-Standards or email [email protected].

(Message COC001)

package.json and LICENSE file list different licenses

$ cat package.json | grep "license" # "license": "ISC",
# --or--
$ jq -r ".license" package.json # ISC
# --or--
$ node -e "console.log(require('./package.json').license)" # ISC

$ head -1 LICENSE # Mozilla Public License Version 2.0

Error running addons-linter

"Path 'addon/webextension/' is not a file or directory or does not exist."

➜  esper-pioneer-shield-study git:(develop) npm run lint

> [email protected] lint /Users/pdehaan/dev/github/mozilla/esper-pioneer-shield-study
> npm-run-all lint:*


> [email protected] lint:addons-linter /Users/pdehaan/dev/github/mozilla/esper-pioneer-shield-study
> addons-linter addon/webextension/

Path "addon/webextension/" is not a file or directory or does not exist.
(node:77047) UnhandledPromiseRejectionWarning: Error: Path "addon/webextension/" is not a file or directory or does not exist.
    at Linter.checkFileExists (/Users/pdehaan/dev/github/mozilla/esper-pioneer-shield-study/node_modules/addons-linter/dist/webpack:/src/linter.js:277:28)
    at /Users/pdehaan/dev/github/mozilla/esper-pioneer-shield-study/node_modules/addons-linter/dist/webpack:/src/linter.js:396:21
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)
    at Function.Module.runMain (module.js:695:11)
    at startup (bootstrap_node.js:188:16)
    at bootstrap_node.js:609:3
(node:77047) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:77047) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

...

Not sure if we should just tweak package.json to the generic "addon" instead of "addon/webextension":

"lint:addons-linter": "addons-linter addon"

---

UPDATE: After the path tweak, I still get the following output:

$ npm run lint:addons-linter

> [email protected] lint:addons-linter /Users/pdehaan/dev/github/mozilla/esper-pioneer-shield-study
> addons-linter addon

Validation Summary:

errors          0
notices         1
warnings        0

NOTICES:

Code                    Message                 Description                                        File   Line   Column
TYPE_NO_MANIFEST_JSON   manifest.json was not   A manifest.json in the root of the extension was
                        found                   not found. See: https://mzl.la/2r2McKv for more
                                                on packaging

Audit unused dependencies

Seems like we have a few unused dependencies which we can ✂️ to save a bit of space and time.

Currently running $ npm install added 945 packages in 43.483s.

Running $ npx depcheck points to a few [potentially] unused dependencies...

$ npx depcheck
Unused devDependencies
* clipboardy
* doctoc
* eslint-config-airbnb-base
* eslint-plugin-import
* eslint-plugin-json
* eslint-plugin-mozilla
* eslint-plugin-no-unsanitized
* get-firefox
* jshint
* moustache
* pioneer-utils
Missing dependencies
*  ajv

... but looks like we can safely remove at least the following:

diff --git a/package.json b/package.json
index 0b2b379..84ec391 100644
--- a/package.json
+++ b/package.json
@@ -22,10 +22,7 @@
   "devDependencies": {
     "addons-linter": "^0.28.2",
     "ajv": "^5.3.0",
-    "clipboardy": "^1.1.4",
-    "doctoc": "^1.3.0",
     "eslint": "^4.10.0",
-    "eslint-config-airbnb-base": "^11.2.0",
     "eslint-plugin-import": "^2.8.0",
     "eslint-plugin-json": "^1.2.0",
     "eslint-plugin-mozilla": "^0.4.5",
@@ -35,16 +32,15 @@
     "fx-runner": "^1.0.8",
     "geckodriver": "^1.9.0",
     "get-firefox": "^2.0.0",
-    "jshint": "^2.9.5",
     "minimist": "^1.2.0",
     "mocha": "^3.4.2",
     "moustache": "0.0.2",
     "npm-run-all": "^4.1.1",
     "nsp": "^2.8.1",
     "onchange": "^3.2.1",
+    "pioneer-utils": "^1.0.10",
     "prettier": "^1.10.2",
-    "selenium-webdriver": "^3.5.0",
-    "pioneer-utils": "^1.0.10"
+    "selenium-webdriver": "^3.5.0"
   },
   "homepage": "https://github.com/motin/esper-pioneer-shield-study",
   "keywords": [

After nuking my node_modules/** and package-lock.json from high orbit ($ rm -rf node_modules package-lock.json) and reinstalling the modules ($ npm i), I now have a saner looking list of potentially unused modules:

$ npx depcheck

Unused devDependencies
* eslint-plugin-import
* eslint-plugin-json
* eslint-plugin-mozilla
* eslint-plugin-no-unsanitized
* get-firefox
* moustache
* pioneer-utils

1. The ESLint ones are probably all inaccurate, due to how ESLint loads plugins and configs (non using traditional require() statements).
2. get-firefox seems to be used in the .circleci/config.yml file and referenced in the test/Dockerfile.
3. moustache is used for generating templates in bin/xpi.sh.
4. pioneer-utils is used in package.json where the file is manually copied into the addon/ directory.

Add more try/catch blocks and let caught errors end the study

(From: #26 (comment))

Add more try/catch blocks and let caught errors end the study (EVENTS.ENDED_NEGATIVE - or EVENTS.ENDED_ERROR if mozilla/pioneer-utils#42 gets implemented)

Await endStudy()

(From: #26 (comment))

Await endStudy() (So that endStudy events get sent more reliably) - (depends on mozilla/pioneer-utils#40)

Clean up promises / async / await with regards to collectPlacesDbBasedAttributes

From https://bugzilla.mozilla.org/show_bug.cgi?id=1450951#c17

StudyTelemetryCollector has a static async method collectPlacesDbBasedAttributes that returns a new Promise... I'm not sure why wrapping in the Promise is necessary here (async functions always wrap the return/throw in a Promise resolve/reject), and I noticed too that the async function that the single caller of this is in does a .then() instead of an await

Wire up circle-ci for linting and tests on each PR?

Looks like the stubs are there, but not sure if it's worth enabling to make sure any future PRs don't break linting or web-ext or tests...

But since it's a short lived Pioneer experiment, it may not be worth it. 🤷‍♀️

Unable to load addon on Firefox 57.0b14 (64-bit) [Beta channel current version]

Error received when following specified testplan on Firefox 57.0b14 (64-bit)
On Mac OS X Sierra 10.12.6

Make sure this addon complies with Pioneer ToS

Specifically we should be using pioneer-utils to ensure that all the data is encrypted as per this line of the ToS:

"The data you submit is encrypted in Firefox and not decrypted until it is on a server that is not connected to the wider internet."

End the study after the ordinary payload is submitted

(From: #26 (comment))

End the study after the ordinary payload is submitted (EVENTS.ENDED_POSITIVE)

The addon still sends data even the extended telemetry pref is unchecked from about:preferences#privacy

Get the add-on ready for QA

Checklist before submitting for QA:

• PR Reviewed by Gregg @gregglind
• Bonus: PR Reviewed by Alessio @Dexterp37 (thanks! :) )
• Confirmation that ESPER pings ends up in Redash
• Tested by me
• Tested by Gregg
• Tested by Martin
• Tested by Alessio
• Confirmation from Gregg that the shield utils ensures that study install event only fires once, so that the esper pings will only be sent once (otherwise: Make sure manually (using add-on-specific config) that the esper pings will only be sent once)

"Tested" here refers to following the test plan (Note that it is in the prepare-for-qa branch)

Expire the study if no payload was submitted within 30 min

(From: #26 (comment))

Expire the study if no payload was submitted within 30 min (EVENTS.EXPIRED)