mozilla / preparatory-survey-for-naar-study-addon

Preparatory Survey for NAAR Study Add-on

The purpose of this extension is to prompt users to participate in the preparatory survey for NAAR.

Shipped as a study add-on since the survey requires metadata about some of the add-ons that the user has installed.

Seeing the add-on in action

See TESTPLAN.md for more details on how to get the add-on installed and tested.

Data Collected / Telemetry Pings

See TELEMETRY.md for more details on what pings are sent by this add-on.

Analyzing data

Telemetry pings are loaded into S3 and re:dash. Sample query:

• All pings

Improving this add-on

See DEV.md for more details on how to work with this add-on as a developer.

References

• Experimenter

preparatory-survey-for-naar-study-addon's Issues

The "Preparatory Survey for NAAR" experiment is wrongly displayed as active in "about:studies" page after it is removed

[Notes]:

• If the page is refreshed, the "Preparatory Survey for NAAR" experiment will be displayed in the "Completed studies" section.

[Affected versions]:

• Firefox 68.0b14 unbranded

[Affected Platforms]:

• All Windows
• All Mac
• All Linux

[Prerequisites]:

• Have a recipe created for the experiment on the Normandy stage server (link to recipe).
• Have the latest Firefox 68.0b14 unbranded build installed.
• Have at least 3 add-ons installed from AMO.
• Have the following prefs set in “about-config” page:
  • The "xpinstall.signatures.dev-root" boolean pref with the "true" value.
  • The “app.normandy.dev_mode preference” to “true” to run recipes immediately on startup.
  • The “app.normandy.logging.level preference” to “0” to enable more logging.
  • The “security.content.signature.root_hash” preference to “DB:74:CE:58:E4:F9:D0:9E:E0:42:36:BE:6C:C5:C4:F6:6A:E7:74:7D:C0:21:42:7A:03:BC:2F:57:0C:8B:9B:90”.
  • The “services.settings.server” to “https://normandy.stage.mozaws.net/api/v1”.
  • The “extensions.shield-recipe-client.api_url” to “https://stage.normandy.nonprod.cloudops.mozgcp.net/api/v1”.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites.
2. Navigate to the "about:studies" page.
3. Click the "Remove" button for the “Preparatory Survey for NAAR” experiment and observe the behavior.

[Expected result]:

• The experiment is instantly moved to the "Completed studies" section.

[Actual result]:

• The experiment is still displayed in the "Active studies" section.

[Additional Notes]:

• The following error message is displayed in the Browser's Console: "Error: Cannot stop study for recipe 781; it is already inactive.".
• Nothing happens if the "Remove" button is clicked multiple times, the experiment remains in the "Active studies" section.
• However, the notification bar is no longer displayed after clicking the "Remove" button and the correct telemetry pings are sent.
• This issue was not reproducible for other add-on experiments we tested.
• Attached a screen recording with the issue:
  

Firefox Peer Review

This study add-on includes a web extension experiment that simulates a Heartbeat prompt, based on Heartbeat.jsm (see expected UX and functionality here)
@mythmon Would you mind peer reviewing this study add-on? Here is a commit with all the add-on code and tests for the review: dc47d20

[Survey] The telemetry disclaimer was not shown

Since this survey received the client id parameter, it should display "Your survey responses may be associated with your Telemetry data" but it currently doesn't.

[Survey] The "Mozilla Privacy" wrongly opens in the same tab after clicking the "Mozilla Privacy Policy" link

[Notes]:

• If the browser's "Back" button is clicked after the "Mozilla Privacy" website is opened from the second or third survey page, the "Document Expired" page is displayed. However, if the page is refreshed or the "Try again" button is clicked, the survey page is redisplayed.

[Affected versions]:

• Preparatory Survey for NAAR 1.0.0

[Affected Platforms]:

• All Windows
• All Mac
• All Linux

[Prerequisites]:

• Have the latest Firefox 68 unbranded build installed.
• Have at least 3 add-ons installed from AMO.
• Have the "xpinstall.signatures.dev-root" boolean pref created with the "true" value.
• Have the "xpinstall.signatures.required" boolean pref set to "false".
• Have the Preparatory Survey for NAAR 1.0.0 signed xpi installed.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites.
2. Click the "Take me to the questionnaire" button from the displayed Heartbeat-styled notification bar.
3. Click the "Mozilla Privacy Policy" link from the second page displayed.
4. Observe the behavior.

[Expected result]:

• The "Mozilla Privacy" website is opened in a new tab.

[Actual result]:

• The "Mozilla Privacy" website is opened in the same tab.

[Additional Notes]:

• Attached a screen recording with the issue:
  

[Survey] The telemetry disclaimer is not displayed on the second and third page of the survey

[Notes]:

• If the "Back" button is clicked when the user is on the second page of the survey, the telemetry disclaimer is no longer displayed on the first page as well.

[Affected versions]:

• Preparatory Survey for NAAR 1.0.1

[Affected Platforms]:

• All Windows
• All Mac
• All Linux

[Prerequisites]:

• Have the latest Firefox 68.0b14 unbranded build installed.
• Have at least 3 add-ons installed from AMO.
• Have the "xpinstall.signatures.dev-root" boolean pref created with the "true" value.
• Have the "xpinstall.signatures.required" boolean pref set to "false".

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites and install the Preparatory Survey for NAAR 1.0.1 signed build.
2. Click the "Take me to the questionnaire" button from the displayed Heartbeat-styled notification bar.
3. Complete the required questions with valid data for each add-on.
4. Click the "Next" button and observe the bottom of the page.

[Expected result]:

• The telemetry disclaimer is displayed under the "Mozilla Privacy Policy" section.

[Actual result]:

• The telemetry disclaimer is not displayed.

[Additional Notes]:

• Attached a screen recording with the issue:
  

UX Review

This study add-on includes a Heartbeat-styled UI element:

Additional information about the expected interaction and the survey is available in https://github.com/mozilla/preparatory-survey-for-naar-study-addon/blob/master/docs/TESTPLAN.md#expected-user-experience--functionality

Please review that the UX elements in this add-on conforms with the current guidelines, thanks.

The "Preparatory Survey for NAAR" add-on is not uninstalled after the study ends

[Notes]:

• The issue is reproducible in the following scenarios:
  • The survey is taken.
  • The Heartbeat-styled notification bar is dismissed.
  • The Heartbeat-styled notification bar is ignored and the browser is restarted twice.
  • The Heartbeat-styled notification bar is ignored for more than 1 day.
• After each Firefox restart, the Preparatory Survey for NAAR extension runs again and the Heartbeat-styled notification bar is redisplayed, even if the study ended.
• The "ended-neutral/expired" and "exit" telemetry pings are correctly sent after the study ends.

[Affected Versions]:

• Preparatory Survey for NAAR 1.0.0

[Affected Platforms]

• All Windows
• All Mac
• All Linux

[Prerequisites]

• Have the latest Firefox 68 unbranded build installed.
• Have at least 3 add-ons installed from AMO.
• Have the "shieldStudy.logLevel" string pref created with "info" value.
• Have the "xpinstall.signatures.dev-root" boolean pref created with the "true" value.
• Have the "xpinstall.signatures.required" boolean pref set to "false".
• Have the Preparatory Survey for NAAR 1.0.0 signed xpi installed.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites.
2. Click the "Take me to the questionnaire" button from the displayed Heartbeat-styled notification bar.
3. Complete and submit the survey.
4. Restart the browser and navigate to "about:support" page.
5. Observe the "Extensions" section.

[Expected results]:

• The Preparatory Survey for NAAR extension is uninstalled and is no longer listed.

[Actual results]:

• The Preparatory Survey for NAAR extension is still installed and the Heartbeat-styled notification bar is redisplayed.

[Additional Notes]:

• Attached a screen recording with the issue.