mozilla / voicefill

A WebExtension To Add Speech To Text Support to Web Pages

Voice Fill

This is a simple WebExtension that adds support to use Speech To Text as an input method in web pages.

Usage

Install the web-ext npm module: npm install --global web-ext and run web-ext run from the extension subdirectory.

voicefill's Issues

The microphone button doesn't remain inside the search input while scrolling on some websites

[Affected versions]:

• Release 54.0.1
• Beta 55.0b6

[Affected Platforms]:

• All Windows
• All Mac
• All Linux

[Prerequisites]:

• Have a Firefox profile with the latest Voice Fill version (1.0 -dev, built on 05/07/2017) installed.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites.
2. Navigate to https://us.yahoo.com/
3. Scroll down and up with different speed using middle click.
4. Observe the microphone button.

[Expected result]:

• The microphone button remains in the same position inside the search input.

[Actual results]:

• The microphone button doesn't keep up with the speed of scrolling and it's placed outside the search input.

[Notes]:

• This issue is also reproducible on other websites (https://twitter.com/, https://www.aliexpress.com/)
• Here is a screencast of the issue:

Save voice sample and submitted text with a method respecting a user's privacy

When saving the voice sample:

• Ensure the voice data is not associated with any person-identifiers, including Firefox or Test Pilot identifiers
• Don't send the voice sample to two places
• No requirement to delete within a time frame

/cc @eleemoz @johngruen

@andrenatal : Can you sketch a diagram of how this works? Kaldi can probably save a sample, but we need to do a separate ping for what was submitted? let's talk! Doesn't need to be a diagram necessarily, just need a plan for this.

handle close event

The overlay is wrongly triggered even if the microphone permission is blocked and remembered

[Affected versions]:

• Release 54.0.1
• Beta 55.0b10

[Affected Platforms]:

• All Windows
• Al Linux
• All Mac

[Prerequisites]:

• Have a Firefox profile with the latest Voice Fill version (1.2.1 -dev built on 07/20/2017) installed.
• Tools -> Page Info -> Permissions -> Use the Microphone -> "Use Default" is unchecked and "Block" is checked.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites and navigate to www.google.com
2. Click on the microphone button and observe what happens next.

[Expected result]:

• Nothing happens, the overlay is not triggered.
  OR
• A notification is displayed stating that there is no permission to access the microphone.

[Actual results]:

• The overlay is triggered and displayed over the website's content area.

[Notes]:

• This issue is also reproducible on https://duckduckgo.com/
• Here is a screen recording with the issue:

"Submit" tooltip is misspelled and doesn't start with capital letter

[Notes]:

• Both "submit" and "reset" tooltips should start with capital letters.

[Affected versions]:

• Release 54.0.1
• Beta 55.0b10

[Affected Platforms]:

• All Windows
• All Linux
• All Mac

[Prerequisites]:

• Have a Firefox profile with the latest Voice Fill version (1.2.1 -dev built on 07/20/2017) installed.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites and navigate to www.google.com
2. Click on the microphone button and on "Allow" option from the door hanger permission.
3. Say one word.
4. Hover the "Submit" button and observe its tooltip.

[Expected result]:

• The "Submit" tooltip is correctly displayed, without misspelling, starting with capital letter.

[Actual result]:

• "sumbit" is displayed instead of "Submit" and it doesn't start with capital letter.

[Additional notes]:

• The issue is also reproducible on https://duckduckgo.com/
• Added a screenshot with the issue:

Add Acceptance Criteria Checklist to Repository

I'll take this one. We need to add a checklist to confirm all UI implementation

Tracking Bug for UX Assets

@fangshih, please add a link for all icons from your visual spec. Icons should be either SVG or Retina (@2x) pngs.

Disable add-on functionality on non-ssl sites

We don't want to run code or display the microphone on sites that aren't served over SSL

Voice fill overlay is not dismissed if the microphone permission is not granted

[Affected versions]:

• Release 54.0.1
• Beta 55.0b10

[Affected Platforms]:

• All Windows
• All Linux
• All Mac

[Prerequisites]:

• Have a Firefox profile with the latest Voice Fill version (1.2.1 -dev built on 07/20/2017) installed.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites and navigate to www.google.com
2. Click on the microphone button.
3. Click on "Don't Allow" option from the permission door hanger and observe what happens next.

[Expected result]:

• The Voice fill overlay is dismissed.

[Actual results]:

• The Voice fill overlay is still displayed on website's content area.

[Notes]:

• The overlay is not dismissed even by clicking the "close" button on the top right side of it.
• This issue is also reproducible on https://duckduckgo.com/
• Here is a screen recording with the issue:

Add Voice Fill UI to popular SERPs

Implemen Voice Fill on Google, DDG, Yahoo

create a installable docker image

Website's toolbar doesn't cover the microphone button if the page is scrolled down

[Affected versions]:

• Release 54.0.1
• Beta 55.0b6

[Affected Platforms]:

• All Windows
• All Linux
• All Mac

[Prerequisites]:

• Have a Firefox profile with the latest Voice Fill version (1.0 -dev, built on 06/07/2017) installed.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites and navigate to https://faq.whatsapp.com/
2. Click inside the search input.
3. Scroll down until the search input is covered by the website's toolbar.
4. Observe the microphone button's behavior.

[Expected result]:

• The microphone button is covered by the website's toolbar.

[Actual results]:

• The microphone button overlaps the website's toolbar content and text.

[Notes]:

• Here is a screencast of the issue:

Unable to search on apple.com

$ git rev-parse --short HEAD # 28cc402

Steps to reproduce:

1. npm run build
2. npm run once
3. Go to https://apple.com
4. Click the search icon in the top toolbar.
5. Try and click the Speak-To-Me icon.

Actual results:

I don't seem to be able to click the speak-to-me icon in the text input field in order to grant microphone access and do a voice search.

Expected results:

Able to search by voice.

Investigate ESLint errors

Looks like a potential scoping issue w/ SpeakToMeVad. I think it's defined in an IIFE, but then accessed later outside of that scope in some module code.

Slightly modified npm run lint:js output is as follows (minus the no-console warnings):

$ npm run lint:js

> [email protected] lint:js /Users/pdehaan/dev/github/mozilla/speaktome
> eslint extension

/Users/pdehaan/dev/github/mozilla/speaktome/extension/content.js
  101:5   warning  Unexpected 'todo' comment            no-warning-comments
  142:1   warning  Unexpected 'todo' comment            no-warning-comments
  177:18  error    Expected '!==' and instead saw '!='  eqeqeq
  181:17  error    Expected '!==' and instead saw '!='  eqeqeq
  285:37  error    Expected '===' and instead saw '=='  eqeqeq
  298:33  error    Expected '===' and instead saw '=='  eqeqeq
  318:58  error    Expected '!==' and instead saw '!='  eqeqeq
  323:21  error    Expected '===' and instead saw '=='  eqeqeq
  324:9   warning  Unexpected 'todo' comment            no-warning-comments
  349:1   error    'SpeakToMeVad' is not defined        no-undef
  427:25  error    Expected '===' and instead saw '=='  eqeqeq
  498:5   warning  Unexpected 'todo' comment            no-warning-comments
  508:19  error    'SpeakToMeVad' is not defined        no-undef

✖ 13 problems (9 errors, 4 warnings)

Ref: #22

Issue with the microphone that keeps trembling on DDG

Microphone button is wrongly overlapped by websites toolbars

[Affected versions]:

• Release 54.0.1
• Beta 55.0b6

[Affected Platforms]:

• All Windows
• All Mac
• All Linux

[Prerequisites]:

• Have a Firefox profile with the latest Voice Fill version (1.0 -dev, built on 05/07/2017) installed.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites.
2. Navigate to www.youtube.com
3. Observe the search input.
4. Using middle click scroll down and up the website very fast.
5. Observe the website's content area under the toolbar's search input.

[Expected result]:
3. The microphone button is displayed inside the search input.
5. The microphone button remains inside the search box and the content area is correctly displayed.

[Actual results]:
3. The microphone button is not displayed inside the search input.
5. The microphone button can be observed under the toolbar's search input, over the website's content area.

[Notes]:

• This issue is also reproducible on www.msn.com
• Here is a screen recording with the issue (since the issue is hard to notice because of the website's content, I zoomed out the website for a better view of this bug):

"q", "y", "p", "g" and "j" letters are wrongly displayed while editing the multiple answers input

[Notes]:

• I have also noticed that the input can be scrolled if you press middle click and move the mouse up/down.

[Affected versions]:

• Release 54.0.1
• Beta 55.0b10

[Affected Platforms]:

• All Windows
• All Linux
• All Mac

[Prerequisites]:

• Have a Firefox profile with the latest Voice Fill version (1.2.1 -dev built on 07/20/2017) installed.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites and navigate to www.google.com
2. Click on the microphone button and on "Allow" option from the door hanger permission.
3. Say one word.
4. Type the following letters in the multiple answers input: p, q, y, j and g.
5. Observe the way the typed letters are displayed inside the field.

[Expected result]:

• The typed letters are correctly displayed and fully visible.

[Actual result]:

• The typed letters are truncated.

[Additional notes]:

• Although on DuckDuckGo the issue doesn't seem to be as visible as on Google, the input can also be scrolled up and down.
• Added a screen recording with the issue:

The microphone button overlaps the text if a long string is written inside the search input

[Affected versions]:

• Release 54.0.1
• Beta 55.0b6

[Affected Platforms]:

• All Windows
• All Linux
• All Mac

[Prerequisites]:

• Have a Firefox profile with the latest Voice Fill version (1.0 -dev, built on 06/07/2017) installed.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites and navigate to www.google.com
2. Start typing a long string until it exceeds the input's limit.
3. Observe the way the search input items are rendered.

[Expected result]:

• The microphone button is placed in a way that doesn't overlap other UI elements nor blocks other search input buttons.

[Actual results]:

• The microphone button overlaps the written text.

[Notes]:

• Here is a screenshot of the issue:

Make sure correct sites work correctly

I think i kinda messed up a few of the required buttons. I'll dive into that code tomorrow!

Microphone button is wrongly displayed outside the email input on the Pinterest "Log in" page

[Affected versions]:

• Release 54.0.1
• Beta 55.0b6

[Affected Platforms]:

• All Windows
• All Linux
• All Mac

[Prerequisites]:

• Have a Firefox profile with the latest Voice Fill version (1.0 -dev, built on 06/07/2017) installed.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites and navigate to https://www.pinterest.com/
2. Click on the "Log in" button on top right of the screen.
3. Observe the microphone's button location on the page.

[Expected result]:

• The microphone button is displayed inside the email or phone input.

[Actual results]:

• The microphone button is displayed outside the email or phone input, overlapping the "Log in" button.

[Notes]:

• On "Welcome" page, the microphone button is correctly displayed inside the email input.
• Here is a screencast of the issue:

Load testing for v1

We're talking about having Stage available on July 11th for load testing. This is a really tight squeeze for launching on July 18th. This issue is to track load testing.

/cc @oremj @milescrabill @jvehent

Provide Animation State Information / Assets Voice Overlay

@fangshih, we need to provide the team with an animation spec for speak to me. Let's touch base tomorrow to see how best to convert your ideas into code.

Ensure that metrics are not sent if DNT is enabled.

You can detect this in a background page by inspecting navigator.doNotTrack.

Microphone button is moved outside the search input if clicked on Imgur

[Affected versions]:

• Release 54.0.1
• Beta 55.0b6

[Affected Platforms]:

• All Windows
• All Mac
• All Linux

[Prerequisites]:

• Have a Firefox profile with the latest Voice Fill version (1.0 -dev, built on 05/07/2017) installed.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites.
2. Navigate to www.imgur.com
3. Click on the magnifying glass search button.
4. Click on the microphone button and observe what happens next.

[Expected result]:

• Permission door hanger to access the microphone is triggered.

[Actual results]:

• The door hanger is not triggered and the microphone button is moved outside the search input, to the right side of the website's area.

[Notes]:

• Here is a screencast of the issue:

Create basic CLI test

@andrenatal,

How would we test this from the CLI (for load testing)?

I found this fetch() call which seems to POST a blob to the stage server.

Is there an easy we can capture some Blobs from the client side (or somewhere) so we can send them through to the proxy server to get us started w/ the load testing?

Once we have this working from the CLI, it should hopefully be "easy" to convert to a Python script and then run it at scale on AWS to do some load testing on the proxy server to see how many incoming connections/blobs it can process before 🔥.

UsageError: Extension exists at the destination path

$ git rev-parse --short HEAD # 28cc402

---

Looks like the npm run build fails if the extension is already built at ./web-ext-artifacts/.
We should either remove that directory prior to running npm run build (using rm -rf, or rimraf module), or apparently just add the --overwrite-dest flag on web-ext build -s extension:

$ npm run build

> [email protected] build /Users/pdehaan/dev/github/mozilla/speaktome
> web-ext build -s extension

Building web extension from /Users/pdehaan/dev/github/mozilla/speaktome/extension

build: UsageError: Extension exists at the destination path: /Users/pdehaan/dev/github/mozilla/speaktome/web-ext-artifacts/speak-to-me-1.0.zip
Use --overwrite-dest to enable overwriting.

The microphone button overlaps other search input special buttons on some websites

[Affected versions]:

• Release 54.0.1
• Beta 55.0b6

[Affected Platforms]:

• All Windows
• All Mac
• All Linux

[Prerequisites]:

• Have a Firefox profile with the latest Voice Fill version (1.0 -dev, built on 05/07/2017) installed.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites.
2. Navigate to www.wikipedia.org
3. Observe the search input items.

[Expected result]:

• The microphone button is placed in a way that doesn't overlap other UI elements nor blocks other search input buttons.

[Actual result]:

• The microphone button overlaps a part of the locales switcher button.

[Notes]:

• This is also reproducible on other websites:

1. http://www.jd.com/ - the special search button is not clickable
2. https://yandex.ru/ - the keyboard button is overlapped by the microphone button

• Here is a screencast of the issue:

Ensure Opus processing is sandboxed

@oremj is working on a solution to this

Check permissions issues when installing testing xpi

make recording animation match spec

Implement Context Click Triggers for Speech Input

Add voice input controls via context click on inputs with type=text. If the we think the algorithm is even remotely up this this we might try to add this control to textareas as well (as shown in mock up below).

Only ever display the microphone icon in a top level document

Meaning, disable functionality within frames

Hook Voice Fill up to Test Pilot

1. We need to make this repository match the spec at https://github.com/mozilla/testpilot/blob/master/docs/development/hosting.md

2. @relud to add this repository to the watcher

Verify microphone div is the highest z-index

Ensure POSTs have a size limit and a request rate limit

I think these can be set in nginx, so I'm ccing @jasonthomas

I don't have recommendations for values here.

Implement Voice Fill control on about:newtab and about:home

Add a voice control button to about:newtab and about:home page search boxes.

This button should have the following features:

• It should match the provided visual spec
• It if the user clicks it, it should trigger the Voice Fill UI
• If the user types in the search field; the voice search button should revert to the default UI

Please review the Main Interface view for UI measurements:
https://www.dropbox.com/sh/iy20ednfyxhsj3l/AACABj4tCdqMySL6UDUqNxlKa?dl=0
note: you may have to download the Dropbox folder to see the measurement spec, we're working on a hosted solution

The microphone button is not displayed inside the search input on the main page of Bugzilla

[Affected versions]:

• Release 54.0.1
• Beta 55.0b6

[Affected Platforms]:

• All Windows
• All Mac
• All Linux

[Prerequisites]:

• Have a Firefox profile with the latest Voice Fill version (1.0 -dev, built on 05/07/2017) installed.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites.
2. Navigate to https://bugzilla.mozilla.org/
3. Click inside either of the 2 search inputs and observe what happens next.

[Expected result]:

• The microphone button is displayed inside the search input.

[Actual results]:

• The microphone button is not displayed inside the search input.

[Notes]:

• On the other pages of Bugzilla the microphone button is properly displayed.
• Here is a screen cast of the issue:

Microphone button is wrongly displayed outside the search input on Nightly

[Affected versions]:

• Nightly 56.0a1

[Affected Platforms]:

• All Windows
• All Mac
• All Linux

[Prerequisites]:

• Have a Firefox profile with the latest Voice Fill version (1.0 -dev, built on 05/07/2017) installed.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites.
2. Navigate to https://duckduckgo.com/
3. Observe the way the microphone button is displayed.

[Expected result]:

• The microphone button is displayed inside the search input.

[Actual results]:

• The microphone button is displayed outside the search input, in different places of the website's content area.

[Notes]:

• On http://www.ebay.com/ the microphone button is not visible nor reachable because the website's notification overlaps it.
• This is also reproducible on the following websites:

1. http://www.bing.com/ - microphone button is displayed on the bottom left of the screen
2. https://www.google.com - microphone button is displayed on the top left of the screen
3. https://www.apple.com/ - microphone button is displayed on the bottom left of the screen

• Here is a screen recording with the issue:

[Nightly] The "Microphone" button is wrongly displayed under the search input

[Notes]:

• This is a regression that is present for now only in Nightly and it was introduced by Bugzilla issue 1334550.
• Here is the regression range:
  Last good revision: ab348cba6c56cf0dfb4acaccee1bfa262ef3b2d8
  First bad revision: 3358bef7a047d347e576bbb83d1461b3776fc5fb
• Pushlog:
  https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=ab348cba6c56cf0dfb4acaccee1bfa262ef3b2d8&tochange=3358bef7a047d347e576bbb83d1461b3776fc5fb

[Affected versions]:

• Nightly 56.0a1

[Affected Platforms]:

• All Windows
• All Linux
• All Mac

[Prerequisites]:

• Have a Firefox profile with the latest Voice Fill version (1.2.2 -dev built on 07/21/2017) installed.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites and navigate to www.google.com
2. Observe the "Microphone" button and its location on the website.

[Expected result]:

• A white microphone with a turquoise background circle is displayed on the right side of the search input.

[Actual result]:

• There is only a small rectangle button that is displayed on the bottom left of the search input.

[Additional notes]:

• "Warming up..." and "Feedback" text are wrongly displayed on the top left side of the screen.
• The issue is also reproducible on https://duckduckgo.com/
• Added a screenshot with the issue:

Store the query results in a relational database

handle reset event

Check the magic number of opus files before processing them

according to the rfc, it's 0x4F70757348656164 ("OpusHead")

Don't hardcode the version number in .travis.yml

The build is broken right now because https://github.com/mozilla/speaktome/blob/master/.travis.yml#L8 and line 13 both have a version number in the file name but the version is increasing

Voice Fill doesn't display the result of recording and becomes unresponsive

[Affected versions]:

• Release 54.0.1
• Beta 55.0b6

[Affected Platforms]:

• All Windows
• All Mac
• All Linux

[Prerequisites]:

• Have a Firefox profile with the latest Voice Fill version (1.0 -dev, built on 06/07/2017) installed.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites and navigate to a search engine website (e.g. www.google.com).
2. Click on the microphone button.
3. Click on "Allow" and say a word.
4. Wait for ~6 seconds and observe what happens next.

[Expected result]:

• The spoken word is displayed as text.

[Actual results]:

• No text is displayed and the add-on becomes unresponsive.

[Notes]:

• The unresponsive dialog box remains blocked in the same place if the page is scrollable.
• The dialog box shrinks if clicked, but remains in the same unresponsive state.
• Here is a screencast of the issue:

Add correct updateURL to add-on

The updateURL will be:

https://testpilot.firefox.com/files/%ID%/updates.json

With the %ID% being the ID of the add-on

Implement the GA metrics

As defined at https://github.com/mozilla/speaktome/blob/master/docs/metrics.md

There is no description of the Voice Fill/Speak To Me add-on in "about:addons" page

[Affected versions]:

• Release 54.0.1 and up

[Affected Platforms]:

• All Windows
• All Mac
• All Linux

[Prerequisites]:

• Have a Firefox profile with the latest Voice Fill version (1.0 -dev, built on 05/07/2017) installed.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites.
2. Navigate to "about:addons" page.
3. Click on "Extensions" option from the left sidebar.
4. Observe the way the "speak-to-me" add-on is described.

[Expected result]:

• A text containing a brief description of the add-on is displayed under the add-on title.

[Actual results]:

• There is no text description of the add-on.

Clicking one of the multiple answers wrongly starts the search process

[Affected versions]:

• Release 54.0.1
• Beta 55.0b10

[Affected Platforms]:

• All Windows
• All Linux
• All Mac

[Prerequisites]:

• Have a Firefox profile with the latest Voice Fill version (1.2.1 -dev built on 07/20/2017) installed.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites and navigate to www.google.com
2. Click on the microphone button and on "Allow" option from the door hanger permission.
3. Say one word.
4. Click on one of the multiple answers from the drop down and observe what happens.

[Expected result]:

• The clicked answer is displayed in the input having a green background and the following message is displayed at the bottom of the overlay: "Your edits help us train the engine".

[Actual result]:

• The search starts automatically.

[Additional notes]:

• This is also reproducible on https://duckduckgo.com/
• Added a screen recording with the issue:

Fail gracefully

In case the server responds status code > 500, or when the response is not a json, and etc.. we should fail gracefully an display an error message.

Implement Voice Input Overlay

See #16 for full set of criteria

For measurements, please review the view for UI measurements:
https://www.dropbox.com/sh/iy20ednfyxhsj3l/AACABj4tCdqMySL6UDUqNxlKa?dl=0
note: you may have to download the Dropbox folder to see the measurement spec, we're working on a hosted solution