mozilla / addons

☂ Umbrella repository for Mozilla Addons ✨

Mozilla Add-ons

This is the general root project for add-ons at Mozilla.

Filing bugs

If you're looking to file a bug relating to the addons.mozilla.org (AMO) website then please file an issue on this project.

Security Bug Reports

The associated production website at addons.mozilla.org is included in Mozilla’s web and services bug bounty program. If you find a security vulnerability, please submit it via the process outlined in the program and FAQ pages. Further technical details about this application are available from the Bug Bounty Onramp page.

Please submit all security-related bugs through Bugzilla using the web security bug form.

Never submit security-related bugs through a Github Issue or by email.

Documentation

This project contains general documentation relating to the Add-ons project. You can view the docs here.

If you would like to contribute to the docs, here's how to set up a development environment:

Using Python and optionally virtualenv, install the dependencies:

pip install -r requirements/docs.txt

Make some changes then build the docs:

make -C docs/ html

Open docs/_build/html/index.html to preview your changes before creating a pull request.

addons's Issues

AMO prod is intermittently down

Prerequisites:
Build identifier: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0

Steps to reproduce:
1.Load AMO prod

Expected results:
AMO prod is correctly loaded

Actual results:
Sometimes, AMO prod is not loading and it is returning a 503 or 502 Bad Gateway

Screencast : http://screencast.com/t/m0Foplvl39QJ

Remove white and blacklist

Because this matters to me: http://mckay.pub/2015-01-07-allowed/

Receipt error in logs

The logs are full of this:

Type: <type 'exceptions.UnicodeDecodeError'>, 'utf8' codec can't decode byte 0xae in position 19: invalid start byte. Data:  :./services/utils.py:143#012Traceback (most recent call last):#012  File "./services/theme_update.py", line 219, in application#012    output = update.get_json()#012  File "./services/theme_update.py", line 160, in get_json#012    return json.dumps(data)#012  File "/usr/lib64/python2.7/json/__init__.py", line 243, in dumps#012    return _default_encoder.encode(obj)#012  File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode#012    chunks = self.iterencode(o, _one_shot=True)#012  File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode#012    return _iterencode(o, 0)#012UnicodeDecodeError: 'utf8' codec can't decode byte 0xae in position 19: invalid start byte

from z.receipt.

Remove the locale and product from the API URLs

Signing is currently failing for me because of:

sort-tabs-by-url jpm sign --api-key=user:5520425:594 --api-secret=... --api-url-prefix=http://amo.dev/api/v3
JPM [info] Starting jpm sign on sort-tabs-by-url
Creating XPI
JPM [info] XPI created at /Users/andy/sandboxes/sort-tabs-by-url/[email protected] (55ms)
Created XPI at /Users/andy/sandboxes/sort-tabs-by-url/[email protected]
Successfully created xpi at /Users/andy/sandboxes/sort-tabs-by-url/[email protected]

Potentially unhandled rejection [2] Error: Received bad response from the server while signing; status: 302; response: 
    at /Users/andy/sandboxes/jpm/lib/amo-client.js:80:13
    at tryCatchReject (/Users/andy/sandboxes/jpm/node_modules/when/lib/makePromise.js:845:30)
    at runContinuation1 (/Users/andy/sandboxes/jpm/node_modules/when/lib/makePromise.js:804:4)
    at Fulfilled.when (/Users/andy/sandboxes/jpm/node_modules/when/lib/makePromise.js:592:4)
    at Pending.run (/Users/andy/sandboxes/jpm/node_modules/when/lib/makePromise.js:483:13)
    at Scheduler._drain (/Users/andy/sandboxes/jpm/node_modules/when/lib/Scheduler.js:62:19)
    at Scheduler.drain (/Users/andy/sandboxes/jpm/node_modules/when/lib/Scheduler.js:27:9)
    at process._tickCallback (node.js:355:11)

I think this because the server is returning a 302:

Location: http://amo.dev/en-US/firefox/api/v3/addons/jid0-asd%40jetpack/versions/0.3.2/

Cannot transfer ownership for Themes

Prerequisites:
Build identifier: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0

Steps to reproduce:
1.Submit a new theme
2.Go to theme edit listing page and try to transfer ownership

Expected results:
You can transfer ownership without problems

Actual results:
You cant transfer ownership(Button is greyed out)

Please see screencast for this bug : http://screencast.com/t/OQSPJX88wC

Do you prefer github issues? Bugzilla issues?

Might be worth mentioning this in README.rst, similar to how
https://github.com/mozilla/addon-sdk/blob/master/README
makes it clear they want bugzilla issues.

Convert all the old API to our new v3 REST based API

We have multiple APIs, a v1.5, a v2, mostly used by clients like Firefox for searching.

We are creating a new API for addons.mozilla.org which will be less XML, more JSON, faster and better in every way. Maybe. Assuming that's the case, we need to deal with legacy clients that use the old API. That means providing an API to move to, a migration plan and a deprecation strategy.

Put WebExtension submission behind a flag

I think the ability to submit add-ons as a WebExtension should be behind a flag, so that we can have it off in prod until we think through all the requirements.

Supports https://bugzilla.mozilla.org/show_bug.cgi?id=1210037

AMO prod updates are applied with a delay

Prerequisites:
Build identifier: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0

Steps to reproduce:
1.Perform any action which should add a visual modification on AMO prod (ie. add many reviews to an app, follow\unfollow a collection, add a new addon to a collection)
2.Refresh the page to see the changes

Expected results:
AMO is instantly updated

Actual results:
Strange behaviors are seen after refresh (ie. reviews disappear , followed collections are still seen as unfollowed , addons are not added to collections, etc... )

I added two screencast with these issues:
http://screencast.com/t/orqXjzTZ
http://screencast.com/t/yC3T21BF82

Password reset can result in a 500 if b64 padding is incorrect

see https://sentry.prod.mozaws.net/operations/olympia-prod/group/151058/

Kick off FxA login from the frontend

Use the config provided by #1585 to begin the authentication flow to FxA. I think opening a popup will be easiest however there is an iframe version that might provide a nicer flow.

This will be similar to startLogin from marketplace-core-modules however that code handles several different FxA authentication methods (native, iframed native and web) and we will just support web.

Use Firefox Accounts on AMO

Lets use Firefox Accounts on AMO, instead of AMOs home grown authentication system and gently migrate our users over to the new system.

This is a tracking bug for all the things we are about to do on this.

Prevent double-submit of forms

Due to the slowness on AMO or unstability of my network connection, my forms gets submitted more than once. This results in duplicate review actions or messages.

Could you add something to prevent double-submit? If possible, check whether someone else has already posted an update before accepting the form (e.g. show "This add-on review has been updated. Please check whether your comment/review/approval/rejection is still relevant." when another message has been added since the last post).

This check could be at the Python backend, or part of the front-end (perform an (ugh) synchronous AJAX request upon submit, and only confirm the request if the last message hasn't changed; this only works if the server immediately replies, but I've noticed that AMO suffers from an aggressive cache, so this idea does not work...).

AMO needs gentler error pages

Yesterday there were site outages and I was presented with a plain page that said "Error 503" and that was about it. No formatting or explanation.

We should have gentler pages for our users.
Most larger sites have some sort of custom pages in these instances which are less harsh and more informative. Let's try and make ours more approachable and informative too.

Don't return HTML in the API

If a request comes into something on the API its asking for json, don't return HTML on an error. Example:

>>> res = requests.get('https://addons-dev.allizom.org/en-US/firefox/api/v3/does-not-exist', headers={'Accept': 'application/json'})
send: 'GET /en-US/firefox/api/v3/does-not-exist HTTP/1.1\r\nHost: addons-dev.allizom.org\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: application/json\r\nUser-Agent: python-requests/2.5.3 CPython/2.7.10 Darwin/14.5.0\r\n\r\n'
reply: 'HTTP/1.1 404 NOT FOUND\r\n'
header: Content-Encoding: gzip
header: Content-Security-Policy-Report-Only: script-src 'self' https://www.google.com https://www.paypalobjects.com https://ssl.google-analytics.com https://addons-dev-cdn.allizom.org/user-media; default-src * data:; style-src * 'unsafe-inline'; frame-src https://ssl.google-analytics.com https://sandbox.paypal.com; object-src 'none'; report-uri /services/csp/report
header: Content-Type: text/xml; charset=utf-8
header: Date: Sun, 01 Nov 2015 17:41:53 GMT
header: Server: nginx
header: Strict-Transport-Security: max-age=31536000
header: Vary: Accept-Encoding
header: Vary: X-Mobile, User-Agent
header: X-Frame-Options: DENY
header: Content-Length: 90
header: Connection: keep-alive
>>> res.content
'<?xml version="1.0" encoding="utf-8" ?>\n      <error>Not Found</error>\n  '

I'm requesting JSON and get back XML.

Change delete add-on form

The delete add-on form currently requires you to enter your password. In a world where passwords are all stored on FxA, that's not going to work, we'll have to come up with something different. We could force FxA authentication or just require something else typed in like the add-on GUID or something.

https://www.dropbox.com/s/97n8gnxodackle1/Screenshot%202015-11-13%2011.22.22.png?dl=0

Show account info form if account was created

If a user's login with FxA resulted in a new account being created then the account details form should be shown to allow the user to complete the registration process.

We may want to delay creating the account initially until this data is collected however that could result in the FxA token expiring. There will be weird edge cases here if the user navigates away or leave and signs in with this account again without completing the process.

Use the returned FxA authentication token to log the user in

Once FxA returns the user's authentication token pass that to the API created in #1588 to log the user in.

This should be similar to fireplace's handle_fxa_login.

The add-ons video is a bit old skool

Go to about:addons, click "watch the video" or "Learn more about add-ons".

A couple of things:

• It says "You can use personas", they got renamed to themes a while ago
• Two of the add-ons that it adds in "F1 by Mozilla Labs" and "Add to Amazon Wishlist" no longer exist

If we want to keep this video around, I bet we can find replacements for those add-ons and just update that.

Integrate performance testing to find regressions

I just found out about http://pytest-benchmark.readthedocs.org/en/stable/ and given the fact that we're planning on doing lots of refactoring and updates I was wondering if it would make sense to get this integrated somehow to see how we're doing.

I doubt it'll be easy to integrate with CI but at least with a small shell script and git we should be able to get reliable performance graphs.

Any ideas on that? @andymckay @magopian

Example: http://pytest-benchmark.readthedocs.org/en/stable/comparing.html

Add in an experimental or beta banner to the API

The API that we've got should probably have a "beta" or "experiemental" flag on it in olympia. The best place to put this is on the key creation page.

Do not localize add-on deletion emails

https://github.com/mozilla/olympia/blob/master/apps/addons/models.py#L458 sends emails when an add-on is being deleted. The property atype is being localized. We don't want that as it makes it very hard to search in those emails.

I'd give a shot a this myself, but I don't know why l10n happens in the first place. Any hints are appreciated.

Example:
The following ΘΈΜΑ was deleted.

Add waffle-flag to transparently call addons-validator

Olympia part to mozilla/addons-linter#62

Support creating accounts with FxA API

If an email address that we don't have an account for is logging in with FxA using the API created in #1588 then that account should be created with some basic info and the client should be notified that this was a new account in the response.

Wrap addons-linter binary

addons-server part to mozilla/addons-linter#62

Implement an extensivle wrapper around calling the addons-validator binary and passing arguments. This should basically just call subprocess.Popen and sorts.

Proper logging and exception handling.

Upgrade to Django 1.7

This is the GitHub tracking issue for https://bugzilla.mozilla.org/show_bug.cgi?id=1164477

TODOS:

• locale/templates/LC_MESSAGES/{django,djangojs}.pot get updated on extract
• Get rid of mysql-pool and use CONN_MAX_AGE config
• https://github.com/mozilla/zamboni/pull/3038/files#diff-aa9311318d4c8ebf2b5495d72c7f356dL41
• get rid of happyforms (https://github.com/mozilla/olympia/issues/1569)

Other cleanup stuff I noticed:

• olympia.services.utils has some weird import hacks 😕 (https://github.com/mozilla/olympia/issues/1570)

General updates, things to notice:

• tower has been replaced by puente (https://github.com/mozilla/puente)

Verify the following dependency updates to exist in wheelhouse:

• django-waffle==0.11
• django-cache-machine==0.9.1
• Jinja2==2.8
• MarkupSafe==0.23
• Django==1.7.11
• jingo==0.8
• django-babel==0.4.0
• puente==0.4
• pytz==2015.7

Most are so this is mostly for documentation reasons.

Reviews being added to stage cause a timeout

Go to any add-on click, write a review, complete form and write a review. Click submit.

I am consistently getting: [HTTP/1.1 502 Bad Gateway 32678ms] back.

According to @jasonthomas "it seems like update_denorm is taking a while to run on stage".

Add a new login endpoint to handle FxA authentication

Add a new login endpoint that will accept an FxA authentication token and return an API key to the client. This should also log the user into the django session based authentication system to maintain backwards compatibility.

This will be similar to zamboni's FxALoginView however it should again be much simpler since we only have one authentication method.

An Add-on cannot be rejected after it was previously approved

Steps to reproduce:

1. Submit a listed add-on on AMO-stage https://addons.allizom.org/en-US/firefox/
2. Approve the submitted add-on (i.e. Addon-approved-rejected)
3. Access the review page of the add-on from Editor Tools->Logs->Add-on Review log
4. Reject the add-on

Expected results:
The add-on status from its review history page is “Disable by Mozilla”. The add-on is no longer available in AMO Public Pages.

Actual results:
The add-on status is still “Fully reviewed” and is also available in public pages.

Notes/Issues:
An add-on can be rejected if this is the first action made.
Verified on FF42(Win 7). Issue is reproducing on AMO-stage.
Screencast for this issue: http://screencast.com/t/OUUsnnLD

Add a link to status.m.o in the footer.

Would be a pretty quick win to add a link to the footer to https://status.mozilla.org. so our users know to look there for status info.

Extend validation storage to store js/python-validator results

• Maybe add a source type for validation (js vs python validator) to file-validation
• Validation results are stored in files.FileValidation, this needs to be extended somehow so that we can compare results of both validators in a later step.

Maybe also think about simply storing the js-results in the same row as the original results so that we don't need to implement new filters everywhere.

Provide UX mocks for FxA migration

Provide some UX mocks and flow chart for a FxA "email funnel" flow. That we'll then break down into bugs. If this could be done as a pull request against this repository into the docs/ux folder, that would be awesome.

http://addons.readthedocs.org/en/latest/ux/index.html

Send WebExtensions off to manual review

Because we aren't quite sure what we need to validate or look at in WebExtensions, it was suggested that instead we just WebExtensions off for manual review on AMO.

Implement compare view

This is the olympia part to mozilla/addons-linter#62

implement a view that compares the results between python and js validator. It appears we might be able to leverage some code from devhub.utils:ValidationComperator.

Convert an existing account to FxA login

We should have a mechanism for a logged in user to convert their old account to an FxA account. This will likely involve logging into the old username/password account and then logging into FxA. This should allow for the user to change the email address stored in olympia.

We should discuss with UX the logistics of this. Should we require a user to log in to their old account to convert it?

Inform users of ongoing service issues.

Yesterday AMO experienced outages and poor load times.
A blog post was made to acknowledge this, but it was observed that a very small number of those affected probably saw this.
We need a site-wide method to notify users should something similar happen again. A simple colored bar with text across the top of the site would probably work well.

Waffle old registration system

We should wrap the old registration process in a waffle so that it can be turned off once #1594 is completed to allow registration with FxA.

Add metrics for validation process and addons-linter wrapper

This is the olympia part to mozilla/addons-linter#62

Add in metrics so that we get an idea of how fast the addons-linter is.

There is the devhub.validator namespace we're using in statsd to track amo-validator times and stats, given that I think devhub.addons_linter might be a good namespace.

Test and make sure that the js-validator is hidden everywhere

This is the olympia part to mozilla/addons-linter#62

Ensure that view pages, api etc return the correct validation, making the js-validator hidden from normal view.

Issues at log in with a new created account in AMO-prod

Steps to reproduce:

1. Load AMO-prod page https://addons.mozilla.org/en-US/firefox
2. Click "Register" link and register as a new user
3. Go to your email and click the confirmation link
4. Try to sign in with the new created account

Expected results:
The message "Successfully verified" is listed in AMO page. After entering the new created email and password the user is logged in.

Actual result:
User cannot log in. An error message is displayed.

Notes/Issues:
Verified on FF42(Win 7). Issue is reproducing on AMO-prod.
Please see the screencast: http://screencast.com/t/nSBAQt540UJ

Unused, possibly dangerous code?

https://github.com/mozilla/olympia/blob/master/apps/translations/forms.py#L56

def as_ul(self):

etc

Initial investigation result is that this is dead/uncalled code.

My concern is that if it gets returned to something else and modified, it will not be mark_safe anymore.

Formatting of API Key email could do with spacing

Looks like this for me:

https://www.dropbox.com/s/0um3guzzh7l8h8b/Screenshot%202015-11-12%2015.42.50.png?dl=0

But the new-key-email.ltxt file seems to be nicely formatted with fancy new lines and all.

Add a CSRF token to the FxA state parameter

This isn't done in zamboni so we have no reference but it is considered a best practice to include some state that is also stored in the user's session in the state parameter passed to FxA. This way we can verify that the authentication request was generated by the app and not a third party. We should include some signing of the state parameter as well.

Creation date of a theme is listed in the “Date Deleted” column from Editor Tools

Steps to reproduce:

1. Load AMO-stage Manage My Submission page https://addons.allizom.org/en-US/developers/addons
2. Click the “My Themes” tab
3. Delete a theme (i.e. 2)
4. Go to Editor Tools->Themes tab
5. Access “Deleted” option from "Logs"
6. Observe the date from “Date Deleted”

Expected results:
The date when the theme was deleted coincide with the value from “Date Deleted” column.

Actual results:
The date when a theme was created is listed in the “Date Deleted” column.

Notes/Issues:
Verified on FF42(Win 7). Issue is reproducing on AMO-stage and AMO-dev.
Screencast for this issue: http://screencast.com/t/G0qtdC4QM1gY

Log stats about FxA

We should collect some stats:

• FxA login success
• FxA login failure
• Request time to get bearer token
• Request time to get profile
• Total verification time (token + profile)

Flag button is missing for theme updates

Steps to reproduce:

1. Open theme updates queue: https://addons.allizom.org/en-US/editors/themes/updates
2. Open the detail of any theme.

Expected results:
There are 5 review buttons Request More Info, Flag, Duplicate, Reject, Approve Theme.

Actual Results:
Flag button is missing.

Notes/Issues:
Verified in FF43(Win7) Issue can be reproduced in addons.allizom.org

Add an icon in the queue for webextensions

The same way we have icons for restartless or jetpack add-ons.

See the "Flags" column below:

This will need https://bugzilla.mozilla.org/show_bug.cgi?id=1219772 first (which depends on #1602)

Generate config needed to begin FxA authentication

We will need to setup a new app in the FxA Stable OAuth Console and provide the client_id, state and redirect_url to the frontend.

This will be similar to the fxa_auth_info helper in zamboni.

Integrate addons-validator into olympia build

Olympia part to mozilla/addons-linter#62

• Add as dependency in package.json (https://www.npmjs.com/package/addons-validator)
• Update paths to find and execute the binary

Display issues for theme name if it contains numbers

Steps to reproduce:

1. Sign into AMO-stage https://addons.allizom.org/en-US/firefox/
2. Go to Tools->Submit a New Theme
3. Give your theme a name that contains numbers/special_chars (i.e. 3Theme)
4. View listing for the submitted theme

Expected results:
There are no display or layout issues.

Actual results:
The numbers/special_chars are not aligned with the rest of the name.

Notes/Issues:
Verified on FF42(Win 7). Issue is reproducing on AMO-stage, dev and prod.