mozilla / newnewtab

New Tab page: content server and recommendation service

newnewtab

newnewtab is a project to enhance Firefox users' apps experience. When opening a new tab, they are offered their own (currently installed) tabs, along with one or more recommended apps.

It is planned for recommendations to be targeted, yet privacy-preserving.

You may find development wiki valuable.

Install

• install node and redis
• clone the repository
• install dependencies using npm npm install -d
• run application node app

mozApps Whitelist

In order to get this to work at all, you'll need to whitelist the address newnewtab is running from in your about:config in Firefox (tested in version 15+ -- i.e. Nightly). Basically, open up your about:config and add the key: dom.mozApps.whitelist. Make sure localhost:3000 (or wherever newnewtab is running from) is there so it can access your installed apps!

You'll need Firefox 15+ for things to work.

Settings

By default, settings are in config-default.js. You can override settings in there with environment variables (PORT will override express:port), but if you want to customize lots of settings locally (either for development or deployment), you can create config-local.json and that file will augment or add to settings read by default.

config-local.json is not version-controlled; feel free to edit it to your heart's content.

Fetching recommendations

Use node script/fetch_recommendations.

newnewtab's Issues

Write simple Marketplace API to feed the recommendations

See bug 755801. To feed live data into this, we'll have to write a simple, simple, public API. Keep it as simple as possible, have it reviewed, land it, off we go.

Recompute interests for users that restart infrequently

Perhaps add a cache timer or use the once-a-day notification.

See uses of the idle-daily notification in mozilla-central:
http://mxr.mozilla.org/mozilla-central/search?string=idle-daily&find=.js&findi=&filter=[Ii]dle-daily&hitlimit=&tree=mozilla-central

Implement marketplace category recomendation based on user interests

This will be implemented via integrating addaon/example code into public/javascript/main.js
The idea is to assign percentages to marketplace categories, then the caller can request a sudo-random marketplace category.
Whereby, the category is returned according to its percentage.
This code will parked in public/javascript/main.js for the time being

Use the bottom row of the new tab page for newnewtab

We should be able to hide a div.newtab-row and create a separate div.newtab-row and put the iframe inside there. So this would get rid of the button toggler and transition/animations.

Show fallback icon if app does not have an icon

Not sure if apps without icons even follow the spec properly so I expect this to be rare; either way, if we don't have anything to show, we should probably just use a generic icon.

Decouple database connection, express app and setting system

Current config logic requires using express even for command line.
We should use nconf (https://github.com/flatiron/nconf) or a simple JSON config.
I've been discussing it on freenode#express and they suggested using just a default.json file that gets loaded by default, and then a {DEV_ENV}.json file that is loaded on top of that to overwrite settings.

Show warning if the app is not whitelisted for access to apps API

If we're not on the whitelist for the apps API, we can't show any installed apps. Try to detect this and show an error message.

Even if the add-on does it for us in the future, until then we don't have to baby everyone through the process of "I think newnewtab is broken"!

Whitelist access for mozApps.mgmt.getAll

Looks like these sites need to be added for now:

https://newnewtab.mozillalabs.com/
https://newnewtab-dev.mozillalabs.com/
https://newnewtab-stage.mozillalabs.com/

Replace jade with another templating language

Something a bit closer to HTML please.

https://github.com/visionmedia/ejs
http://mustache.github.com/

Provide an API to get recent categorization

Might be useful to allow the recommendations to play around with recent vs long-term categories. When implementing this, it would be good to see how long it takes to calculate it and perhaps it might not need any caching.

Use the most recent 7 days of history for the categorization?

Compute user interests on demand

Hook up docs to readthedocs

Not a high priority, but if the docs were on newnewtab.readthedocs.org, that would be peachy.

Update existing MPL 1.1 headers to MPL 2

Some existing code was committed with the MPL 1.1 headers probably before the big switch for the rest of the Mozilla codebase to MPL 2.0

Create basic JSON recommendation backend

Hook up a URL that serves a JSON blob. Determine what this format should look like (probably a content string (contains HTML), a time-to-live for each blob). Anything else?

Respect if-modified-since header, do not serve JSON unless necessary.

Create frontend JS updating recommendations

The initial page should be served with generic recommendations to start off with. On a schedule, ask the server for new updates. Take the JSON, store it locally and rotate the recommendations on a schedule, unless their TTL has run out. Ask the server for updates, if-modified-since, and replace the recommendations if the server gives you new ones, otherwise keep showing the existing ones. If you're out of recommendations to show, fall back to the generic ones the page shipped with.

Update visit counting logic to leverage rev_host grouping in SQL

Right now every url is fetched then visit counts aggregated in javascript after extracting the host, but we can reuse the existing rev_host column to do the aggregation in the query.

Create basic HTML layout for page

Make it look like the about:newtab for now. Feel free to start off with empty boxes.

Find out if we need to track views at all, or just clicks

Hey @fwenzel and @zalun -- we should find out if we need to track views at all for metric or just clicks. Who shall we ask?

Write README with quick installation instructions

Please explain, shortly, how to set this up. I imagine it's simple, but an empty readme doesn't help. :-/

Whitelist app for localstorage usage?

@Mardak Not sure if this is possible, but Nighlty pops down a bar asking to approve usage of localstorage:

https://skitch.com/fredw/8625y/newnewtab

Will the add-on be capable of automatically whitelisting the newnewtab URL to use localstorage? The less friction the better.

``make test`` is throwing as well as ``node app.js``

different errors, maybe my installation faults, but please help.

make test

/home/zalun/Projects/newnewtab/node_modules/mocha/lib/runner.js:382
if ('failed' == runnable.state) return;
^
TypeError: Cannot read property 'state' of undefined
at Runner.uncaught (/home/zalun/Projects/newnewtab/node_modules/mocha/lib/runner.js:382:27)
at process. (/home/zalun/Projects/newnewtab/node_modules/mocha/lib/runner.js:429:10)
at process.emit (events.js:115:20)
make: *** [test] Error 1

node app.js
LOADING /home/zalun/Projects/newnewtab/config-default.json
LOADING /home/zalun/Projects/newnewtab/config-development.json
IGNORED /home/zalun/Projects/newnewtab/config-local.json (file doesn't exist)
path.existsSync is deprecated. It is now called fs.existsSync.
Bad locale=[en_US] file(s) do not exist [/home/zalun/Projects/newnewtab/locale/en_US/LC_MESSAGES/messages.mo]. See locale/README
Express server listening on port 3000 in development mode
Database ( 7 ) connection status: undefined

events.js:66
throw arguments[1]; // Unhandled 'error' event
^
Error: Redis connection to 127.0.0.1:6379 failed - connect ECONNREFUSED
at RedisClient.on_error (/home/zalun/Projects/newnewtab/node_modules/redis/index.js:140:24)
at Socket. (/home/zalun/Projects/newnewtab/node_modules/redis/index.js:74:14)
at Socket.emit (events.js:88:17)
at net.js:336:14
at process._tickCallback (node.js:244:9)

Switch to built in jetpack window watcher to remove unload+/watchWindows

Jetpack has an api to watch for new windows being opened and closed, so we can use that instead of the custom watchWindows file. And when removing that, there'll be no other uses of unload either so that can be removed.

Choose documentation system

We need some documentation next to the code.
I've got no experience in documenting except of python stuff (Sphinx is the king IMO, it's got some JS hooks)
We might just use the wiki, but it could be nicer to have it with the code.

Fill some/most boxes with your own apps

Most of the boxes should point to your own apps (I think). Only one-or-so should be a recommendation.

To access your installed apps, hook into the mozApps API: https://developer.mozilla.org/en/Apps/Apps_JavaScript_API

Note that this is a whitelisted API (plus one that I think needs you to install the Apps browser extension), so you might have to fiddle a little. Make sure to note what help we might need from the newnewtab add-on to make this happen.

Create node.js shell for newnewtab

node shell, pointing to a single view (which will serve the basic layout of the page, along with JS and CSS)

Newtab page gets reloaded in multiple windows

Create multiple windows
Add newtab page in each
The create a newtab in one of them and observe the newtab pages in the rest of windows being reloaded.

Make clicks on recommendations countable

We need to be able to count clicks. (Note, that does not mean we actually have to count them ourselves in our own database. Generating Apache logs could suffice).

One option is to link the recommendations to a URL on the server and when you click on it, the server redirects you to your real target URL.

Another option would be to count clicks via AJAX (i.e. capture the click event and issue a count query via AJAX).

Opinions?

Auto-build extension

In pdf.js's readme file, there's a link to a xpi built off of trunk. I think we should do the same (probably on the dev server)?

@tofumatt do you run an update script on there?

Make sure launching apps opens the native runtime instead of pinned tab

The core platform bug is here: https://bugzilla.mozilla.org/show_bug.cgi?id=745924

But we should be able to implement it in the add-on before it lands on Nightly.