This service is responsible for allocating Firefox Sync users to one of several Sync Storage nodes. It provides the "glue" between Firefox Accounts and the SyncStorage API, and handles:
- Checking the user's credentials as provided by FxA
- Sharding users across storage nodes in a way that evenly distributes server load
- Re-assigning the user to a new storage node if their FxA encryption key changes
- Cleaning up old data from e.g. deleted accounts
The service was originallly conceived to be a general-purpose mechanism for connecting users to multiple different Mozilla-run services, and you can see some of the historical context for that original design here and here.
In practice today, it is only used for connecting to Firefox Sync.
Like this:
$ make install
$ ./local/bin/pip install gunicorn
$ ./local/bin/gunicorn --paste etc/tokenserver-dev.ini
Firfox Sync clients must first obtain user credentials from FxA, which can be either:
- A BrowserID assertion with audience of
https://token.services.mozilla.com/ - An OAuth access token bearing the scope
https://identity.mozilla.com/apps/oldsync
They then provide this in the Authorization header of a GET request to the Tokenserver,
which will respond with the URL of the user's sync storage node, and some short-lived credentials
that can be used to access it.
More detailed API documentation is available here.
To access the user's sync data using BrowserID, the client must obtain a BrowserID assertion
with audience matching the tokenserver's public URL, as well as the user's Sync encryption key.
They send the BrowserID assertion in the Authorization header, and the first half of the
hex-encoded SHA256 digest of the encryption key in the X-Client-State header, like so:
GET /1.0/sync/1.5
Host: token.services.mozilla.com
Authorization: BrowserID <assertion>
X-Client-State: <hex(sha256(kSync))[:32]>
To access the user's sync data using OAuth, the client must obtain an FxA OAuth access_token
with scope https://identity.mozilla.com/apps/oldsync, and the corresponding encryption key
as a JWK. They send the OAuth token in the Authorization header, and the kid field of the
encryption key in the X-KeyID header, like so:
GET /1.0/sync/1.5
Host: token.services.mozilla.com
Authorization: Bearer <access_token>
X-KeyID: <JWK['kid']>
The tokenserver will validate the provided credentials, and either look up the user's existing storage node allocation or assign them to a new one. It responds with the location of the storage node and a set of short-lived credentials that can be used to access it:
{
'id': <token>,
'key': <request-signing secret>,
'api_endpoint': 'https://db42.sync.services.mozilla.com/1.5/12345',
'uid': 12345,
'duration': 300,
}
The value of <token> is intended to be opaque to the client, but is in fact an encoded JSON blob
signed using a secret key shared between the tokenserver and the storage nodes. This allows
the tokenserver to securely communicate information about the user to their storage node.
The fields contained therein include:
uid: A numeric userid that uniquely identifies this user, on this storage node, using this encryption keynode: The intended storage node on which these credentials can be usedexpires: A timestamp for when the credentials expirefxa_uid: The user's stable FxA user id, as a hex stringfxa_kid: The key-id of the JWK representing the user's sync encryption key
The data store is specified in the [tokenserver] section as sqluri. If you want to use a longer
lived data store than sqlite3, you will need to specify the Data Source Name (DSN) as a Universal
Resource Locator (URL) such as: sqluri = mysql+pymysql://scott:tiger@localhost/tokenserver. See the
SQLAlchemy Engine specification for details.
TokenServer comes with support for pymysql, mysqldb, and sqlite3 by default. Additional databases
will require pip install.
The core of the TokenServer's data model is a table named users that maps each user to their storage
node, and that provides enough information to update that mapping over time. Each row in the table
contains the following fields:
uid: Auto-incrementing numeric userid, created automatically for each row.service: The service the user is accessing; in practice this is alwayssync-1.5.email: Stable identifier for the user; in practice this is always<fxa_uid>@api.accounts.firefox.com.nodeid: The storage node to which the user has been assigned.generation: A monotonically increasing number provided by the FxA server, indicating the last time at which the user's login credentials were changed.client_state: The hash of the user's sync encryption key.keys_changed_at: A monotonically increasing timestamp provided by the FxA server, indicating the last time at which the user's encryption keys were changed.created_at: Timestamp at which this node-assignment record was created.replaced_at: Timestamp at which this node-assignment record was replaced by a newer assignment, if any.
As you can see, this table contains some unnecessarily general names; these are a legacy of earlier plans to re-use Tokenserver for multiple Mozilla services and with multiple identity providers.
The generation column is used to detect when the user's FxA credentials have been changed
and to lock out clients that have not been updated with the latest credentials.
Tokenserver tracks the highest value of generation that it has ever seen for a user,
and rejects BrowserID assertions in which the generation number is less than that high-water mark.
Note that OAuth clients do not provide a generation number, because OAuth tokens get
revoked immediately when the user's credentials are changed.
The client_state column is used to detect when the user's encryption key changes.
When it sees a new value for client_state, Tokenserver will replace the user's node assignment
with a new one, so that data encrypted with the new key will be written into a different
storage "bucket" on the storage nodes.
The keys_changed_at column tracks the timestamp at which the user's encryption keys were
last changed. BrowserID clients provide this as a field in the assertion, while OAuth clients
provide it as part of the X-KeyID header. Tokenserver will check that changes in the value
of keys_changed_at always correspond to a change in client_state, and will use this pair of
values to construct the fxa_kid field that is communicated to the storage nodes.
When replacing a user's node assignment, the previous column is not deleted immediately.
Instead, it is marked as "replaced" by setting the replaced_at timestamp, and then a background
job periodically purges replaced rows (including making a DELETE request to the storage node
to clean up any old data stored under that uid).
For this scheme to work as intended, it's expected that storage nodes will index user data by either:
- The tuple
(fxa_uid, fxa_kid), which identifies a consistent set of sync data for a particular user, encrypted using a particular key. - The numeric
uid, which changes whenever either of the above two values change.
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent