An implementation of Amazon's DynamoDB, focussed on correctness and performance, and built on LevelDB (well, @rvagg's awesome LevelUP to be precise).
This project aims to match the live DynamoDB instances as closely as possible (and is tested against them in various regions), including all limits and error messages.
Because it's too buggy! And it differs too much from the live instances in a number of key areas (see below)
$ dynalite --help
Usage: dynalite [--port <port>] [--path <path>] [options]
A DynamoDB http server, optionally backed by LevelDB
Options:
--help Display this help message and exit
--port <port> The port to listen on (default: 4567)
--path <path> The path to use for the LevelDB store (in-memory by default)
--ssl Enable SSL for the web server (default: false)
--createTableMs <ms> Amount of time tables stay in CREATING state (default: 500)
--deleteTableMs <ms> Amount of time tables stay in DELETING state (default: 500)
--updateTableMs <ms> Amount of time tables stay in UPDATING state (default: 500)
Report bugs at github.com/mhart/dynalite/issues
Or programmatically:
// Returns a standard Node.js HTTP server
var dynalite = require('dynalite'),
dynaliteServer = dynalite({path: './mydb', createTableMs: 50})
// Listen on port 4567
dynaliteServer.listen(4567, function(err) {
if (err) throw err
console.log('Dynalite started on port 4567')
})
Once running, here's how you use the AWS SDK to connect (after configuring the SDK):
var AWS = require('aws-sdk')
var dynamo = new AWS.DynamoDB({endpoint: 'http://localhost:4567'})
dynamo.listTables(console.log.bind(console))
With npm do:
$ npm install -g dynalite
- Add ProvisionedThroughput checking
- Add config settings to turn on/off strict checking
- Use efficient range scans for
Query
calls - Explore edge cases with
Query
andScanIndexForward: false
(combine with above) - Implement
ReturnItemCollectionMetrics
on all remaining endpoints - Implement size info for tables and indexes
- Is the
ListTables
limit of names returned 100 if noLimit
supplied?
Part of the reason I wrote dynalite was due to the existing mock libraries not exhibiting the same behaviour as the live instances. Amazon released their DynamoDB Local Java tool recently, but the current version (2013-09-12) still has quite a number of issues that have prevented us (at Adslot) from testing our production code, especially in a manner that simulates actual behaviour on the live instances.
Some of these are documented (eg, no ConsumedCapacity
returned), but most aren't -
the items below are a rough list of the issues we've found (and do not exist in dynalite), vaguely in order of importance:
- Returns 400 when
UpdateItem
uses the defaultPUT
Action
without explicitly specifying it (this actually prevents certain client libraries from being used at all) - Does not return correct number of
UnprocessedKeys
inBatchGet
(returns one less!) - Returns 400 when trying to put valid numbers with less than 38 significant digits, eg 1e40
- Returns 200 for duplicated keys in
BatchGetItem
- Returns 200 when hash key is too big in
GetItem
/BatchGetItem
- Returns 200 when range key is too big in
GetItem
/BatchGetItem
- Returns 200 for
PutItem
/GetItem
/UpdateItem
/BatchGetItem
/Scan
/etc with empty strings (eg,{a: {S: ''}}
) - Returns 413 when request is over 1MB (eg, in a
BatchWrite
with 25 items of 64k), but live instances allow 8MB - Returns
ResourceNotFoundException
inListTables
ifExclusiveStartName
no longer exists - Does not return
ConsistentRead
property inUnprocessedKeys
inBatchGet
even if requested - Returns 200 for empty
RequestItems
inBatchGetItem
/BatchWriteItem
- Returns 200 when trying to delete
NS
fromSS
orNS
fromN
or addNS
toSS
orN
toNS
- Allows
UpdateTable
when read and write capacity are same as current (should be an error) - Tables are created in
ACTIVE
state, notCREATING
state - Tables are removed without going into a
DELETING
state - Tables are updated without going into a
UPDATING
state PutItem
returnsAttributes
by default, even though none are requested- Does not add
ProvisionedThroughput.LastIncreaseDateTime
inUpdateTable
- Does not update
ProvisionedThroughput.NumberOfDecreasesToday
inUpdateTable
- Different serialization and validation error messages from live instances (makes it hard to debug)
- Uses uppercase
Message
for error messages (should only use uppercase forSerializationException
) - Often returns 500 instead of 400 (or similarly appropriate status)
- Doesn't return
ConsumedCapacity
(documented - but makes it very hard to calculate expected usage) - Does not calculate the
Scan
size limits correctly so can return too many items - Does not return
LastEvaluatedKey
on aQuery
when at end of table - Does not return
LastEvaluatedKey
on aScan
whenLimit
equals number of matching items - Does not return
X-Amzn-RequestId
header - Does not return
X-Amz-Crc32
header - Does not return
application/json
ifapplication/json
is requested - Fails to put numbers 1e-130 and -1e-130 (succeeds on live instances)
- Returns an error when calling
Query
on a hash table (succeeds on live instances) - Returns 500 if random attributes are supplied (just ignored on live instances)
- Does not convert doubles to booleans (returns 500 instead)