jed / dynamo

DynamoDB client for node.js

This library has been deprecated.

Please use Amazon's SDK for node.js instead.

dynamo

This is a node.js binding for the DynamoDB service provided by Amazon Web Services. It aims to abstract DynamoDB's implementation (request signing, session tokens, pagination), but not its tradeoffs/philosophy, by providing two APIs:

• a low-level-but-ugly API that supports all 13 DynamoDB operations as-is, and
• a high-level API that uses the above to provide a more natural interface.

Example

var dynamo = require("dynamo")
  , client = dynamo.createClient()
  , db = client.get("us-east-1")

// High-level API

db.get("myTable")
  .query({id: "123", date: {">=": new Date - 6000 }})
  .get("id", "date", "name")
  .reverse()
  .fetch(function(err, data){ ... })

// Same call, using low-level API

db.query({
  TableName: "myTable",
  HashKeyValue: {S: "123"},
  RangeKeyValue: {
    ComparisonOperator: "LE",
    AttributeValueList: [{N: "1329912311806"}]
  },
  AttributesToGet: ["id", "date", "name"],
  ScanIndexForward: false
}, function(err, data){ ... })

Installation

This library has no dependencies, and can be installed from npm:

npm install dynamo

API

dynamo = require("dynamo")

This module exposes the createClient method, which is the preferred way to interact with dynamo.

client = dynamo.createClient([credentials])

Returns a client instance attached to the account specified by the given credentials. The credentials can be specified as an object with accessKeyId and secretAccessKey members such as the following:

client = dynamo.createClient({
  accessKeyId: "...",    // your access key id
  secretAccessKey: "..." // your secret access key
})

You can also omit these credentials by storing them in the environment under which the current process is running, as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

If neither of the above are provided, an error will be thrown.

db = client.get(regionName)

Returns a database in the selected region. Currently, DynamoDB supports the following regions:

• us-east-1
• us-west-1
• us-west-2
• ap-northeast-1
• ap-southeast-1
• eu-west-1

Once you have a database instance, you can use either of the provided APIs:

High-level API (blue pill)

The primary purpose of this library is to abstract away the often bizzare API design decisions of DynamoDB, into a composable and intuitive interface based on Database, Table, Item, Batch, Query, and Scan objects.

See the wiki for more information.

Low-level API (red pill)

All of the original DynamoDB operations are provided as methods on database instances. You won't need to use them unless you want to sacrifice a clean interdace for more control, and don't mind learning Amazon's JSON format.

See the wiki for more information.

Testing

Testing for dynamo is handled using continuous integration against a real DynamoDB instance, under credentials limited to Travis CI.

If you'd like to run the test stuie with your own credentials, make sure they're set using the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, and then run the tests:

npm test

The test suite creates three tables called DYNAMO_TEST_TABLE_1, DYNAMO_TEST_TABLE_2, and 'DYNAMO_TEST_TABLE_3` before the tests are run, and then deletes them once the tests are done. Note that you will need to delete them manually in the event that the tests fail.

To do

• Factor out tests into integration tests and unit tests
• Make all callbacks optional, returning an event emitter no callback given
• Add method to specify Limit and ExclusiveStartKey

Credits

• Travis CI for an awesome open-source testing service
• @chriso for letting me have the "dynamo" name on npm
• @skomski for turning me on to IAM credentials
• @mranney for inspiration from the venerable node_redis
• @visionmedia for making testing easy with mocha and should.js

Copyright

Copyright (c) 2012 Jed Schmidt. See LICENSE.txt for details.

Send any questions or comments here.

dynamo's Issues

Doc for Scan seems inconsistent, cf get() and count()?

Scan example on https://github.com/jed/dynamo/wiki/High-level-API is:

table.scan({
        name: "dynamo",
        iq: {">=": 100}
      })
      .get(["name", "iq", "status"])
      .count()
      .fetch(function(err, count){ ... })

But documentation for scan.count() just below this says: "cannot be used if Scan#get() is called"

error while retrieving and adding data

Hello
I was trying to use dynamo module for a little class project but its not working and giving a unusual error "healthy: dynamodb.us-east-1.amazonaws.com
^
SyntaxError: Unexpected token h "

I am all torn out seeking out the reason for this error.
Please let me know what could be the possible error.
here is the code that i have been trying to use
var dynamo = require("dynamo")
, client = dynamo.createClient()
, db = client.get("us-east-1")

db.createTable({
TableName: "DYNAMO_TEST_TABLE_1",

ProvisionedThroughput: {
ReadCapacityUnits: 5,
WriteCapacityUnits: 5
},

KeySchema: {
HashKeyElement: {
AttributeName: "hash",
AttributeType: "S"
}
}
}, function(err, data){ ... }

Thanks and regards

please tag versions on github

(preferrably in the format "v1.2.3")

It makes tooling easier :)

clarify retry strategy

while currently undocumented, this library takes a different approach to retries using the (err, data, next) signature in callback functions. in this case, next is a function that has the currently specified query in a closure:

• if err does not exist, next is used for iteration/pagination to get the next set of results.
• if err does exist, next is a retry function that can be executed at user discretion

the idea behind diverging from other SDKs is that retrying shouldn't be baked into the library, but performed according to user policy. would love to get feedback on this before settling on it in the documentation.

publish latest to npm

Hi Jed,

Can you please publish the latest to npm with the fix from this commit?

51b871a

I just spent quite a while tracking down this bug, only to realize it was already fixed on github.

Thanks a bunch.

error in readme

in the first code section 'table.name' should actually be 'table.TableName'

Feature request: Deleting items

I wasn't able to find how to delete items in the high-level api documentation. Perhaps I overlooked something?

How can I use pagination?

I know that DynamoDB has limit of 100 items GET.

And it returns continuing key to continue to fetch

next 100 results.

"Dynamo" says It aims to abstract DynamoDB's implementation (request signing, session tokens, pagination)

How can you do pagination by using Dynamo ? Can you provide me an example?

error getting started

`healthy: dynamodb.eu-west-1.amazonaws.com
^

SyntaxError: Unexpected token h
at Object.parse (native)
at IncomingMessage. (/Users/arpecop/Desktop/API/lambda/zanimaime/node_modules/dynamo/lib/Request.js:37:36)
at emitNone (events.js:72:20)
at IncomingMessage.emit (events.js:166:7)
at endReadableNT (_stream_readable.js:905:12)
at doNTCallback2 (node.js:450:9)
at process._tickCallback (node.js:364:17)`

createClient() with optional sessionToken

For the use case of using dynamo via command line program running in a shell, it would be nice to be able to pass an optional saved session token when creating a client in order not to get rate limited by AWS Security Token Service.

db = dynamo.createClient({
  accessKeyId: "...",
  secretAccessKey: "...",
  sessionToken: "..." // optional saved session token
});

item.destroy doesn't actually do anything

1. It doesn't delete anything from the table
2. It returns null as err (first param) regardless

keep track of capacity units

the capacity units consumed per operation are currently undersupported. would be nice to make these more visible in the high-level API.

item.save(callback) not implemented

It appears that item.save(callback) is not implemented as spec'ed out in the API.

AWS SDK - Doesn't seem as good as your library

I know you say that this is deprecated for the aws-sdk. But the aws-sdk is missing all of your sweet high-level functions.

I want the blue pill :)

low level API - listTables error

I get an error that the first argument to db.listTables(args,callback) should be a string. When I put the empty string, it seems to work, but this shouldn't be necessary. I am new to all this so another explanation is that I'm not making any sense....

Also, fleshing out the low-level api with some examples would be very helpful. I'm not exactly sure what the arguments should be like for many of the functions.

add support for other zones

amazon has added several zones (US, tokyo, singapore, ireland, etc.) since launch, which need to be supported.

schema specification shouldn't rely on object key order

The current API pays relies key order in the schema parameter passed to new db.Table().

However, this can fail on an ECMA-262 5th Ed. complaint interpreter. Perhaps it should be changed to a list:

recipeTable = new db.Table({
   name:  "recipes",
   schema: [ {userId: Number}, {date: String} ] // key order insensitive
   throughput: {read: 10, write: 10}
})

Fail to get an item from dynamo table (parsing error?)

raw response from dynamoDB below fails when Attributes.prototype.parse(data)

{
ConsumedCapacityUnits: 0.5,
Item:
{
name: { S: 'Christian Lewis' },
id: { S: '28664d40-6195-11e1-8d85-0be46e39396b' },
screen_name: { S: 'Tyler Young' }
}
}

I'm suspecting it's to do with ConsumedCapacityUnits

Am I not supposed to do table.get(key).fetch(opts, cb)?

It's basically failing to parse the raw response from Dynamo with stacktrace below

TypeError: Object.keys called on non-object
at Function.keys (native)
at Object.parse (node_modules/dynamo/lib/Value.js:17:23)
at node_modules/dynamo/lib/Attributes.js:20:34
at Array.forEach (native)
at Object.parse (node_modules/dynamo/lib/Attributes.js:18:23)
at node_modules/dynamo/lib/Item.js:54:32
at node_modules/dynamo/lib/Database.js:67:44
at IncomingMessage. (node_modules/dynamo/lib/Request.js:38:43)

require table schema

currently, tables are specified by name only, and the schema is inferred at query time.

it would probably be better to avoid issues like #9 by making table schemas mandatory, so that queries that violate the schema fail before execution.

support BatchWriteItems in the high-level API

this API was added last week, and only exists in the low-level API for now.

writing multibyte characters creates invalidserviceexceptions

Writing a character like 人

raises this error from dynamodb

com.amazon.coral.service#InvalidSignatureException: The request signature we calculated does not match the signature you provided.

High level Api no longer usable?

I installed everything last week and it worked like a charm, today I run the npm install dynamo in a new laptop and the same code is no longer running.
If I run

var dynamo = require("dynamo"), 
       client = dynamo.createClient({
        accessKeyId: "xxxxx",
        secretAccessKey: "xxxxxx"
    }),
    db = client.get("ap-northeast-1");

client.get is raising this error

../dynamo/node_modules/dynamo-client/index.js:247
throw new Error("No secret access key available.")
^
Error: No secret access key available.

So I was playing around with the different options and I found out that if I write

var dynamo = require("dynamo"), 
       client = dynamo.createClient("ap-northeast-1",
        {
        accessKeyId: "xxxxx",
        secretAccessKey: "xxxxxx"
    })

It works. This is the way dynamo-client is creating the object, so I'm not sure if the High level Api is still usable.

Also if I run

client = dynamo.createClient("ap-northeast-1",
        {
        accessKeyId: "xxxx",
        secretAccessKey: "xxxx"
    }),
    db = client.get("ap-northeast-1");

It will complain with this error

db = client.get("ap-northeast-1");
^
TypeError: Object # has no method 'get'

So looks like some of the methods are no longer available
Am I doing something wrong?

Thanks for the hard work!

Predicates + Update.when()

I tried to use the method item.when(predicates).
In every version passed the predicate object i could not generate a valid conditional request like:

Expected: { id: { Exists: false } }

So i check your code and found out you forcing the value of the predicate to an array. Predicate.js:39

So it's only possible to get the operator = NE or EQ, witch where necessary to the item.when(), if you pass something like this:

new Predicate( { id: { "!=": [ undefined ] } } ) // a like bit weird ;-)

Now the problem

Within the Update.js:28 you check for null in combination with NEor EQ. I think that this can never be reached with the current predicate implementation.

A small rewrite to make it possible to generate a valid conditional request Update.js:28:

 if (value === null || value === undefined) {

But this is not really a nice solution.

Can you check the method item.when() for conditional requests or can you tell me how to use them.

Update with an attribute set to zero does not work

If you try to add 0 (zero) via an update command, you get the exception:

com.amazon.coral.validate#ValidationException: One or more parameter values were invalid: Only DELETE action is allowed when no attribute value is specified

I think the bug is in Update.js, line 60 when you check if "value" is true. Zero is false, so the value is never set.

Thanks for the great library!