yegor256 / dynamo-archive

Archive and Restore DynamoDB Tables, from the Command Line

There are two simple Node.js scripts that archive and restore an entire AWS Dynamo DB table in JSON format.

Install it first (I assume you have Node.js and Npm installed already):

$ npm install -g dynamo-archive

Create a user in Amazon IAM and assign a policy to it (how?):

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["dynamodb:Scan", "dynamodb:DescribeTable"],
      "Resource": "arn:aws:dynamodb:us-east-1:019644334823:table/test"
    }
  ]
}

Where 019644334823 if your AWS account number, us-east-1 is AWS region, and test is the name of your Dynamo DB table (can be a *, if you grant access to all tables).

Run it first without arguments and read the output:

$ dynamo-archive.js

To restore a table from a JSON file run:

$ dynamo-restore.js

Crontab automation

I'd recommend to use this simple bash script to automate backups of your Dynamo DB tables and save them to S3 (I'm using s3cmd):

#/bin/bash

AWS_ACCESS_KEY_ID=AKIAJK.......XWGA5AA
AWS_SECRET_ACCESS_KEY=7aDUFa68GN....................IGcH0zTf3k
#optional endpoint for DynamoDB local
AWS_DYNAMODB_ENDPOINT=http://localhost:8000/
declare -a TABLES=(first second third)
for t in ${TABLES[@]}
do
  dynamo-archive/bin/dynamo-archive.js --table=$t > $t.json
  s3cmd --no-progress put $t.json s3://backup.example.com/dynamo/$t.json
  rm $t.json
done

How to contribute

Read these guidelines. Make sure you build is green before you contribute your pull request. You will need to have NodeJS and installed. Then:

$ npm install
$ npm test

If it's clean and you don't see any error messages, submit your pull request.

dynamo-archive's Issues

aws iam role

could you please implement the same mechanism that aws cli employs in order to retrieve and use temporary credentials assigned to the instance by use of iam roles (key, secret, token) ?

here's an example output :

curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-role-name

{
  "Code" : "Success",
  "LastUpdated" : "2016-02-07T07:54:27Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "ASIAXXXXXXXXXXXXXXXXXXXX",
  "SecretAccessKey" : "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
  "Token" : "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX==",
  "Expiration" : "2016-02-07T14:16:27Z"
}

this way we can all use your tool in complex automatic deployment scenarios that make use of aws cloudformation without embedding credentials in any script...which i think is very useful as a new feature.

npm install dynamo-archive fails

I have a brand new installation of node and npm on OS X 10.10.4

$ uname -a
Darwin macbook 14.4.0 Darwin Kernel Version 14.4.0: Thu May 28 11:35:04 PDT 2015; root:xnu-2782.30.5~1/RELEASE_X86_64 x86_64
$ node -v
v0.12.7
$ npm -v
2.12.1

When I run npm install dynamo-archive I get the following messages. I cannot figure out why it does not work, and I am a node novice so I am not sure where to go next.

$ npm install dynamo-archive

[email protected] install /Users/my_user/node_modules/dynamo-archive/node_modules/sleep
node build.js || nodejs build.js

child_process: customFds option is deprecated, use stdio instead.
gyp: dynamo-archive not found (cwd: /Users/my_user/node_modules/dynamo-archive/node_modules/sleep) while trying to load dynamo-archive
gyp ERR! configure error
gyp ERR! stack Error: gyp failed with exit code: 1
gyp ERR! stack at ChildProcess.onCpExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:355:16)
gyp ERR! stack at ChildProcess.emit (events.js:110:17)
gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:1074:12)
gyp ERR! System Darwin 14.4.0
gyp ERR! command "node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild" "dynamo-archive"
gyp ERR! cwd /Users/my_user/node_modules/dynamo-archive/node_modules/sleep
gyp ERR! node -v v0.12.7
gyp ERR! node-gyp -v v2.0.1
gyp ERR! not ok
[sleep]: Error: Failed to execute 'node-gyp rebuild dynamo-archive' (1)
sh: nodejs: command not found
npm ERR! Darwin 14.4.0
npm ERR! argv "node" "/usr/local/bin/npm" "install" "dynamo-archive"
npm ERR! node v0.12.7
npm ERR! npm v2.12.1
npm ERR! file sh
npm ERR! code ELIFECYCLE
npm ERR! errno ENOENT
npm ERR! syscall spawn

npm ERR! [email protected] install: node build.js || nodejs build.js
npm ERR! spawn ENOENT
npm ERR!
npm ERR! Failed at the [email protected] install script 'node build.js || nodejs build.js'.
npm ERR! This is most likely a problem with the sleep package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node build.js || nodejs build.js
npm ERR! You can get their info via:
npm ERR! npm owner ls sleep
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR! /Users/my_user/npm-debug.log

How about incremental backups/restores?

This script is useless for large tables with gazillions of records, it would be nice to have a incremental backups and restores. There is already solutions like https://github.com/mapbox/dynamodb-replicator but it's not well documented and requires some other tools for deployment.

Memory leak in restore script

I'm running restore script on a modest 18Mb file and it throws an exception after a while:

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [/usr/local/bin/node]
2: node::FatalException(v8::Isolate*, v8::Localv8::Value, v8::Localv8::Message) [/usr/local/bin/node]

I assume it's a memory leak?

updated dependencies

Estimate project size

There is no estimate in the project yet. We have to create one as explained here

puzzles

This is driving me nuts. Great apologies for wasting an issues on this. How do you generate PUZZLES.md? My Google foo is completely failing me.

Indie tech review no.1

Review our existing source code and all other technical artifacts and report at least 8 bugs. More about such reviews you can read here. The task will be closed once we have that bugs reported and classified. Don't forget to add itr tag to each of them.

Error with dynamo-archive.js

Hi,

I am running:
"node dynamo-archive.js ... "

After a few lines of output, I get the error:
Error: TypeError: Object # has no method 'translate'

I do not know how to trouble-shoot that. Thank you for your assistance.

reading access keys from awscli config?

Hello again,

wouldn't it be more secure and convenient if the script could read access keys from standard awscli config (perhaps even including the profile feature?)

0.2

Please, publish a new release 0.2 (or higher). Don't forget to update us here regularly (at least once a week), as explained in this article.

dynamo-restore bombs out during restore attempt

when attempting to restore a previously backed up table, restore operation throws an exception, not immediately obvious what is wrong here:

$ ./dynamo-restore.js --table development.main.clans_b < ~/fs.main.clans-150316 
{"data":{"B":{"type":"Buffer","data":[120,156,171,86,202,205,47,73,81,178,82,82,170,5,0,22,50,3,111]}},"clan_id":{"N":"251"}}
{ [InvalidParameterType: Expected params.Item['data'].B to be a string, Buffer, Stream, Blob, or typed array object]
  message: 'Expected params.Item[\'data\'].B to be a string, Buffer, Stream, Blob, or typed array object',
  code: 'InvalidParameterType',
  time: Tue Mar 15 2016 13:55:12 GMT+0000 (UTC) } 'InvalidParameterType: Expected params.Item[\'data\'].B to be a string, Buffer, Stream, Blob, or typed array object\n    at ParamValidator.fail (/opt/dynamo-archive/node_modules/aws-sdk/lib/param_validator.js:50:37)\n    at ParamValidator.validatePayload (/opt/dynamo-archive/node_modules/aws-sdk/lib/param_validator.js:251:10)\n    at ParamValidator.validateScalar (/opt/dynamo-archive/node_modules/aws-sdk/lib/param_validator.js:133:21)\n    at ParamValidator.validateMember (/opt/dynamo-archive/node_modules/aws-sdk/lib/param_validator.js:94:21)\n    at ParamValidator.validateStructure (/opt/dynamo-archive/node_modules/aws-sdk/lib/param_validator.js:75:14)\n    at ParamValidator.validateMember (/opt/dynamo-archive/node_modules/aws-sdk/lib/param_validator.js:88:21)\n    at ParamValidator.validateMap (/opt/dynamo-archive/node_modules/aws-sdk/lib/param_validator.js:117:14)\n    at ParamValidator.validateMember (/opt/dynamo-archive/node_modules/aws-sdk/lib/param_validator.js:92:21)\n    at ParamValidator.validateStructure (/opt/dynamo-archive/node_modules/aws-sdk/lib/param_validator.js:75:14)\n    at ParamValidator.validateMember (/opt/dynamo-archive/node_modules/aws-sdk/lib/param_validator.js:88:21)'
readline.js:925
            throw err;
            ^

InvalidParameterType: Expected params.Item['data'].B to be a string, Buffer, Stream, Blob, or typed array object
    at ParamValidator.fail (/opt/dynamo-archive/node_modules/aws-sdk/lib/param_validator.js:50:37)
    at ParamValidator.validatePayload (/opt/dynamo-archive/node_modules/aws-sdk/lib/param_validator.js:251:10)
    at ParamValidator.validateScalar (/opt/dynamo-archive/node_modules/aws-sdk/lib/param_validator.js:133:21)
    at ParamValidator.validateMember (/opt/dynamo-archive/node_modules/aws-sdk/lib/param_validator.js:94:21)
    at ParamValidator.validateStructure (/opt/dynamo-archive/node_modules/aws-sdk/lib/param_validator.js:75:14)
    at ParamValidator.validateMember (/opt/dynamo-archive/node_modules/aws-sdk/lib/param_validator.js:88:21)
    at ParamValidator.validateMap (/opt/dynamo-archive/node_modules/aws-sdk/lib/param_validator.js:117:14)
    at ParamValidator.validateMember (/opt/dynamo-archive/node_modules/aws-sdk/lib/param_validator.js:92:21)
    at ParamValidator.validateStructure (/opt/dynamo-archive/node_modules/aws-sdk/lib/param_validator.js:75:14)
    at ParamValidator.validateMember (/opt/dynamo-archive/node_modules/aws-sdk/lib/param_validator.js:88:21)

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update dependency aws-sdk to 2.1681.x
• Update dependency dotenv to v16
• Update dependency proxy-agent to v6
• Update dependency tape to v5
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

github-actions

.github/workflows/npm.yml

• actions/checkout v4
• actions/setup-node v4

.github/workflows/pdd.yml

• actions/checkout v4
• ubuntu 22.04

.github/workflows/xcop.yml

• actions/checkout v4
• ubuntu 22.04

npm

package.json

• aws-sdk 2.814.x
• dotenv 0.5.x
• minimist 1.2.x
• proxy-agent ^2.0.0
• sleep 6.3.x
• tape ~2.14.0
• faucet 0.0.x
• dynalite 3.2.x

---

• Check this box to trigger a request for Renovate to run again on this repository

Old version of AWS-SDK

Is there any reason why this module is tied to an old version of the AWS-SDK?

I've only tested the archiving functionality (not the restoring) but it seems to work OK.

Current version: 2.2.33
Semver range for dynamo-archive: 2.0.x

Is there easy way to precreate DynamoDB table for the restore process

Hi,

I tried to run dynamo-restore.js in a different region, and I get an error that the table does not exist. I was hoping the backup script would save the table definition and create it upon restore. :-) Is there any other easy way to pre-create the table?

ResourceNotFoundException: Requested resource not found: Table: Device2 not found

Thank you

rate-limiting the scan?

Have you considered a way to rate-limit the scan to a specified portion of available Read capacity?

Not pulling at full provisioned rate, --rate not appearing to modify the rate at which data is being pulled

I've tried different --rate from 50 to 1000, nothing seems to change. The best I get is ~50% of max.

deploy via rultor doesn't work

report correctly when table is not found

Now it fails like this:

TypeError: Cannot read property 'Table' of null
    at Response.<anonymous> (/home/ubuntu/backup/dynamo-archive/dynamo-archive.js:67:25)
    at Request.<anonymous> (/home/ubuntu/backup/node_modules/aws-sdk/lib/service.js:123:18)
    at Request.callListeners (/home/ubuntu/backup/node_modules/aws-sdk/lib/sequential_executor.js:132:20)
    at Request.emit (/home/ubuntu/backup/node_modules/aws-sdk/lib/sequential_executor.js:100:10)
    at Request.emitEvent (/home/ubuntu/backup/node_modules/aws-sdk/lib/request.js:392:10)
    at Request.failRequest (/home/ubuntu/backup/node_modules/aws-sdk/lib/request.js:353:12)
    at Request.<anonymous> (/home/ubuntu/backup/node_modules/aws-sdk/lib/request.js:339:32)
    at Request.<anonymous> (/home/ubuntu/backup/node_modules/aws-sdk/lib/request.js:396:20)
    at Request.callListeners (/home/ubuntu/backup/node_modules/aws-sdk/lib/sequential_executor.js:135:24)
    at Request.emit (/home/ubuntu/backup/node_modules/aws-sdk/lib/sequential_executor.js:100:10)

--rate setting does not work

params.limit -> params.Limit

refresh