pborreli / dynamodbtocsv

Dump DynamoDB data into a CSV file

AWS DynamoDBtoCSV

This application will export the content of a DynamoDB table into CSV (comma-separated values) output. All you need to do is update config.json with your AWS credentials and region.

The output is comma-separated and each field is enclosed by double quotes ("). Double quotes in the data as escaped as "

This software is governed by the Apache 2.0 license.

Usage

typically, to use you'd run:

node dynamoDBtoCSV.js -t Hourly_ZEDO_Impressions_by_IP > output.csv

or even:

node dynamoDBtoCSV.js -t Hourly_ZEDO_Impressions_by_IP -f output.csv

to export to CSV

Use -d to describe the table prior so you can have an idea of the number of rows you are going to export

node dynamoDBtoCSV.js -t Hourly_ZEDO_Impressions_by_IP -d

to get some information about the table.

Full syntax is:

node dynamoDBtoCSV.js --help
	Usage: dynamoDBtoCSV.js [options]

Options:

  -h, --help                            output usage information
  -V, --version                         output the version number
  -t, --table [tablename]               Add the table you want to output to csv
  -i, --index [indexname]               Add the index you want to output to csv
  -k, --keyExpression [keyExpression]   The name of the partition key to filter results on
  -v, --KeyExpressionValues [value]     The expression for filtering on the primary key
  -S, --select [list of fields]         The list of fields to select on
  -c, --count                           Only get count, requires -pk flag
  -a, --stats [fieldname]               Gets the count of all occurances by a specific field name 
                                        (only string fields are supported presently)
  -e, --endpoint [url]                  Endpoint URL, can be used to dump from local DynamoDB
  -f, --file [file]                     Name of the file to be created
  -d, --describe                        Describe the table
  -p, --profile [profile]               Use profile from your credentials file
  -ec --envcreds                        Load AWS Credentials using AWS Credential Provider Chain

Pre-requisites

You'll need to install a few modules, including:

• aws-sdk
• commander
• dynamodb-marshaler
• papaparse

npm install

should do it.

Example output

"HashOf10","DateIPAdID","adcount"
"37693cfc748049e45d87b8c7d8b9aacd","2013011720024058205168000000010002","1"
"37693cfc748049e45d87b8c7d8b9aacd","2013011720050084232194000000010002","1"

Advanced queries

Output a selection of columns

node dynamoDBtoCSV.js -t my-table -i rule_type_id_index -k "rule_type_id = :v1" -v "{\":v1\": {\"S\": \"my_primary_key_valye\"}}" -s "rule_type_id, created_by" -r us-west-2

Output stats

node dynamoDBtoCSV.js -t my-table -i rule_type_id_index -k "rule_type_id = :v1" -v "{\":v1\": {\"S\": \"my_primary_key_valye\"}}" -s "rule_type_id, created_by" -r us-west-2 -a created_by