bufferapp / rsdf

Some functions to help Pandas DataFrames communicate with Redshift

RSDF

Set of utils to connect Pandas DataFrames and Redshift. This module will add a new function to the DataFrame object. Inspired by josepablog gist.

Installation

To install rsdf, simply use pip:

$ pip install rsdf

If you were using the older version, you can also install it with pip:

$ pip install git+git://github.com/bufferapp/rsdf.git@d1a5feca220cef9ba7da16da57a746dfb24ee8d7

Usage

Once rdsf is imported, the DataFrame objects will have new functions:

import pandas as pd
import rsdf


engine_string = 'redshift://user:password@endpoint:port/db'

users = pd.read_sql_query('select * from users limit 10', engine_string)

users['money'] = users['money'] * 42

# Write it back to Redshift
users.to_redshift(
    table_name='users',
    schema='public',
    engine=engine_string,
    s3_bucket='users-data',
    s3_key='rich_users.gzip',
    if_exists='update',
    primary_key='id'
)

Alternatively, if no engine is provided, the rsdf module will try to figure out the engine string from the following environment variables:

• REDSHIFT_USER
• REDSHIFT_PASSWORD
• REDSHIFT_ENDPOINT
• REDSHIFT_DB_NAME
• REDSHIFT_DB_PORT

Since rsdf uploads the files to S3 and then runs a COPY command to add the data to Redshift you'll also need to provide (or have them in the environment loaded) these two AWS variables:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY

License

MIT © Buffer

rsdf's Issues

Hide credentials from exceptions

Add docstrings

Document functions to describe their parameters and what they do

Add tests

These could be integration tests that actually hit redshift.

Add S3 cleanup

After uploading the data to Redshift, we can delete the S3 object.