continuumio / dbadapter

SQL database adapters forked from IOPro

DBAdapter

Database adapters forked from IOPro

DBAdapter is a Python module containing optimized data adapters for importing data from a variety of database sources into NumPy arrays and Pandas DataFrame. Database adapter leverages an optimized pyodbc module for accessing any relational database that supports the ODBC interface (SQL Server, PostgreSQL, MySQL, etc).

Build Requirements

Building DBAdapter requires a number of dependencies. In addition to a C/C++ dev environment, the following modules are needed, which can be installed via conda:

• NumPy
• Pandas
• unixodbc 2.3.4 (C lib, Linux only)

Building Conda Package

Note: If building under Windows, make sure the following commands are issued within the Visual Studio command prompt for version of Visual Studio that matches the version of Python you're building for. Python 2.6 and 2.7 needs Visual Studio 2008, Python 3.3 and 3.4 needs Visual Studio 2010, and Python 3.5 needs Visual Studio 2015.

1. Build DBAdapter using the following command:

   conda build buildscripts/condarecipe --python 3.5
   

2. DBAdapter can now be installed from the built conda package:

   conda install dbadapter --use-local
   

Building By Hand

Note: If building under Windows, make sure the following commands are issued within the Visual Studio command prompt for version of Visual Studio that matches the version of Python you're building for. Python 2.6 and 2.7 needs Visual Studio 2008, Python 3.3 and 3.4 needs Visual Studio 2010, and Python 3.5 needs Visual Studio 2015.

For building DBAdapter for local development/testing:

1. Install most of the above dependencies into environment called 'dbadapter':

   conda env create -f environment.yml
   

   Be sure to activate new dbadapter environment before proceeding.

2. Build DBAdapter using Cython/distutils:

   python setup.py build_ext --inplace
   

Testing

Tests can be run by calling the dbadapter module's test function. By default only the TextAdapter tests will be run:

TODO: The pyodbc tests live in the dbadapter/pyodbc/test* directories. They are not being run (yet).

python -Wignore -c 'import dbadapter; dbadapter.test()'

(Note: numpy.testing might produce a FurtureWarning that is not directly relevant to these unit tests).

Related projects

• TextAdapter (CSV, JSON, etc): https://github.com/ContinuumIO/TextAdapter
• PostgresAdapter (PostgreSQL): https://github.com/ContinuumIO/PostgresAdapter
• AccumuloAdapter (Apache Accumulo): https://github.com/ContinuumIO/AccumuloAdapter
• MongoAdapter (MongoDB): https://github.com/ContinuumIO/MongoAdapter

dbadapter's Issues

Fix unit tests

As a follow-up to PR #1, we should do the following to get this repo into more of a production-ready state:

• Get the unit tests (under the dbadapter/pyodbc/tests* directories) working
• Add instructions to README.md on how to setup local databases for testing
• Integrate the unit tests into the run_test.py script, after filling out the test function in dbadapter/__init__.py
• Remove TODO lines related to unit tests (for example, in top-level README.md and run_test.py)

Test compatibility among Python versions and OSes

As a follow-up to PR #1, we should do the following to get this repo into more of a production-ready state:

• Test Python v2.7 compatibility
• Test Windows compatibility, Python 3.5 and 2.7
• Test MacOS compatibility, Python 3.5 and 2.7

This issue should be addressed after #2 so that we can rely on the tests to identify breakage.