villebro / sqltask

ETL tool for performing mostly SQL-based data transformation

Hi there 👋

sqltask's Issues

Possibility to append data to existing tables

In classical RDBM-systems data is written to existing tables. New table creation might be behind cumbersome user rights management processes.

To extend the usage scope of sqltask it would be good to have a following feature:

• insert to existing table appending existing data
  -example:
• data is written to table customer with data per 20191031
• table customer is appended with data per 20191131

It might be that this is already inside sqltask and it is just me and my limited capabilities in sqlalchemy :)

Add redis caching support

When developing ETL jobs, re-retrieving expensive OLAP queries and lookups can be quite time consuming and cause unnecessary billing on cloud DWs. Being able to cache queries based on the connection string + select statement would drastically improve performance and limit unnecessary DW load.

Add CI + unit/integration tests

Start with mypy linting, then add first round of unit and integration tests.

Better instructions on examples

Examples have great code example but user instructions on how to run example are missing. Create markdowns on how-to-run-stuff.