alteryx / featuretools_sql

Automated creation of EntitySets from relational data stored in SQL databases

Automated creation of EntitySets from relational data stored in SQL databases

---

The featuretools_sql library allows you to directly import your relational data into Featuretools to run automated feature engineering.

Installation

Install with pip:

python -m pip install "featuretools[sql]"

or from the Conda-forge channel on conda:

conda install -c conda-forge featuretools

Example

Simply pass in the database connection information:

from featuretools_sql.connector import DBConnector

sql_connector = DBConnector(
    system_name = "mysql",
    host = "127.0.0.1:3306"
    user = "root",
    password = "password",
    database = "db"
) 
entityset = sql_connector.get_entityset()

The entityset object will have the relationships and DataFrames already populated, allowing you to call featuretools.DFS and run automated feature generation.

import featuretools as ft

feature_defs, feature_matrix = ft.dfs(
    entityset=entityset,
    target_entity='target_table_name'
)

We currently supports importing data from the following relational database systems:

• MySQL
• PostgreSQL
• Snowflake

Support

The Featuretools community is happy to provide support to users. Project support can be found in four places depending on the type of question:

1. For usage questions, use Stack Overflow with the featuretools tag.
2. For bugs, issues, or feature requests start a Github issue.
3. For discussion regarding development, use Slack.
4. For everything else, the core developers can be reached by email at [email protected]

Built at Alteryx

featuretools_sql is an open source project maintained by Alteryx. To see the other open source projects we’re working on, visit Alteryx Open Source. If building impactful data science pipelines is important to you or your business, please get in touch.

featuretools_sql's Issues

Fill out setup.cfg with proper package information

• Here is a good example
• https://github.com/alteryx/featuretools/blob/main/setup.cfg

Add .pre-commit-config.yaml to project

Add support for creating EntitySet from questDB

• questdb is an open source time-series database
• We should just be able to support this with no or minimal effort (as it is a Postgresql database)
• We may need to update our docs to specify this

Add GitHub action workflow for unit test with latest dependencies

• https://github.com/alteryx/featuretools/blob/main/.github/workflows/unit_tests_with_latest_deps.yaml

Add this package to conda-forge

• https://conda-forge.org/docs/maintainer/adding_pkgs.html

Add release github action workflow to upload package to PyPI

• https://github.com/alteryx/featuretools/blob/main/.github/workflows/release.yaml

Add minimum dependency checker

• https://github.com/alteryx/featuretools/blob/main/.github/workflows/minimum_dependency_checker.yaml

Add `select_only` parameters and update test cases

• Add a filter parameter that the user can use to specify which tables they want to import from. Update the test cases to add tests that cover this case.

Add lint check github action workflow

• https://github.com/alteryx/featuretools/blob/main/.github/workflows/lint_check.yaml

Release v0.0.3

• Need to release again to get Windows compatibility

Improve Postgres testing

-Add relationship information for retail dataset to Testing Postgres Database (Retail Demo for all)
-Test that number of output dataframes match input dataframes
-Test types match (this might fail because WW types aren't the same)
-Pull out EntitySet setup and tear down into a reproduceable mechanism that we can reuse in other tests
-Ensure relationship information matches

Add GitHub action workflow for unit test with minimum dependencies

• https://github.com/alteryx/featuretools/blob/main/.github/workflows/unit_tests_with_minimum_deps.yaml

release featuretools-sql v0.0.2

Add mixed-case column names to Postgres and MySQL CI

• We should verify that mixed-case column names are correctly handled when importing dataframes and relationships.

Write blog post

Write blog post introducting featuretools-sql library

Fix Pandas SQLAlchemy data type issues and resolve warnings

Pandas's IO library is throwing warnings when we use it with a non-SQLAlchemy connector object. Furthermore, there seems to be issues with snowflake.connector objects handling particular data types.

We should decide how we want to resolve this issue, either by working around it or switching to SQLAlchemy.

Move requirements into pyproject.toml

• All project requirements should be specified in pyproject.toml (including dev, test)

Determine what to do in event user has not defined primary key

• Suppose the user attempts to import tables where there is no primary key constraint explicitly defined. In this event, the code that creates the relationships data structure will fail. What is the best behavior in this scenario? For now, we can throw an informational warning and leave it to the user to set a primary key.

Clean up readme

• The readme should have the following
  • badges
  • a library image
  • example usage
  • benefits of using the library
  • links to documentation

Rename main folder to featuretools_sql

• Standard Python practice is to use underscore for folder names

Remove tensorflow requirements from setup.cfg

• These requirements should not be there

complete =
    tensorflow >= 1.14.0; sys_platform!="darwin" or platform_machine!='arm64'
    tensorflow-metal >= 0.4.0; sys_platform=="darwin" and platform_machine=='arm64'
    tensorflow-macos >= 2.8.0; sys_platform=="darwin" and platform_machine=='arm64'
    tensorflow_hub >= 0.4.0

Add Dask DataFrame compatibility

We should support converting the SQL tables into Dask DataFrames; we can use the functions detailed below:

https://docs.dask.org/en/stable/dataframe-sql.html

Add support for creating an EntitySet from timescaledb

• timescaledb is an open-source time-series SQL database
• We should just be able to support this with no or minimal effort (as it is a Postgresql database)
• We may need to update our docs to specify this

Add support for creating an EntitySet from influxdb

• influxdb is open source time series database
• This open source library has a python client that we can use to support creating an EntitySet
  • https://github.com/influxdata/influxdb-client-python

Add `select_only` parameter to `get_entityset` method

I should be able to use the select_only parameter when calling get_entityset, not just when calling populate_dataframes

Investigate using DuckDB instead of `read_sql_query`

We may be able to achieve significant performance boosts by replacing read_sql_query with DuckDB's API. For more info, refer here: https://duckdb.org/2021/05/14/sql-on-pandas.html

Use fakesnow to run Snowflake test locally

• We can use this library to fake a Snowflake instance (rather than using a live instance for testing)
• https://github.com/tekumara/fakesnow

Allow users to filter down to specifc set of tables from DB

Add MySQL testing with featuretools tests

• Add MySQL tests that spin up MySQL instance containing featuretools sample data, and allow user to run tests on that

Add `featuretools_sql` installation instructions to install.md in Featuretools

After releasing on conda-forge (#6) , we should update Featuretools's install.md to reflect the availability of featuretools_sql on conda-forge

Add `do_not_select` parameter

Suppose user has 100 tables, and wants to include all but n. It may be useful to have a do_not_select parameter instead of having to enumerate the 100-n tables they do want in the select_only argument.

Add latest dependency tracker to project to keep up to date

• When new dependencies are released we should test our package with those latest release
• https://github.com/alteryx/featuretools/blob/main/.github/workflows/latest_dependency_checker.yaml