This is an ELT process, were we have the data located in the bucket s3, and we are going to import it into snowflake.
In this ELT, we will follow this process:
- Connect to snowflake
- Create the databases (Landing, Production)
- Create a temporary stage to load the data
- Create a transient table, were we will load the data
- Create the production table
- Transform the data and load it into the production table
In order to be able to accomplish this, we require the following packages:
Package | Description |
---|---|
Jinja2 | To read a SQL template |
parse | To read the initial parameters |
snowflake-connector-python | To connect to snowflake |
pyyaml | To read tables parameters |
To be able to run this ELT in Docker, it is needed to run the following commands:
docker build -t etl:1.0 .
docker run --entrypoint python etl:1.0 main.py --user <username> --password <password> --account <account> --warehouse <warehouse> --file_format <file_format> --database <database> --schema <schema> --table <table> --bucket <bucket> --table_config <table_config>
The parameters needed in order to be able to run the ETL are:
Parameter | Description |
---|---|
--user |
User to connect to snowflake |
--password |
Password to connect to snowflake |
--account |
Snowflake account |
--warehouse |
Snowflake warehouse |
--file_format |
File format to create the Stage |
--database |
Destination Database |
--schema |
Destination Schema |
--table |
Destination Table |
--bucket |
Bucket URL |
--table_config |
Initial configuration of the table |