starschema / antares-idl Goto Github PK

Once #16 is completed, create a Sink connector similar to Snowflake, but persist to an S3 bucket.

Currently the MSK broker and the Snowflake connector are deployed in the same file (deployment/aws/msk.py).

We should separate the broker and the connector into separate execution branches (see deployment/aws/main.py for execution branch concept)

We should refactor the Snowflake deployment the following way:

1. Have the database creation on the top level
2. Have the msk and the hvr schema creation in its own files.

Currently the Antares database creation and all MSK landing related scripts are in msk_snowflake.py.

Once the Snowflake deployment issue is completed , we should create an example project that

1. Reads a file from S3 and stores it record by record in Snowflake
2. Processes the file in DBT and creates a data-mart

Example data-file will be provided outside of this issue.

Originally I built all the MSK stack and Tamas built the EKS + kubernetes (HVR, etc) stack. We would need to be able to deploy these all in the same stack so they can work together.

I am attaching the config file for the aws deployment that deploys the MSK side of things. @tfoldi , can you add an example config here so we can start merging these?

Once #17 is completed, please create a Glue job and crawler deployment that

1. Regularly parses then deletes the JSON documents stored in the landing S3 bucket and creates a partitioned Athena table
2. The crawler regularly looks at the glue job's output and registers / updates the table in the Glue Data Catalog

Success criteria: I can log in to AWS Athena and query the table.