edgui-appolonicorreia / pyroduct Goto Github PK

To develop Lambda caller using the Airflow lib.

Create two VPCs and 3 subnets for each one.

One VPC it's totally secure and has no external connection:

• Name: internal-datalake-vpc;
• 3 Subnets;
• Each subnet should have 251 IPs;

The other VPC should have an external connection:

• Name: external-datalake-vpc;
• 3 Subnets;
• Each subnet should have 251 IPs;

One route table with CIDR block "0.0.0.0./0" and linked with the external VPC.

To develop a dynamic DAG engine to serve the most purposes. Each DAG should have a yml config file.

Obs: Perhaps the yml config file could be an Airflow variable, but not for now.

The Pyroduct has 3 layers:

• Bronze;
• Silver;
• Gold;

The Bronze and Silver, are S3 Buckets and the Gold layer is a Postgres database.

Obs: in the future, I have plans to implement one or two more layers and use a Redshift database in the Gold layer, replacing the Postgres database.