Student: Trung Nguyen Viet - Instructor: M. Eng Duc Tien Bui
Developed with Python 3.11.x, PostgresSQL 12.x - PG Admin 7.x(if needed), Apache Nifi 1.27.x, following pip dependencies packages as below.
-
Download pip package in terminal/command line:
pip install faker pandas configparser coloredlogs psycopg2 jupyter squarify seaborn scikit-learn
-
Type in command line/terminal
jupyter notebook
-
Run a Jupyter notebook file as usual
-
Update Database config and connection info in
dwh_pipelines/config.ini
-
Manually create databases with the same name as
config.ini
in your database instance -
Install dependencies:
pip install faker pandas configparser coloredlogs psycopg2 jupyter squarify seaborn scikit-learn
-
Run python script:
python gen_staging.py
create dimension tablepython gen_fact.py
generate fact table for data mart
This Example use the Inmon Approach in designing data warehouse, some details layer would not completely follow the method.
A simple data pipeline of Generating a file and Sending to local filesystem
- Create and Setting up first Processor of type GenerateFlowFile
- Create and Setting up f2nd Processor of type PutFile:
- Start 2 processor to run the pipeline
- Check Data Provenance
A simple data pipeline of type "Copy Activity" to transform from CSV to Azure SQL table
- Tạo tài nguyên Blob Storage trong 1 module quản lý Account Storage
- Tạo các Linked Service để kết nối đến các Dataset
- Tạo các Dataset Source và Sink(nguồn và đích) cho mỗi Activity
- Thêm Activity(Copy) và Import Mapping
Tham khảo: