- Clone repo
- Install docker + docker-compose
- Run
docker-compose up -d
- Run
bash superset_init.sh
- Done! Checkout the service endpoints:
Trino: http://localhost:8080/ui/
(username can be anything)
Minio: http://localhost:9001/
(username: minio_access_key
, password: minio_secret_key
)
Superset: http://localhost:8088/
(username: admin
, password: admin
)
AirFlow: https://localhost:9090/
(username: admin
, password: airflow
)
- Go to
data
dropdown and clickdatabases
- Click the
+ database
button - For
Select a database to connect
choosepresto
- In
SQLALCHEMY URI
puttrino://hive@trino-coordinator:8080/hive
- Switch over to
Advanced
tab - In
SQL Lab
select all options - In
Security
selectAllow data upload
docker exec -it trino-hive-superset-docker_trino-coordinator_1 trino
Upload parquet file on MinIO bucket datalake and run commands:
CREATE SCHEMA IF NOT EXISTS hive.LPD_datasets_metadata
WITH (location = 's3a://datalake/');
# Path s3a://datalake is the holding directory. We dont give full file path. Only parent directory
CREATE TABLE IF NOT EXISTS hive.LPD_datasets_metadata.LPD_datasets_metadata (
img_name VARCHAR,
size_img INTEGER,
img_w INTEGER,
img_h INTEGER,
area_img INTEGER,
x_min DOUBLE,
y_min DOUBLE,
x_max DOUBLE,
y_max DOUBLE,
bbox_w DOUBLE,
bbox_h DOUBLE,
area_bbox DOUBLE,
xmin_scale DOUBLE,
ymin_scale DOUBLE,
xmax_scale DOUBLE,
ymax_scale DOUBLE,
area_scale DOUBLE,
bbox_wscale DOUBLE,
bbox_hscale DOUBLE
)
WITH (
external_location = 's3a://datalake/',
format = 'PARQUET'
);
When we need to run and set up the tasks, we need to attach to AirFlow's container and install the corresponding lib