bakdata / google-airflow-premier-league Goto Github PK

View Code? Open in Web Editor NEW

19.0 19.0 14.0 136 KB

Data Warehousing Made Easy with Google BigQuery and Apache Airflow

Dockerfile 15.45% Shell 50.46% Python 17.38% TSQL 16.71%

google-airflow-premier-league's People

Contributors

Stargazers

Watchers

Forkers

kiranvits myslxy iamalonso djoguns tycho-1 philipsales phani111 navkumar2245 kozakjefferson alcarazo99 dwaynekj pjinthehouse rbrto

google-airflow-premier-league's Issues

Help wanted about ext_schema parameter in the schema

Hi @MarcusBaitz, I would like to know what ext_schema parameter means in the schema definition. I've been searching on internet about that parameter but I have not found anything. Please your help.

broken dag -> premier_league_data_to_gc.py missing module util.

Great blog post. But I had consistent broken module with "util" missing.

error on Dataflow

Broken DAG: [/home/airflow/gcs/dags/dags/premier_league_data_to_gc.py] No module named 'util'

After modifying the code as follows, it works fine:

google_init.sh

echo "deploy airflow DAG's" && \
BUCKET_NAME=$(gcloud composer environments describe ${GC_PROJECT_ID} --location ${LOCATION} --format="get(config.dagGcsPrefix)")
BUCKET_NAME=${BUCKET_NAME%dags}

bash google_deploy.sh && \

gsutil -m cp -r ../google_deploy/dags/* ${BUCKET_NAME}

echo "your premium league environment is now ready and its project id is: ${GC_PROJECT_ID}"
...