Takes a yaml file as input.
- stages:
- extract.postgres:
parameter:
sql: "01extract.sql"
conn:
parameter:
host: !Var dbhost
port: !Var dbport
dbname: !Var dbname
user: !Var dbuser
password: !Password dbuser
input: null
output: raw
In the example yaml above, you define each stages of the pipeline with three must have keys,
- parameter
- input
- output
This creates a node in the DAG, that is executable. yaml standard has been exxtended to make yaml parameterized. Either external parameters can be passed through an additional json file or some parameters like Date, and UUID can be generated.
run_id: !UUID
current_date: !Date today
would resolve at runtime as:
[('run_id', '9de49f84-f448-4f52-bad9-be41804be443'),
('current_date', '2021-10-17 07:32:14.249768')]
!Password tag uses keyring to resolve pasword given a user name.