debussy-labs / debussy_concert Goto Github PK

PR #28 addresses a quick solution

As a long-term solution we should add a is_metadata: bool (default to False) attribute to the schema file.

I will add more info to this later

When loading an empty parquet file into BigQuery (through the job API) using the STRING type for all fields, some fields gets parsed as INTEGER, resulting in the following error:

BigQuery error in load operation: Error processing job '[project]:[job]': Provided Schema does not match Table [project]:[dataset].[table]. Field [attribut] has
changed type from STRING to FLOAT

Relates to this stackoverflow issue.

Environment details

• Environment: Google Cloud Composer 1.17.9
• Airflow version: 2.1.4

Steps to reproduce

1. Create an empty parquet file
2. Create a BigQuery table with the same schema of the previous file, but with all fields as STRING
3. Create a data pipeline using the DataIngestionBase Composition, to read the file and load it into the BigQuery table

The problem:
Currently, our examples are heavily dependent on dag_run.start_date as the metadata used for _ingestion_ts
And this add an issue that we can't have a partial reprocessing as the dag_run.start_date will be the start date of the reprocessing run and the framework wont be able to correctly find the parquet files on the currect directory _logical_ts={{ execution_date }}/_ingestion_ts={{ dag_run.start_date }}

Parameter extract_query_from_temp from Reverse ETL composition does not support setting a .sql file.
This query need to get the value of reverse_etl_table_uri that is not passed/parsed by jinja and is internally replaced using f-string.
The solution might be to pass an user defined macro to be used in a jinja templating (see user_defined_macros in https://www.astronomer.io/guides/templating/)