'bigquery' key error when running compare-reports fails to produce diff summary about piperider

[DaveFlynn]

@Stochastic-Squirrel Thanks for reporting this issue. The team is taking a look and we'll get back to you shortly

[Stochastic-Squirrel]

Hi Dave, thanks for reaching out!

I have been using the --debug flag throughout and nothing is printed to the console. Unfortunately I haven't been able to glean any more info!
I tried the website now and I experienced a server error when attempting uploads
Sentry event id: 229c6e399c2b48d08f61682dff0ac69a
I have tried uploading a single manifest as well as two at the same time.

Unfortunately I don't feel comfortable sharing the manifests in their entirety. I'll try to cut them down to a minimal set.
What are the essential keys needed? I am thinking of maybe isolating a single table that does not contain any sensitive information.

However, I do have an update on what causes the error!
I experimented some more and I noticed that it is only when changing the model YAML files that the error occurs. Changes to the SQL models seem to work fine.

Here are some scenarios that causes the error

• Changing the data type of an existing column in prod array<string> to array
• Renaming an existing column in prod fruit to fruits_and_vegetables in staging in the model yaml
• Switching enforcement of contracts on and off has no effect
• Adding a new column to the model YAML in staging that is not in prod does raise an error

In my case, I have to tweak model yamls for SQL models that are affected by a change e.g. data type change.

I hope this makes it a bit easier to recreate the issue on your end.

[popcornylu]

@Stochastic-Squirrel

Hi, thanks for your information. I know there is some privacy concern for providing the real run.json. So Is it possible to provide the two run.json from a dummy project? It would help us to reproduce the issue.

Expected reproduce steps

1. Download your two run.json
2. run

   piperider compare-reports --base run_base.json --target run.json --output ./comparison_report
   

3. Get error result:

   Selected reports:
     Base:   run_base.json
     Target: run.json
   Warning:
   'bigquery'
   Got problem to generate changeset.
   Comparison report: 
   /data_warehouse/comparison_report/index.html
   

[DaveFlynn]

Hi @Stochastic-Squirrel

We're having some difficulty in reproducing the issue. We'll continue to look into this.

It looks like you already ran PipeRider with --debug ? If not, I would suggest that as a further debugging step. There may be a lot of output - If there's nothing sensitive in the output you could share that with us. Either attach it here, or email to [email protected]

In the meantime, you could try out an Impact Report manually by using the DBT Manifest Analyzer in PipeRider Cloud:

1. Log into PipeRider Cloud
2. Click the Analyze tab
3. Upload two manifest files into the Manifest Analyzer

I'll follow up on this when we have had more success reproducing the issue.

Thanks,

Dave

[Stochastic-Squirrel]

thanks @popcornylu. I'll try to reproduce this error in a dummy project as soon as I can!