Solving for large stage depths about dbt-trino

[hovaesco]

The stage depth in starburst is currently set to 150

This could be changed using https://trino.io/docs/current/admin/properties-query-management.html#query-max-stage-count. What is a sample stage count you are getting?

Use this to build a graph of stages, find inflection points of stage accumulation or when near limit.

How to build something like that and how to use it?

[wjhrdy]

We are getting around 250 for stage depth, prior to materialization.

[wseaton]

@hovaesco here's an example of a dev tool we've made internally: https://gist.github.com/wseaton/0e0cfecb7421bbba6792222e8fec7cf6.

There are some assumptions that it makes about the plan output that might not hold true in practice (the bit about "virtual" stages, for one), but it's been helpful for us in directionally identifying nodes in the dbt graph that contribute to sudden explosions in complexity.

If we could get the direct number of stages from the query planner, and/or more insight into how the plan is generated that would be helpful.

I'm also trying to think of ways to show this to users easier, not sure what makes the most sense. For now, a rendered graph seems to work. We are thinking about embedding this in CI tooling or potentially running this periodically to help as a diagnostic tool.