Pass an (optional) list of table-names to generate_source about dbt-codegen

[pixie79]

Can we get this merged? - #51

It appears to be very useful, especially for automated pipelines. I could use it to add new sources as part of a loading pipeline. Hopefully dbt is happy with multiple files describing the source, either that or we would need to merge them somehow.

[rahulj51]

Hi, any chance if this making it to a release?

[dbeatty10]

Thank you @rahulj51 for implementing this feature 👏 #51 is merged and will be included in the next release.

@pixie79 thank you for affirming the value for your use-case and bringing up the topic of merging YAML files.

There's two main options for describing the source:

1. Multiple files
2. A single file

I think 1) is a legit option since dbt allows multiple YAML files with source info as long as the model names aren't duplicated.

However, 2) is probably the most common case, and @pixie79 correctly pointed out that they need to be merged somehow.

The merging solution I'm most attracted to is a some kind of Python script that can "full outer join" two YAML files together (and/or other types of join behavior). The thing that gives me pause:

• The two YAML libraries I've used before (ruamel.yaml and PyYAML) each left something to be desired and would possibly cause such a script to exhibit buggy behavior for the first N iterations (where N is an uncomfortably large number).

Until the magical day that such a script exists, the only remaining option is to do the merge by hand.