Add support for TaskGroups from within the Databricks Workflow about astro-provider-databricks

[RafaelCartenet]

Exactly this, thanks @jlaneve

[dimberman]

Hi @jlaneve @RafaelCartenet

I think this functionality already exists within the current feature set :).

Ultimately, you seem to want to have the abstraction of Airlfow task groups and the "single job run" of a DatabricksWorkflowTaskGroup. This should be doable by placing your Airflow task groups inside of a DatabricksWorkflowTaskGroup.

So if you have

dg = DatabricksWorkflowTaskGroup()
with dg:
    tg1 = TaskGroup()
    with tg:
        ...

That should accomplish what you want, where there is a single launch for multiple task groups.

The downside of this is that your entire DAG will be inside of a single task group. One solution could be to create a DatabricksWorkflowDAG that assumes the entire DAG is a single databricks workflow (cc: @jlaneve @tatiana for whether we want to prioritize that or not), but this should at least unblock the functionality (even if it is a bit clunky in the UI).

[RafaelCartenet]

@dimberman Very true!
Thanks a lot! Will try it out!
Have a good one