Color code different sources in the lineage graph about dbt-docs

[guy-adams]

+1 for being able to map colours to tags. One additional thought - may people (e.g. in an ingestion,curation,calcuation,consumption architecture) have clearly defined 'layers' or groupings (e.g. staging, intermediate marts, core marts) of models. To be able to force the models into virtual columns (with a title) would be fantastic e.g.

I'm already denoting these layers using model tags.

[ptc-sgauglitz]

Hi -- would like to upvote both of these feature requests (ability to color code, and ability to set columns for designated layers). This would be super useful, including for reviews such as demoed in your own video here: https://www.youtube.com/watch?v=5W6VrnHVkCA&list=PL0QYlrC86xQmPf9QUceFdOarYcv3ETSsz

(also raised in dbt's Slack community)

Thoughts?

[MartinGuindon]

I agree with @yarodmi, it would be really useful to be able to define/override color rules.

I really like the idea of being able to define background colors and border colors for the nodes. I'd certainly use it to differentiate sources, staging models and fact/dimension tables. I could easily see this as a simple parameter in the source or dbt_project YAML files, eg:

schema.yml

sources:
  - name: retently
     docs_background_color: BFBFC0
     docs_border_color: 3346F3
     ..other source config..

  - name: gsheets
    docs_background_color: BFBFC0
    docs_border_color: 267C0B
    ..other source config..

dbt_project.yml

models:
  my_project:
    marts:
      docs_background_color: 10C3C3

    staging:
      docs_background_color: 4F0BC4

I think it would require being able to set background color, text color and border color at the minimum.

[yarodmi]

Hi Team,

I really think this will be a very useful feature.
Thou, I would suggest providing the ability to apply custom colour coding (rather than just different sources) as there might be different use cases.

Few examples where I would love to use it :

• quite often I am having multiple database schemas representing different architecture layers. Would be great to see them(schemas) in different colours so it is clear whats where
• highlight facts and dimensions in different colors
• highlight different materializations (what is a view and what is a table for example). We often start from 'everything as a view' and then deciding what models are useful to materialize. Seeing this in a graph will be very useful.

To simplify the adoption, might be reasonable to pre-define some colours for some tags (config(tags=["dimension"]) or config(tags=["fact"]) config(tags=["base"]) plus the ability to define a custom mapping.

Thanks in advance!

[jtcohen6]

In v1.3, a custom node_color config will be supported for all models, seeds, snapshots, and tests: https://docs.getdbt.com/reference/resource-configs/docs#custom-node-colors

We chose not to extend it to sources, exposures, and metrics for now. Still, that capability should go a way toward enabling a number of the desires in this thread!

[bcolbert978]

Reposting here from the dbt Slack #suggestions channel per Drew's advice - another extension of the lineage graph color-coding I'd like to see (not related to sources, but rather the search filter). When I filter down to specific --models it would be helpful if the model I name in my search were lit up (kind of like things light up purple when you select them). I often use the +model+ or @model searches here pretty frequently and sometime there are too many models loading in to stay grounded around my search.

[diegocamelo92]

I also think this could be very valuable, it would help us identify regions of different sources.

[JC-Lightfold]

@drewbanin - columns and colours would be close to my HIGHEST priority right now. Why? Because we're teaching every client how to live/breathe/think DBT and those two things help the DAG be the central unifying concept of the whole. Without them, any moderate sized DAG becomes fairly useless for understanding as a whole.

[PowdyPowPow]

@drewbanin i guess it could become very colorful and you would need to choose "easy" colors. But as currently the sources get distributed all over the graph it is already colorful with just the green. Maybe if you could somehow cluster them together it would probably also be easier to use some more discreet common element like colored border or just a different shades of grey? Or maybe you could group the different source tables together and put them in a (colored) box/container if they were all in the same place?

[ngouass]

Hello @everyone, I came accross this issue one week ago while looking for a solution to improve the lineage view of my project.

After unsucessful searchs, I'm planning to explore some solutions that I could build in an external tool. Before starting any development, I would like to know if some participants are still looking for a solution, if yes please let me know with A VOTE on this comment.

Below are potentital features, I was able to extract based on comments in this issue:
-> (7) apply custom color coding
- highlight database schemas
- highlight facts and dimensions in different colors
- highlight different materializations (what is a view and what is a table for example).
- Apply color based on tags
-> (46) Group model by layers and Highlight layers
-> (5) Search functionality with model highlighted when matching search query

Thanks.

[drewbanin]

@PowdyPowPow cool idea! I think it might look kind of wonky to color every source differently... maybe that's just something that we need to experiment with. It could be interesting to tone down the color (like use gray instead of green, or something) and then use a differently colored border, for instance.

What do you think?

[tnightengale]

@drewbanin Hey Drew! Hope all is well :) this issue seems quite stale but it's something that our team would love to have. Any suggestions on where to start in the code base to make a contribution to this?

Specifically, implementing alignment "zones" based on tags like in @guy-adams drawing above!

[yu-iskw]

It would be important to visually distinguish nodes by layers. At the same time, I would like to cluster nodes by schema and database in terms of dbt, because our BigQuery tables exist in different google cloud projects and datasets. That would enable us to easily manage and audit dbt models.

[saraleon1]

+1 dbt Cloud user

They would like to visually represent the modelling layers as different color nodes. Ex: stg models are represented by blue vs intermediate represented by orange etc. This is to make differentiating the different types of models easier within the DAG visualization.

[saraleon1]

Great news - thanks Jeremy!!

[belasobral93]

+1 I would like to visually represent Databrick's medallion architecture

[serene-capybara]

In v1.3, a custom node_color config will be supported for all models, seeds, snapshots, and tests: https://docs.getdbt.com/reference/resource-configs/docs#custom-node-colors

We chose not to extend it to sources, exposures, and metrics for now. Still, that capability should go a way toward enabling a number of the desires in this thread!

How does one use node_color on seed nodes?

[patriklundberg]

Would it be possible to change the text color of the node name?

[dbeatty10]

How does one use node_color on seed nodes?

@serene-capybara to use node color on seeds, you'd do something like this in your dbt_project.yml:

seeds:
  +docs:
    node_color: "#cd7f32"

Or something like this within a YAML file like models/_seeds.yml:

seeds:
  - name: my_seed
    docs:
      node_color: purple

[dbeatty10]

Would it be possible to change the text color of the node name?

@patriklundberg changing the text color is not possible, but #408 has some discussion of how to achieve higher contrast between the text and the background color.

[ptc-sgauglitz]

Hi @ngouass

Color coding is now supported since dbt 1.3 (see @jtcohen6 's Sep 6, 2022 post); I believe that can cover items 1-5 on your list (using either tags or model file/pathnames as intermediary to apply the color coding to as needed). I've used it to indicate the different materializations (your item 4).

Personally, I'd still be very interested in options to improve the layouting, in particular the ability to have models assigned to certain columns, to represent the layers -- very nicely visualized by @guy-adams in his Oct 25, 2019 post above (your item 6, I believe). This could be in the form of a strict (column) assignment or as "hints" to the auto-layouting. Whether I'd be interested in using an additional tool for that would depend on the ease of use/ease of integration in the toolchain.

[ngouass]

Hi @ptc-sgauglitz,

Thank you for your answer, I will check the documentation to use that colour code feature.

About the columns assignment to models, I will try to address that in the incoming weeks and get back to you in order to have some feedbacks.

[polmonso]

+1 for being able to map colours to tags. One additional thought - may people (e.g. in an ingestion,curation,calcuation,consumption architecture) have clearly defined 'layers' or groupings (e.g. staging, intermediate marts, core marts) of models. To be able to force the models into virtual columns (with a title) would be fantastic e.g.

I'm already denoting these layers using model tags.

DId you open another issue/feature requests with the layer positioning idea?

[ptc-sgauglitz]

DId you open another issue/feature requests with the layer positioning idea?

I did not open another issue nor am I aware of one; it seems both requests were discussed on this one thread.
At least for my purposes, the "color" aspect of it is addressed since dbt 1.3; I would support opening a separate issue specifically for the layouting/"assign columns" aspect. I think this would still be very helpful. I'd be happy to open that issue, or please copy me on it if somebody else opens it.

[mattyb]

#330 and #438 address layout, but maybe not specifically "columns"

[ptc-sgauglitz]

thanks @mattyb !
#438 looks to be more specific if I see that correctly.
#330 is, to my understanding, quite similar to (at least some of) the asks/suggestions here, but it hasn't seen any traction and has been closed due to inactivity -- which is curious given that the ask here has repeatedly drawn lots of upvotes/interest 🤔

I'd be inclined to open a new issue, citing the tickets and above and referencing the interest voiced in this thread too 🤔