Overrides Configuration Is Not Working about tempo

[mapno]

I've drafted a PR to fix the bug grafana/helm-charts#3236. I've tested it locally both with new and legacy overrides, but more testing is welcome. Sorry for the inconveniences.

[markustoivonen]

@mapno How is one supposed to use overrides currently with helm, when using tempo-distributed? the tempo-distributed helm chart expects the value be

overrides: |
  overrides: 
    foo: bar

And what is in the documentation is:

overrides:
  defaults:
    foo: bar

doesn't work, as it results in an error

template: tempo-distributed/templates/_helpers.tpl:191:14: executing "tempo.overridesConfig" at <.Values.overrides>: wrong type for value; expected string; got map[string]interface {}

When using the legacy method with the legacy names (e.g. ingestion_rate_strategy), in my deployment logs there is the following message:

msg="Overrides config type mismatch" err="per-tenant overrides config type does not match static overrides config type" config_type=legacy static_config_type=new

I am also using the latest helm chart version.

[mapno]

Hi! Do you have per-tenant overrides? You need to use the same format in both overrides configs.

[randrewy]

Hello!
I have multi-tenancy disabled in the same config file

[ftong2020]

indeed, I am encountering this issue, too. It is quite confusing. Documentation is no where to be found how tempo tells which configuration is "new" and which is "legacy".

[AlexDCraig]

I get this too using the tempo-distributed helm chart and:

global_overrides:
  defaults:
    metrics_generator:
      processors:
        - span-metrics
      processor:
        span_metrics:
          histogram_buckets: [null]
          dimensions:
            - kubernetes_cluster
            - kubernetes.cluster
            - http.status_code

[mapno]

Documentation is no where to be found how tempo tells which configuration is "new" and which is "legacy".

That's fair. We probably didn't add the best documentation of the change. As a rule of thumb: long, unindented params are legacy (eg. metrics_generator_forwarder_queue_size), indented params are new (eg. metrics_generator.forwarder.queue_size).

With the new config, a new key defaults was also added after the overrides key, to be more explicit of the contents of the config block.

Finally, there is a command in the tempo-cli that allows you to convert legacy config to new overrides config.

[knylander-grafana]

Thank you for the feedback on the docs. I've created a doc issue to make sure that we capture these changes and address the issues you've had.

#3820

[markustoivonen]

@randrewy @AlexDCraig were you able to solve this somehow?

[randrewy]

@markustoivonen I had to use old format despite of what docs are saying. So my overrides section looks like this

overrides:
  metrics_generator_processors: [service-graphs, span-metrics]
  metrics_generator_processor_local_blocks_complete_block_timeout: "1h"

despite the fact I use latest stable version available atm (2.5.0).

I am not using helm, so this is easy for me to do

[knylander-grafana]

@mapno I know we need to update the docs to address new vs legacy. With your fix, will the example in the docs linked in. previous comments still work?

[mapno]

@mapno I know we need to update the docs to address new vs legacy. With your fix, will the example in the docs linked in. previous comments still work?

The examples are now correct, they didn't work before. Yes, legacy overrides still work too :)