[metrics-generator] overrides per tenant do not work for metrics-generator about tempo

[joe-elliott]

Thanks for following up!

@knylander-grafana can we make sure this is documented somewhere? this has caught multiple people before.

[knylander-grafana]

WIll do! I'll create a doc issue: #3462

[joe-elliott]

Are there any relevant logs that might help? At first glance I'm not seeing anything wrong with your config. We use the overrides all the time internally so it generally works.

Can you review metrics to help narrow down the issue. Are the distributors still sending spans to the generators? Are spans being dropped for any reason?

One sharp edge that might be causing this is that if an per tenant override block is matched than the entire block is used for that tenant (including all 0s). Tempo does not override at the field level.

[ToonTijtgat2]

Dear @joe-elliott, thanks for checking my configuration.

I tried the legacy and the new config way with the overrides per tenant, but from the moment an override is used on the metrics_generator it seems to stop working.

I'll try to set the log level to debug to see If I can find any more relevant logs.

Since the only change is the small overrides part, and traces still seem to come trough, I think no spans are dropped. is there a way to check this?

[ToonTijtgat2]

After the change to the broken code, I indeed see all lines like rpc error: code = ResourceExhausted desc = RATE_LIMITED: ingestion rate limit (0 bytes) exceeded while adding 923 bytes for user tracing-apps-app-dev

Why does it start doing this while I only wanted to override metrics_generator...

[ToonTijtgat2]

  overrides.yaml: |

    overrides:
      tracing-apps-app-dev:
        ingestion:
          rate_strategy: local
          rate_limit_bytes: 15000000
          burst_size_bytes: 20000000
          max_traces_per_user: 10000
        read:
          max_bytes_per_tag_values_query: 5000000
        global:
          max_bytes_per_trace: 1500000
        metrics_generator:
          processors: [service-graphs, span-metrics]
          remote_write_headers:
            X-Scope-OrgID: prometheus-apps-app-dev

With this config it works, so when you override, make sure you override everything...

Thanks @joe-elliott for the solution.