Code(499) desc = context canceled about tempo

[joe-elliott]

It appears the query is timing out. I would consider increasing trace_by_id -> query_shards and scaling up your queriers.

You can also use the Tempo CLI to pull the trace directly from the blocks to get information about it:

https://grafana.com/docs/tempo/latest/operations/tempo_cli/#query-trace-summary-command

[kosov73]

I increased Traces_by_id -> query_shards to 1000 and increased the number of queries to 4, it got better, but the ingester and querier started crashing with OOMKiller, the resources each had 25 GB of memory

ingester:
  replicas: 6
  config:
    replication_factor: 6
querier:
  replicas: 4
queryFrontend:
  replicas: 1
  config:
    search:
      concurrent_jobs: 2000
      target_bytes_per_job: 904857600
    trace_by_id:
      query_shards: 1000

[joe-elliott]

set replication_factor = 3. i have only very vague ideas on how Tempo will behave with RF6.