Allow a way to query for large traces that exceeded the ingester limit about tempo

[joe-elliott]

So there are questions you can ask Tempo using TraceQL to get an understanding of what is creating your enormous trace. These features are in main now.

{ trace:id = "foo" } | by(resource.service.name) | select(trace:rootService, trace:rootName)
{ trace:id = "foo" } | rate() by(resource.service.name, name)

The above can help, but what I'd really like to do is create a /api/v2/traces/<id> endpoint that returns the trace nested one layer deep. This would allow us to return something like:

{
  trace: {...},
  warning: "trace exceeds max size. partial trace returned."
}

There is currently no good way to tell a client that we are returning only part of the trace. HTTP status code 206 is kind of similar and may also be an option, but i'd prefer revving the endpoint and having a way to pass detailed messages back to the client.

[atavakoliyext]

Thanks for your response @joe-elliott. Having the endpoint you described sounds like a good long-term direction; but I was still surprised that there's any query-level enforcement of trace size limits, given that #1225 is still open (I see it was just closed). Was this delivered through a different issue, or was it always the case that trace size limits were enforced in queries? If this is expected behavior, does my request to have the configuration for ingest-level vs query-level trace limit enforcement sound reasonable, even with the endpoint you described?

{ trace:id = "foo" } | by(resource.service.name) | select(trace:rootService, trace:rootName)
{ trace:id = "foo" } | rate() by(resource.service.name, name)

These features are in main now.

Thanks for sharing these! I'm excited to try them out. Are these currently scoped to (or exist in some form in) a release?

[joe-elliott]

The primary reason to enforce this on the query path is to prevent instability from parsing and combining 100s of MBs of trace data. Thank you for pointing out that issue! As you noticed I closed it :).

If this is expected behavior, does my request to have the configuration for ingest-level vs query-level trace limit enforcement sound reasonable, even with the endpoint you described?

I think splitting this limit into two is a fine idea. We would definitely accept that PR.

Are these currently scoped to (or exist in some form in) a release?

I think we are in luck. They should be doable in 2.5. I thought perhaps trace:id did not make the cutoff but it did.