Multiple root spans and orphaned spans being combined into a single trace about tempo

[mapno]

Hi @codedinsugar. At first glance, this looks like an issue with the instrumentation—ie. the same "context" is being used for both calls. Can you verify that you're independently creating new traces for each call without reusing something that might carry the context of previous traces? Or, would it be possible to have a reproducible setup that generates those traces? Manually instrument a Java application with OTel framework 1.37.0 is too generic.

[joe-elliott]

Other things to consider:

• Time clock skew may be the cause of the 8s gap
• There is also the possibility of trace id collision. Check out the log_received_spans config section on the distributor. Perhaps we could use this to diagnose.

[codedinsugar]

@mapno that is something that we're considering. We've recently refactored our app along these lines but I think the tracer instantiation might still be an issue for us. One thing we're trying is to add traceId as a span attribute with Span.current().getSpanContext().getTraceId() but the value is always all 0's and we're not sure why. Any thoughts here?

We'd like to provide a reproducible setup but this is a proprietary monolith and cannot be shared. We might be able to build a smaller sanitized version but that'll take time.

@joe-elliott thanks for the suggestion and we'll consider it, our only concern is the "This is not recommended for production environments" statement.

[joe-elliott]

This is not recommended for production environments

I would definitely not leave it on permanently, but for a short time period it maybe helpful.