OpenTelemetry Trace forwarding to subscribers is not working correctly about emqx

[SergeTupchiy]

Hi @singhbaljit.

This is an intended behavior that is based on MQTT spec and OpenTelemetry semantic conventions for messaging systems:

1. If EMQX receives trace context in a published message, e.g., traceparent/tracestate User-property for MQTT v5.0, it must be sent unaltered when forwarding the Application Message to a Client to conform with MQTT specification 3.3.2.3.7.
2. https://github.com/open-telemetry/semantic-conventions/blob/main/docs/messaging/messaging-spans.md#context-propagation :

A message may traverse many different components and layers in one or more intermediaries when it is propagated from the producer to the consumer(s). To be able to correlate consumer traces with producer traces using the existing context propagation mechanisms, all components must propagate context down the chain.

Messaging systems themselves may trace messages as the messages travels from producers to consumers. Such tracing would cover the transport layer but would not help in correlating producers with consumers. To be able to directly correlate producers with consumers, another context that is propagated with the message is required.

A message creation context allows correlating producers with consumers of a message and model the dependencies between them, regardless of the underlying messaging transport mechanism and its instrumentation.

The message creation context is created by the producer and should be propagated to the consumer(s). Consumer traces cannot be directly correlated with producer traces if the message creation context is not attached and propagated with the message.

A producer SHOULD attach a message creation context to each message. If possible, the message creation context SHOULD be attached in such a way that it cannot be changed by intermediaries.

[SergeTupchiy]

In other words, traces emitted by EMQX provide extra level of details and cover intermediary level (messaging system itself).
One can disable OpenTelemetry traces in EMQX and EMQX will still perfectly participate in distributed tracing by simply propagating trace context from publishers to subscribers (as described above).

[singhbaljit]

@SergeTupchiy thank you for responding promptly, and thank you for referencing the documentation. It was wrong for me to call it "incorrect". EMQX is following the MQTT v5 spec correctly: that is, the user properties are not mutated. However, I do still think this is functionally incorrect. Here's effectively what the trace map looks like:

[singhbaljit]

I also don't think the OpenTelemetry document is saying to forward traceparent unmutated. It is only saying that the context must be propagated, which means trace id must be unmutated; the span id could/should change. This is exactly what happens in Spring Boot Kafka (Micrometer).

I do think the MQTT v5 spec and the W3C context propagation for MQTT are in contradiction. I can raise this issue there as well. But do you think it possible to add another configuration to enable overriding the traceparent?

[SergeTupchiy]

Messaging systems themselves may trace messages as the messages travels from producers to consumers. Such tracing would cover the transport layer but would not help in correlating producers with consumers.
A producer SHOULD attach a message creation context to each message. If possible, the message creation context SHOULD be attached in such a way that it cannot be changed by intermediaries.

Doesn't it say that any tracing done by messaging system itself is internal and should not affect producer/consumer correlation?
Mutating trace id breaks any correlation and makes any distributed tracing impossible. This is quite obvious, so I don't think that this is the only point of the Open Telemetry document statements mentioned above.

Yes, it is technically feasible to override traceparent in User Properties sent out from EMQX to subscribers (so that subscribers can link to EMQX send_published_message spans) and make this behaviour configurable. It may be considered for development as a new feature (if not rejected), but it's definitely not a bug.