Trace data is being generated even though there are no request from service A to service B about tempo

[mapno]

Hi! Service graphs have a number of ways of identifying communication between services—for Tempo they're described in the docs. Connections not necessarily need represent HTTP requests.

* A request across a messaging system where the outgoing and the incoming span must have `span.kind`, `producer`, and `consumer` respectively.

This is what's identifying a connection between the two services.

[mapno]

Hey! span_kind is not a label of service graph metrics (it's set on span-metrics though). I'm not sure if it'd make sense to add it in the first place, since it's implicit by the connection type—ie. if connection_type is messaging_system, the spans must have had kind consumer and producer.

[finda-yeongjo]

Hey @mapno Your answer was fantastic. I have perfectly removed the problematic parts from the dashboard and various graphs using Tempo as a data source. I blame myself for not carefully reading the docs.

May I ask one more question?
When specifying span_kind, there is no data (span_kind_consumer, producer, server, client and unspecified). Is there any additional configuration needed? Simply setting connection_type=messaging_system shows all servers communicating through MSK

I am using auto-instrumentation because I cannot enforce spans on all technical teams, which makes it difficult for me to directly control headers, span kinds, IDs and etc....