Setting DD_TRACE_SAMPLING_RULES doesn't sample traces about dd-trace-py

[emmettbutler]

@connorbrinton Thanks for reporting this and including a small reproduction case. As you've discovered, the sampling code is complex. We'll look into this and see if we can figure out what's going on.

[mabdinur]

Hi @connorbrinton,

1. "dropping trace ... with ... spans" is only logged if a span is unsampled (dropped) AND DD_TRACE_COMPUTE_STATS=True.
   • If the reproduction is run with DD_TRACE_COMPUTE_STATS=True then we see this log message on fastapi.request spans
2. Sampling decisions are currently made by the first span in a trace. Sampling on resource name on root spans are not supported. This allows us to avoid generating partial traces.
3. If allow_false == True then the span will be kept no matter what. This field is used to override sampling decisions for RateSamplers. By default the tracer uses a DatadogSampler (which is not a rate sampler) so this field is irrelevant to most users. allow_false is a confusing argument that maintains legacy behavior. We are looking into removing it.

I apologize for the confusion. Trace sampling is a confusing feature that we are actively trying to simplify. By default the tracer makes a sampling decision by setting the _sampling_priority_v1 tag on spans and then all spans are sent to the Datadog agent. If _sampling_priority_v1 equals 2 or 1 the span is kept by the agent and if _sampling_priority_v1 is equal to 0 or -1 the agent will drop the span. Below is a span generated by hitting the /ready endpoint in your reproduction. In this payload you can see '_sampling_priority_v1': -1 is set so spans are being dropped.

finishing span name='fastapi.request' id=364833566968571663 trace_id=135397972624940070559008041258285491854 parent_id=None service='fastapi' resource='GET /readyz' type='web' start=1708962676.4127822 end=1708962676.4250052 duration=0.012223 error=0 tags={'_dd.base_service': '', '_dd.p.dm': '-3', '_dd.p.tid': '65dcb37400000000', 'asgi.spec_version': '2.3', 'asgi.version': '3.0', 'component': 'fastapi', 'http.method': 'GET', 'http.route': '/readyz', 'http.status_code': '204', 'http.url': 'http://localhost:8000/readyz', 'http.useragent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36', 'http.version': '1.1', 'language': 'python', 'runtime-id': '31c4c041e9e14cbc92f4fd47e4104cd0', 'span.kind': 'server'} metrics={'_dd.rule_psr': 0.0, '_dd.top_level': 1, '_dd.tracer_kr': 1.0, '_sampling_priority_v1': -1, 'process_id': 60174} (enabled:True)
creating new intake connection to http://localhost:8126 with timeout 2
Sending request: PUT v0.5/traces {'Datadog-Meta-Lang': 'python', 'Datadog-Meta-Lang-Version': '3.10.5', 'Datadog-Meta-Lang-Interpreter': 'CPython', 'Datadog-Meta-Tracer-Version': '2.5.2', 'Datadog-Client-Computed-Top-Level': 'yes', 'Content-Type': 'application/msgpack', 'X-Datadog-Trace-Count': '1'}

Based on my investigation sampling appears to be working as expected in the tracer, however there could be an issue in the datadog agent. I would reccomend opening an issue here and our support team can help investigate the root cause: https://www.datadoghq.com/support/

[mabdinur]

Feel free to reopen if you have any further questions and thanks again for bringing this to our attention.