Tempo-cli trace-summary issue about tempo

[joe-elliott]

I've never seen this command hang and we use it semi regularly. Could you use fmt.Printlns to determine where it was hanging?

[edgarkz]

@joe-elliott can you please provide an example of the above function usage? im not familiar with.

[edgarkz]

Some updates, i managed to get an output using additional argument --percentage 0.01 but its pretty empty of info.

Is there an option to add --percentage param to tempo-cli query blocks so i can try and dump those spans?

total blocks to search:  7
..0.....Number of blocks: 4 
Span count: 176817 
Trace size: 130369402 B 
Trace duration: 111822 seconds 
Root service name:  
Root span info:
No root span found
top frequent service.names: 
[xxxxx.Service xxxx.Service xxxx.Service xxxx.Service xxxx.Service

[joe-elliott]

@joe-elliott can you please provide an example of the above function usage? im not familiar with.

Just place fmt.Println("foo") anywhere in the codebase and you will see it print out when the line is hit. It can be used to get a sense of where the application is hanging. Since you are running the cli directly from source go run ./cmd/tempo-cli it's quite easy.

Some updates, i managed to get an output using additional argument --percentage 0.01 but its pretty empty of info.

Scanning 1% of your total blocks you found 176K spans and 130MB of trace data. The cli is likely hanging b/c this trace is enormous. If trace summary isn't returning then query blocks definitely won't. This suggests that your trace is 17M spans and 13GB.

Is there an option to add --percentage param to tempo-cli query blocks so i can try and dump those spans?

Sure, give it a shot. Here is the code that restricts to a %age of blocks:

https://github.com/grafana/tempo/blob/main/cmd/tempo-cli/cmd-query-trace-summary.go#L95-L103

You could copy that into the query blocks code here:

https://github.com/grafana/tempo/blob/main/cmd/tempo-cli/cmd-query-blocks.go#L86

[edgarkz]

Thank you @joe-elliott
I'll raise a PR with those changes since its really useful to have this limit in place.