Tempo/Vulture: GetObject S3 API operation - costs. about tempo

[joe-elliott]

I honestly have no idea how much vulture costs per day. Two options that may help reduce spend:

Increase the time between calls by using these params:

https://github.com/grafana/tempo/blob/main/cmd/tempo-vulture/main.go#L71-L72

Add a bloom/footer cache to reduce GETs on trace by id lookup.

[bmteller]

We noticed a similar issue with some of our internal tooling which was doing trace-id lookup and if you know the timestamp where the trace occurs then you can significantly reduce the number of blocks that tempo will have to process by specifying a start and end window around the timestamp where the trace would be created. We have ~300 blocks so that would be 300 requests to s3 for all of the bloom filter lookups. When specifying a window +/- 20 minutes around the trace this reduced the number of blocks checked to 1. Though, it's always possible the window could straddle two blocks then it would check 2 blocks. These blocks were also in the past so I suspect for something like vulture using a window would mean the blocks would not be checked at all because the trace would not be committed to any block yet.

https://github.com/grafana/tempo/blob/main/cmd/tempo-vulture/main.go#L469