Grafana only shows raw data from Thanos about thanos

[douglascamata]

@stalemate3 you are telling Grafana that you want the minimum step to be 1 hour. This means one datapoint per hour at minimum... at this step size Thanos decided that it will use raw data to answer the query. By default, Thanos does not reply to queries mixing data from different levels -- only a single is used to answer queries. So this is why you don't see the downsampled data there.

You can do two things:

1. Enable add the CLI arg --query-range.request-downsampled=true to your Query Frontend, if you run it. This makes Thanos query for all the downsampling levels, going up in resolution one by one, to check if there's data.
2. Remove or change that min step in your query to try to hit an specific downsampling levels.

Personally I would go with option 1.

[stalemate3]

Sorry for the late reply and thanks for the suggestions, there is some progress with my issue. I've tried both of your suggestions, the 1st one doesn't seem to do anything when I redeploy Query Frontend with that arg from Grafana side. For the second one, that's only there because of my desperation to try random steps to magically fix my problem, sorry about that.

From another issue here I set the --query.auto-downsampling to Query and as I've played around with the min step I set it to 5h and voila:

Anything below that is going to give back the same data as in my original post. I'm not sure why 5h is this magic spot but now there are these gaps in the data. If I set it above 5h the gap widens. Any ideas to fix these gaps?

Thanks for the ideas, I really appreciate it!

[MichaHoffmann]

I think there is some heuristic that max resolution is step/5 or something somewhere! 5h would be answered from 1h down samples data then

[MonolithProjects]

I face the same problem with a similar setup (just running it on on-prem K8s and using the latest Thanos and the latest Bitnami Helm chart). For some reason i don't see in Grafana down-sampled data even if the query component has --query.auto-downsampling set. When i go to Thanos-frontend and choose Auto downsampling i can see the metrics without any issue. If i use custom parameter max_source_resolution=3600000 for my thanos datasource in Grafana ot set the "Min step" to 5h like @stalemate3, i can see the metrics also there. Anyone any idea?