prometheus_tsdb_head_series increases to double previous value during WAL replay then drops about prometheus

[milesbxf]

This is partly as much a question as a bug report; this happens so consistently that I wouldn't be surprised at all if this is just expected behaviour and a lack of understanding of WAL replay on my part, but I could swear I haven't noticed this before, and I'm interested to see if there is an explanation - cc. @bboreham

[bboreham]

First suspicion would be this is caused by #12286. Since you are running 2.43, you get all the series from 3 compaction cycles back. 2.45 reduced this to 1 extra.

[milesbxf]

First suspicion would be this is caused by #12286. Since you are running 2.43, you get all the series from 3 compaction cycles back. 2.45 reduced this to 1 extra.

I did see that one and we tried 2.45.2, but got the same result. It's unlikely we'll get a chance to prioritise more upgrades soon, but let me know if there's anything else we can look at. This isn't impacting us or high priority, more of a curiosity - it's good to know it's not expected behaviour at least!

[bboreham]

You would get the same result on first start; the fix only helps after one or two compaction cycles.