batchlog: time to replay failed batches can explode as the batchlog table accumulats tombstones about scylladb

[michoecho]

We didn't notice this before, because we would only replay all failed batches on unbootstrap, which is rare and a heavy and slow operation on its own right already.

Not quite. Even if there is no repair or unbootstrap happening, we still replay the batchlog every 60s. It happens in the background, and nothing waits for it, so it's hard to notice it without looking at the metrics, but it's still eating CPU.

[denesb]

Indeed, I only looked at external callers so didn't notice the replay loop.