Persistent state about prometheus-gravel-gateway

[sinkingpoint]

Hi there! Similar to other Push Gateways, we don't persist state here. The logic is that this behaves as sort of a proxy for your metrics - if you restart a standard service, then you lose state so we figured that it wasn't important here.

That being said, I think the idea of persisting state, at least for an opt-in subset of metrics is interesting. Did you have a usecase in mind?

[kradalby]

Similar to other Push Gateways, we don't persist state here. The logic is that this behaves as sort of a proxy for your metrics - if you restart a standard service, then you lose state so we figured that it wasn't important here.

I understand, we have been using the "original" Pushgateway which does allow you to define a persistent file. It is written to the file system which we have had varied success with when running on Kubernetes and with network storage.

That being said, I think the idea of persisting state, at least for an opt-in subset of metrics is interesting. Did you have a use case in mind?

Our use case is for batch and other types of long running jobs that have a more "loose" coupling to the push gateway, as in the push gateway will be a continuous proxy, but the jobs will come and go throughout the day.

The concern with having the push gateway being able to "recover" after a restart would be for very long running jobs that might not emit continuous metrics. In that scenario, if the push gateway is restarted, we might end in a scenario where the last metric emitted by a job is "lost" and marked absent by Prometheus since it is no longer in the push gateway.

I agree that this isnt necessary for every use case, so a opt-in sounds like a reasonable approach.

Blob/S3 is suggested to make the application a bit more resilient to things that attempt to mimic POSIX semantics (like a persistent kubernetes volume) but might fail because of networking etc.