Persist Retroactive Rule Reevaluations about prometheus

[yeya24]

I have implemented a very basic version of this https://github.com/yeya24/backfiller. If the high-level idea of my implementation is similar, I want to work on this and add this utility to promtool

[matttproud]

@juliusv, we know that we can support retroactive rule evaluation now. Is that sufficient to close this issue, or should it be repurposed to handle the case of "archiving samples after retroactive rule evaluation"?

[juliusv]

It's not our top priority, but persisting timeseries resulting from rule evaluations against historical data is still a nice-to-have feature, so we can keep this around.

[matttproud]

Great. I'll re-title this issue. :)

2013/3/28 juliusv [email protected]

It's not our top priority, but persisting timeseries resulting from rule
evaluations against historical data is still a nice-to-have feature, so we
can keep this around.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/11#issuecomment-15575551
.

[brian-brazil]

My thoughts on this are to permit backfilling via the API, and then we can have a separate tool that does this.

[bwplotka]

Hi, I would like to start a new discussion on this.
Basically, there are some clear benefits of having some back-in-time rule evaluation option for Prometheus.

Currently, there are certain UX issues while using rules. Let's have some basic example from a user perspective:

• I created some cool, new metric.
• I created some fancy dashboard.
• Everybody loves my dashboard and it's quite popular, so I want to create a rule for the query used there to take out some load from Prometheus.
• I created a rule.

Problems:

1. How to test my rule? I need to wait some time to have it filled to check if I got it right for a longer period (that could include interesting cases). That requires lot's of context switches in my work and frustration (multiple frustrations by 100x when one is a newbie in PromQL)
2. I cannot immediately put my rule on the dashboard because it's not yet filled

I totally agree that this is not critical and quite hard to achieve when we are talking about persisting new time series back in time when that time range is already packed in blocks and maybe even compacted or backed up/downsampled in case of thanos or remote-written.

But what about some simple solution (A), for example:

• User created new rule
• If flag XXX is set (or certain RPC is invoked on API), reevaluate the rule against all samples you have in memory (WAL)

The only unknown here is remote write, but when we make this feature optional - you can control if you want to have back-in-time rule evaluation or not.

Or maybe another option (B):

• User created new rule
• Backfill the rule in ephemeral space only (mem only) up to flag-rule-reeval-time or API RPC argument to make testing and using this rule on dashboards easier. "Past" samples will be ephemeral. Only new samples will be written normally.

Any thoughts?

CC @juliusv @brian-brazil @fabxc @devnev

[brian-brazil]

The requirement here is basically for bulk import, as I don't think this functionality belongs in Prometheus itself (though may be tied together via promtool).

[bwplotka]

bulk import

Sorry, what do you mean by that?

[brian-brazil]

#535

[aamkye]

Hi, I'd like to ask if any progress has been made regarding generation of metrics in the past based on recording rules, I've described use case in #6440.

[yeya24]

Could anyone give some hints or directions about this feature? Should it be in the internal Prometheus or promtool?

[roidelapluie]

I would really loooove to see it reuse #5887 which is actively being worked on

[yeya24]

I would really loooove to see it reuse #5887 which is actively being worked on

Yes, totally agree with you. Waiting for that getting merged.

[supra08]

Hi @juliusv @codesome, I am looking forward to contribute to Prometheus and participate in GSOC 2020. How can I proceed with this project?

[Harkishen-Singh]

I am taking this up. Thank you.

[roidelapluie]

Could you please wait for the previous issues to be settle on? We also probably want to agree on a small design document first. Le lun. 24 févr. 2020 à 16:53, Harkishen Singh <[email protected]> a écrit :

…

I am taking this up. Thank you. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#11?email_source=notifications&email_token=AACHHJUWZQACNRAESJH6ZFLREPUQ7A5CNFSM4ACXUIS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMYMGIQ#issuecomment-590398242>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACHHJTTZ4AYNTSBETCI2IDREPUQ7ANCNFSM4ACXUISQ> .

[yeya24]

I am taking this up. Thank you.

Hi, I already implemented this feature https://github.com/yeya24/backfiller here, but blocked on upstream promtool importer. Can I take this?

[codesome]

We also probably want to agree on a small design document first.

this ^

And there are many takers here :)
@Harkishen-Singh @yeya24: are you both planning to participate in GSoC as well?

[Harkishen-Singh]

@Harkishen-Singh @yeya24: are you both planning to participate in GSoC as well?

Yes

[bwplotka]

Available as Community Bridge project.

[yeya24]

Hello, 👋 I just started a design doc for this feature. https://docs.google.com/document/d/1h2MBA-tc2ALYBTr5b2bKLmQBaRP2aVJQOLNsIDFo8W4/edit?usp=sharing

I didn't add many detailed there, just some high-level ideas. Feel free to edit and comment on the doc directly!

[Abhik1998]

Hello, I want to take this up as a community bridge mentee project.

[Godtide]

Hi, @bwplotka I'm looking to work on this project as a community bridge mentee. How do we get started, already have my resume on the Portal

[codesome]

Closed by #7675

[roidelapluie]

For the remaining tasks see #8667