Comments (12)
Bryan and I discussed on Slack; over text we'd misunderstood each other. His config changes were definitely valid given the low scrape load he had. Remote write has some gaps when it comes to handling timely sending of data in that kind of a scenario, the hard coding of the reshard check ticker is just one of those gaps.
I'll be opening a few issues soon for some things we can try out, there are a number of people interested in taking on some smaller tasks in RW and those could be good first issues for them.
from prometheus.
I notice in a similar previous issue @csmarchbanks said #7124 (comment)
At very low remote write volumes it is very easy to go through multiple batch send durations without new samples coming in
Which matches the situation in this case (volume was about 500 series, scraped every 5s).
Anyway this judgement seems highly dependent on the value of BatchSendDeadline
; is it unreasonable to set it to 100ms?
low volumes are unlikely to ever reshard above the minimum anyway
More context: the machine was occasionally under heavy CPU load; I believe this generated a backlog on the send queue.
(Sadly I don't have metrics to confirm this)
from prometheus.
is this good @bboreham or do we need to wait for others review .
from prometheus.
Best to comment on the PR within the PR itself.
from prometheus.
okay @bboreham .
from prometheus.
Which matches the situation in this case (volume was about 500 series, scraped every 5s).
Anyway this judgement seems highly dependent on the value ofBatchSendDeadline
; is it unreasonable to set it to 100ms?
I would have suggested/assumed people would drop the batch size max_samples_per_send
a lot lower before dropping the BatchSendDeadline
that low.
from prometheus.
I don't think that helps. In my example, Prometheus scraped 509 series every 5 seconds; I wanted it to send those 509 series without waiting 5 seconds.
If I reduce max_samples_per_send
from 2000 default to 100, say, it will send 500 of them, but I still want it to send all 509 series.
from prometheus.
Personally I would still lower the batch size before the send deadline timeout, but even so I think guarding against excessive resharding checks is a valid change. Reviewing the PR again today.
from prometheus.
I don't think I am understanding your point. What would you lower max_samples_per_send
to, given my example?
from prometheus.
I don't think I am understanding your point. What would you lower
max_samples_per_send
to, given my example?
Something below 509
? Or even just 1000
or so and lower the send deadline to ~1s. I don't know exactly what your use case is but scraping a small amount of samples and then always sending all of them via remote write ASAP isn't really a situation we've designed for. Setting the send deadline to 100ms is just a workaround that's worked in your case.
This is separate from the issue of the the resharding check happening too often when the send deadline is < 5s, which I don't have any issue with merging a fix for.
from prometheus.
lower the send deadline to ~1s
OK, that case still shows the issue I am talking about, because two times 1 second is way less than the 10s interval it checks at.
always sending all of them via remote write ASAP
That isn't what I asked for; I asked for:
in a timely manner
and
without waiting 5 seconds
Setting the send deadline to 100ms is just a workaround that's worked in your case.
I disagree, it matches what I wanted.
Bryan
from prometheus.
yeah after seeing and understanding code @bboreham assumption here is valid as we dont need to wait for 5 seconds , and yeah as @bboreham also asked for feeding data timely manner . and also we need to remove that hard coding of the resharding check ticker .
from prometheus.
Related Issues (20)
- allow to specify label sets for which `keep_firing_for` applies
- an alternative to `keep_firing_for` that takes the `up`ness of metrics/targets into account
- remote write lose sample when target is unavailable
- Prometheus v2.52.0 raises "Error on ingesting samples with different value but same timestamp" for kube-state-metrics HOT 19
- Export a metric for WAL size on disk HOT 2
- Request to validate the Hashmod config HOT 1
- metric with empty label value will add a new metric record HOT 4
- CVE-2023-31484 and CVE-2023-50387 HOT 1
- PromQL(histograms): Make more use of Kahan summation for native histograms
- Visualizing tarjets "not happy" if you have more than 20 jobs defined HOT 2
- Emit a warning when `sort` is used for range queries HOT 1
- Gitpod: Missing Packages in `GOROOT` during Build HOT 2
- remote write: all samples lost when server returns 500 HOT 1
- Running `promtool` with * from the docker image doesn't behave the same way with snap HOT 4
- Provide alternative distro for prometheus image HOT 1
- How to effectively monitor abnormal occurrences of CPU and memory usage in containers using PrometheusQL expressions?
- API: Consider delivering _info_ level annotation in separate JSON array HOT 1
- [prometheus] Secure Prometheus with enabling Ldap authentication HOT 1
- EC2 service discovery: show scrape pool in targets page when configured but no targets found yet HOT 1
- field path not found in type config.plain HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from prometheus.