Comments (8)
Hello,
Yes, VictoriaMetrics will drop "null" value from /api/v1/import
when store
VictoriaMetrics/lib/storage/storage.go
Lines 1811 to 1816 in 5e8c087
But as I can see, this behavior hasn't changed since /api/v1/import
added support to ingest value like null
in v1.82.0.
VictoriaMetrics/lib/protoparser/vmimport/parser.go
Lines 142 to 143 in 5e8c087
So if you import timeseries with all "null" values like your example above, the whole timeseries will be dropped and they won't be counted in vm_slow_row_inserts_total
since they're not inserted.
But if you import timeseries with part of "null" values like "values":[3,null,13]
, the timeseries will be registered and not be marked as slow_insert next time.
Same metrics are processed correctly in v1.87.14 - they are counted in active series, we can access them and no strange 'slow inserts', 'cache misses' and so on.
Did you ingest the same values to v1.87.14, set all the values to "null"?
from victoriametrics.
Hi Haleygo,
We are using vmagent for sending metrics to VM. In this case /api/v1/write is used but not /api/v1/import.
Did you ingest the same values to v1.87.14, set all the values to "null"?
We have vmagent sending the same metrics to VM v1.87.14 and v1.93.14 with absolutely same settings. Only v1.93.14 has the issue with registering new timeserries if they have nulls value and hadn't existed before.
from victoriametrics.
Ok, that's not expected.
What the version of vmagent here? Did you test with target that down for a while and didn't see NaN values only in v1.93.14?
I did a quick test with vmsingle v1.93.14(vmsingle shares same code with vmagent and vmcluster storage) and the NaN works. My test steps are:
- set up vmsingle v1.93.14 to scrape a target for few minutes;
- stop the target for a while;
- check the results.
Only v1.93.14 has the issue with registering new timeserries if they have nulls value and hadn't existed before.
You mean the new timeseries start with null values, like the target expose metrics with null values?
In my test, the NaN value is attached automatically by vmagent as stale marker.
from victoriametrics.
What the version of vmagent here?
1.93.14
You mean the new timeseries start with null values, like the target expose metrics with null values?
We have process_exporter which is scraped by vmagent and then metrics sent via one more vmagent to two VMs v1.93.14 and v1.87.14
In vmagent's log we see:
2024-04-29T17:56:44.033Z warn VictoriaMetrics/lib/promscrape/scrapework.go:387 cannot scrape target "http://127.0.0.1:9256/metrics" 1 out of 1 times during -promscrape.suppressScrapeErrorsDelay=0s; the last error: the response from "http://127.0.0.1:9256/metrics" exceeds -promscrape.maxScrapeSize=16777216 (the actual response size is 359579335 bytes); either reduce the response size for the target or increase -promscrape.maxScrapeSize
In v1.87.14 i see metric like NaNs in VMUI and like nulls in export:
In v1.93.14 I see no inserted metrics but slow insert and cache miss every minute with new scrape cycle.
Look's like related to staleness markers somehow but not sure how to reproduce issue from the scratch.
Please check this tcpdump vm_bug.pcap.zip
first tcp stream is to v1.93.14 (127.0.0.1) and it's not inserted with 'slow insert' and 'cache miss' every minute
second stream to 1.87.14 (10.111.150.2) and it's inserted and has no 'slow insert' and 'cache miss' every minute
from victoriametrics.
Managed to reproduce it with promremotecli - just modified it for sending "staleNaNBits uint64 = 0x7ff0000000000002".
So when I send "0x7ff0000000000002" as value for new metric to VM v1.93.14 it doesn't register it, increases slow inserts and cache misses.
from victoriametrics.
VictoriaMetrics should stop creating new time series when it receives staleness marker for new time series
Related to #5069.
In VictoriaMetrics, there are two different NaN, one is called staleNaN [using uint64 = 0x7ff8000000000002]
VictoriaMetrics/lib/decimal/decimal.go
Lines 407 to 409 in d386a68
the other one is NormalNaN as math.NaN() [using uint64 = 0x7ff8000000000001]
vmagent or vmsingle only generate staleNaN
value when metrics get missing, like target down, see this doc for details.
When scraping target which expose metric like metric1_0{bar="baz"} NaN
, or importing data which contains metric1_0{bar="baz"} null
using /import
APIs, VictoriaMetrics recognizes value like "null", "NaN", "nan", and set them to NormalNaN
instead of staleNaN
.
Then when it comes to store, VictoriaMetrics can tell the difference between staleNaN and NormalNaN, and only store staleNaN values.
VictoriaMetrics/lib/storage/storage.go
Lines 1811 to 1816 in 5e8c087
But in v1.93.14, the "NormalNaN" is dropped as always, the "staleNaN" won't be considered as valid value and be dropped as well, so the series won't be registered successfully.
But this won't happen if time series has "real" values, at least from time to time.
Could you please elaborate your use case here, why store time series only with NaN values?
from victoriametrics.
Could you please elaborate your use case here, why store time series only with NaN values?
We don't need to store time series with only NaN values. We want VM to be fast and stable when it (for some reasons) gets a lot of NaNs.
In our case we just switched traffic from one VM cluster to other and got the issue with ingesting performance.
We gathered metrics from process_exporter and sent them via vmagent to "old" cluster. When we added to vmagent remotewrite url of "new" cluster with v1.93.14 it started to have performance issues because every minute got millions of NaNs.
One more thing that it was difficult to identify cause of the issue. We saw only strange slow inserts and cache misses and in the same time small number of active series and low churn rate.
from victoriametrics.
We don't need to store time series with only NaN values. We want VM to be fast and stable when it (for some reasons) gets a lot of NaNs.
This looks like a very narrow case. In your example, you're trying to ingest StaleNaNs, a reserved type of NaN for staleness detection. VM accepts StaleNaNs only if series for this sample was registered before with any value different from StaleNaN. There is no sense in recording/registering series which contains only StaleNaNs. To verify if series contains only StaleNaNs, VM does cache and index lookup, which is counted as cache miss and slowInsert.
vmagent will create a stale marker in two cases:
- target, that was previously successfully scraped, was removed from the scrape targets
- target returned list of metrics different to previous scrape. In this case, vmagent will find metric names from prev scrape missing in current scrape and will send stale markers only for these series.
either reduce the response size for the target or increase -promscrape.maxScrapeSize
I wasn't able to reproduce vmagent to send stale markers with this error. It is likely something weird is happening to vmagents in your setup. Could you try setting -promscrape.noStaleMarkers
on vmagents side and see if issue can be reproduced?
from victoriametrics.
Related Issues (20)
- vmauth per-user metrics can cause high memory usage in the long term HOT 2
- vmbackup cannot open a snapshot that it just created HOT 7
- How to drop selected metrics received at victoriametrics (single) from api/v1/write remoteWrite vmagents? HOT 4
- Scrape CRD's support by single-node VictoriaMetrics HOT 2
- victorialogs crash HOT 4
- ERROR: 422 on query with binary operation and keep_metric_names HOT 4
- vmui requests are not automatically quoted HOT 1
- Graphite queries not calculated correctly HOT 1
- how to set custom tag in filebeat output.elasticsearch _stream_fields HOT 2
- vmagent k8s target discovery is too slow HOT 4
- Data integrity problem after the vmstorage breaks down HOT 3
- victoria logsQL sort unexpected result. HOT 3
- How to search log as quickly as possible while writing log in client side HOT 2
- Attempts to configure VM for small memory footprint don't yield expected results HOT 2
- Maximum advised storage capacity for a vmstorage instance? HOT 1
- VictoriaMetrics Datadog APM/Trace Agent URL integration HOT 3
- MetricsQL: Document stalness marker differences between `default_rollup` and `last_over_time` HOT 2
- How to properly setup remoteWrite credentials in vmagent HOT 8
- VictoriaLogs UI sometimes loses log records in Firefox HOT 5
- Teach -httpAuth.username to read content of a file HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from victoriametrics.