GetRowsDiff func takes a lot of time when vmagent pushes kube-state-metrics data about victoriametrics

[tiny-pangolin]

have you tried VMAgent in a cluster mode to spread the scrape jobs out over multiple instances of VMAgent?

There are instructions in the docs for running in cluster mode on k8s and a blog post explaining how cluster mode works.

[f41gh7]

600MB is really huge metrics response. And It'll take a lot of time to process it.

I can suggest to disable staleMarkers detection no_stale_markers: true for this job (or for all jobs with flag -promscrape.noStaleMarkers").
job config example:

scrape_configs:
- job_name: kube-state-metrics
  no_stale_markers: true
  # make sure that series limit is not set or has default 0 value
  series_limit: 0
  static_configs:
  - targets: ["kube-state-url:1234"]

It must skip GetRowsDiff function execution.

[MajLuu]

600MB is really huge metrics response. And It'll take a lot of time to process it.

I can suggest to disable staleMarkers detection no_stale_markers: true for this job (or for all jobs with flag -promscrape.noStaleMarkers"). job config example:

scrape_configs:
- job_name: kube-state-metrics
  no_stale_markers: true
  # make sure that series limit is not set or has default 0 value
  series_limit: 0
  static_configs:
  - targets: ["kube-state-url:1234"]

It must skip GetRowsDiff function execution.

Good idea, but I want to know if there are other problems? And although setting no_stale_markers: true, I consider it also run GetRowsDiff func once ref. If we could optimize the GetRowsDiff func? Or we shard kube-state-metrics component.
Thanks for replies.