How to drop selected metrics received at victoriametrics (single) from api/v1/write remoteWrite vmagents? about victoriametrics

[tekert]

Oh Thanks.

 -relabelConfig string
        Optional path to a file with relabeling rules, which are applied to all the ingested metrics. The path can point either to local file or to http url. See https://docs.victoriametrics.com/#relabeling for details. The config is reloaded on SIGHUP signal

Somehow missed this, google keep sending me to
https://docs.victoriametrics.com/vmagent/#relabeling
instead of
https://docs.victoriametrics.com/#relabeling

And somehow my fault too for not realizing there are two relabeling docs.

Going to try it now.

[tekert]

All working now, following the https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config spec since victoriametrics points me that the spec to follow. And the guidance of https://valyala.medium.com/how-to-use-relabeling-in-prometheus-and-victoriametrics-8b90fc22c4b2

For anyone intered for droping metrics for go_ and process_ for example just write this in the file pointed by -relabelConfig

- action: drop
  source_labels: [__name__]
  regex: "go_.+|process_.+"

Thanks to vylyala post, if someone whats to drop go_ and process_ metrics only if the have a specific label or labels remeber they are parsed as: label1;label2;etc for regex.
The doc is all over the place but is nice to read and easy to find :)

quoting:

Then source_labels contains multiple labels, they are concatenated with ; char before matching the provided regex. 
For example, the following relabeling rule would drop metric with {job="foo",instance="bar"} labels:

- action: drop
  source_labels: [job, instance]
  regex: "foo;bar"

[jiekun]

I haven't tried it myself but there are some documents related to it:
https://docs.victoriametrics.com/single-server-victoriametrics/#relabeling

Also, if you are looking at the source code and issues of vminsert, you could find:

1. people are using -relabelConfig flag in vminsert: #638
2. For prometheus remote-write handler, vminsert will check if relabel config exists and do:

   VictoriaMetrics/app/vminsert/promremotewrite/request_handler.go

   Line 56 in 80f3644

   ctx.ApplyRelabeling()

Please try it out and provide feedback here. I would be happy to test it myself if you encounter any unexpected issues while following the documentation.

[tekert]

Just to add for anyone looking at this in the future.

Since RE2 regex doen't support negative lookahead for words we have to use the enhancements made for relabeling from https://docs.victoriametrics.com/vmagent/#relabeling-enhancements
(It's in the vmagent section but works perfectly from victoriametrics process single)

Example to drop all metrics that match a regex except for a particular job named victoriametrics:

# Drop all metrics that containt go_ OR process_ ONLY IF job != "victoriametrics"
- action: drop
  if:
  - '{job != "victoriametrics"}'
  source_labels: [__name__]
  regex: 
    - "go_.+"
    - "process_.+"

# Now drop only go_ metrics (while keeping process_ intact) of the metrics from job victoriametrics
- action: drop
  if:
  - '{job = "victoriametrics"}'
  source_labels: [__name__]
  regex: 
    - "go_.+"