Problem with 10 Millions metrics from Prometheus every 5 minutes about prometheus-postgresql-adapter

[niksajakovljevic]

Hi @dmitriy-lukyanchikov

The reason for getting errors seems to be invalid sample format. You should not escape label_value double quotes: so in your example cluster=\"sf-prod-2\" should instead be cluster="sf-prod-2".

Can you tell what version of the adapter are you running? The older versions of the adapter were causing high CPU on DB (due to some slow triggers). Most recent versions don't have that problem and should easily handle 100K/s inserts on an average sized cloud instance.

[dmitriy-lukyanchikov]

Hm strange problem with format because i did not push this messages by hand, this format is from prometheus remote_write
version of apps is next
prometheus 2.5.0
prometheus-postgresql-adapter v0.4.1

[niksajakovljevic]

You're right, the backslash is actually being inserted by the logger. I've tested your sample and it was inserted successfully which is weird. The error you are getting is coming from pg_prometheus extension. What version of pg_prometheus have you installed?

To make sure we have proper sample format can you run the adapter with -pg.prometheus-log-samples=true and it will print raw samples to stdout. Then we can find troublesome samples and try inserting them directly into DB by running SQL:
INSERT INTO metrics_copy VALUES ('node_cpu_frequency_min_hertz{cluster="sf-prod-2",colo="sf",colo_type="prod",container_name="node-exporter",cpu="cpu3",host_name="hss-sf-prod-2-8",instance="199.241.123.9:9100",job="consul-services",prometheus="long_retention"} 1200000000 1541523310332');

I didn't quite understand what exporter(+version) is generating metrics?

[dmitriy-lukyanchikov]

Hello, thank you for the help
We use latest dockerized pg_prometheus with timescaledb 1.0.0
we . use hundreds of different exporters, but the string is from node-exporter 0.15.2 version, i can try to run this in adapter, but there is millions of strings not sure that we can find needed string, i will try to do that. For now i just want . to understand how to tune pg_prometheus and adapter to ingest millions of strings

[niksajakovljevic]

For DB configuration I'd recommend going through TimescaleDB docs: https://docs.timescale.com/v1.0/getting-started/configuring (use pg_tune for PostgreSQL specific configs).
There is no configuration needed for pg_prometheus.
I am guessing that you are having poor insert performance due to small batch size. The adapter itself has no batching, but it receives data from Prometheus in batches. So you should increase batch size in Prometheus by tweaking max_samples_per_send (https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cremote_write%3E). Set it to 1000.
Let me know if it helps.

You might want to experiment with changing connection pool size in the adapter but go first for the batch size.

[dmitriy-lukyanchikov]

Hello, i used pg_tune to get configuration that i post in first message.
i try to tune batch size and have this configuration in remote write prometheus

remote_write:
  - url: "http://x.x.x.x:9201/write"
    remote_timeout: 100s
    queue_config:
      capacity: 10000000
      max_shards: 1000
      max_samples_per_send: 10000
      batch_send_deadline: 5s
      max_retries: 3

i try different settings but none working well without loosing metrics

What can i try in next step?

[niksajakovljevic]

Hmm ok. There are two different problems:

1. Did you see an improvement in write performances after tweaking Prometheus batch size?
2. I assume you are still encountering pq: invalid input syntax for prometheus sample: Unexpected number of input items assigned: 1\n so data doesn't get inserted?

For the 2nd problem it's still unclear why is it happening. You can try running the SQL I posted using troublesome samples as input and let me know how it goes?

[dmitriy-lukyanchikov]

actually i start using with this settings that i post, because with limit of 100000 metrics its not possible to ingest 10000000 metrics,i try to tune max_samples_per_send sometimes its ingest normally but sometimes its failed with a lot of errors like

level=warn ts=2018-11-06T16:57:33.815570729Z caller=log.go:29 msg="Error sending samples to remote storage" err="pq: invalid input syntax for prometheus sample: Unexpected number of input items assigned: 1\n" storage=PostgreSQL num_samples=1355

anyway the disk usage is very huge so i want to tune postgress settings first, maybe its related, if there is some additional settings for huge data? i mean maybe i need to change anything in my current config?

[niksajakovljevic]

I'd recommend using ZFS file system so data gets compressed around 4X. You will also notice performance improvements with ZFS. However, TimescaleDB can't match Prometheus disk usage b/c Prometheus is special purpose built storage with lot of compression tricks ( delta-delta encoding...)

[bboule]

@dmitriy-lukyanchikov are you still facing some of thee issues? We have recently released a new version of the adapter, and are preparing to release a new version of Timescale that I think would help with your storage concerns. Let me know if you are interested in re-visiting ?