Comments (10)
The reason for getting errors seems to be invalid sample format. You should not escape label_value
double quotes: so in your example cluster=\"sf-prod-2\"
should instead be cluster="sf-prod-2"
.
Can you tell what version of the adapter are you running? The older versions of the adapter were causing high CPU on DB (due to some slow triggers). Most recent versions don't have that problem and should easily handle 100K/s inserts on an average sized cloud instance.
from prometheus-postgresql-adapter.
Hm strange problem with format because i did not push this messages by hand, this format is from prometheus remote_write
version of apps is next
prometheus 2.5.0
prometheus-postgresql-adapter v0.4.1
from prometheus-postgresql-adapter.
You're right, the backslash is actually being inserted by the logger. I've tested your sample and it was inserted successfully which is weird. The error you are getting is coming from pg_prometheus
extension. What version of pg_prometheus
have you installed?
To make sure we have proper sample format can you run the adapter with -pg.prometheus-log-samples=true
and it will print raw samples to stdout. Then we can find troublesome samples and try inserting them directly into DB by running SQL:
INSERT INTO metrics_copy VALUES ('node_cpu_frequency_min_hertz{cluster="sf-prod-2",colo="sf",colo_type="prod",container_name="node-exporter",cpu="cpu3",host_name="hss-sf-prod-2-8",instance="199.241.123.9:9100",job="consul-services",prometheus="long_retention"} 1200000000 1541523310332');
I didn't quite understand what exporter(+version) is generating metrics?
from prometheus-postgresql-adapter.
Hello, thank you for the help
We use latest dockerized pg_prometheus with timescaledb 1.0.0
we . use hundreds of different exporters, but the string is from node-exporter 0.15.2 version, i can try to run this in adapter, but there is millions of strings not sure that we can find needed string, i will try to do that. For now i just want . to understand how to tune pg_prometheus and adapter to ingest millions of strings
from prometheus-postgresql-adapter.
For DB configuration I'd recommend going through TimescaleDB docs: https://docs.timescale.com/v1.0/getting-started/configuring (use pg_tune
for PostgreSQL specific configs).
There is no configuration needed for pg_prometheus
.
I am guessing that you are having poor insert performance due to small batch size. The adapter itself has no batching, but it receives data from Prometheus in batches. So you should increase batch size in Prometheus by tweaking max_samples_per_send
(https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cremote_write%3E). Set it to 1000.
Let me know if it helps.
You might want to experiment with changing connection pool size in the adapter but go first for the batch size.
from prometheus-postgresql-adapter.
Hello, i used pg_tune to get configuration that i post in first message.
i try to tune batch size and have this configuration in remote write prometheus
remote_write:
- url: "http://x.x.x.x:9201/write"
remote_timeout: 100s
queue_config:
capacity: 10000000
max_shards: 1000
max_samples_per_send: 10000
batch_send_deadline: 5s
max_retries: 3
i try different settings but none working well without loosing metrics
What can i try in next step?
from prometheus-postgresql-adapter.
Hmm ok. There are two different problems:
- Did you see an improvement in write performances after tweaking Prometheus batch size?
- I assume you are still encountering
pq: invalid input syntax for prometheus sample: Unexpected number of input items assigned: 1\n
so data doesn't get inserted?
For the 2nd problem it's still unclear why is it happening. You can try running the SQL I posted using troublesome samples as input and let me know how it goes?
from prometheus-postgresql-adapter.
actually i start using with this settings that i post, because with limit of 100000 metrics its not possible to ingest 10000000 metrics,i try to tune max_samples_per_send
sometimes its ingest normally but sometimes its failed with a lot of errors like
level=warn ts=2018-11-06T16:57:33.815570729Z caller=log.go:29 msg="Error sending samples to remote storage" err="pq: invalid input syntax for prometheus sample: Unexpected number of input items assigned: 1\n" storage=PostgreSQL num_samples=1355
anyway the disk usage is very huge so i want to tune postgress settings first, maybe its related, if there is some additional settings for huge data? i mean maybe i need to change anything in my current config?
from prometheus-postgresql-adapter.
I'd recommend using ZFS file system so data gets compressed around 4X. You will also notice performance improvements with ZFS. However, TimescaleDB can't match Prometheus disk usage b/c Prometheus is special purpose built storage with lot of compression tricks ( delta-delta encoding...)
from prometheus-postgresql-adapter.
@dmitriy-lukyanchikov are you still facing some of thee issues? We have recently released a new version of the adapter, and are preparing to release a new version of Timescale that I think would help with your storage concerns. Let me know if you are interested in re-visiting ?
from prometheus-postgresql-adapter.
Related Issues (20)
- how to clean data in timescaledb HOT 1
- Docker Container Image Outdated HOT 1
- [Thanos Integration] Implementing other API's
- prometheus-postgresql-adapter crashes on prometheus reload HOT 1
- Adapter High memory usage - 20GB+ HOT 1
- too many open files error HOT 1
- Suppot other locales than english
- How to configure to connect to Postgres with a different user account than postgres HOT 3
- Grafana Dashboards not showing correct data from Promethus which is using Timescale db HOT 2
- Error: manifest for timescale/pg_prometheus:latest not found HOT 2
- Error: prometheus_postgresql_adapter on 127.0.0.11:53: no such host
- Too many metrics data are logged in pg-adapter container log HOT 1
- No documentation how to use RAFT based cluster manager like ETCD
- Prometheus-postgresql-adapter in HA mode starts to write with 2 nodes after database restart HOT 5
- Adapter triggers "CREATE TABLE metrics_labels" every start HOT 3
- no docs about how adapter work
- Md5sum for binaries HOT 1
- Logging plaintext password HOT 3
- Leader election not happening if leader resigns due to prometheus timeout
- Run adapter without superuser
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from prometheus-postgresql-adapter.