Comments (11)
@caleb15 Yeah, 24h is a good place to start - in my experience that resolves the issues 90% of the time.
The reason the statement_timeout setting doesn't take effect is because the collector sets a session-local session timeout:
collector/input/postgres/establish_connection.go
Lines 95 to 104 in 30e17bf
from collector.
This appears to have worked, thanks!
from collector.
The pganalyze container pointed at the heavy mirror is using a bunch of memory compared to the other instances for some reason:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
490ccc38d41d pganalyze-mirror-heavy 0.00% 2.991GiB / 7.698GiB 38.85% 10.2GB / 6.79GB 26.1MB / 1.09GB 10
36389b04b5f7 pganalyze-mirror1 0.00% 1.342GiB / 7.698GiB 17.44% 13.7GB / 3.46GB 167MB / 9.24GB 10
8f4c79d8ea41 pganalyze-mirror 0.00% 1.472GiB / 7.698GiB 19.12% 13.8GB / 3.48GB 228MB / 9.1GB 10
246474c64fb0 pganalyze 0.00% 97.4MiB / 7.698GiB 1.24% 10.2GB / 2.33GB 157MB / 338MB 10
May be unrelated.
from collector.
@caleb15 Could you try doing a "SELECT pg_stat_statements_reset()" to see if that addresses the timeouts?
(when the Postgres query text file grows very large, retrieving the text sometimes takes too long)
from collector.
Sure, I'll try that out, although I still don't see how it can be timing out with statement timeouts disabled.
Unrelated question: when you recommended that I set Collector: Query Statistics Reset Frequency
what value did you recommend again? 24h?
from collector.
I enabled it, I'll see if that fixes it. For the Collector: Query Statistics Reset Frequency
should I just set that to 24h for the master or do it for the master and the mirrors?
from collector.
@caleb15 I would set it on all the servers where you are seeing timeouts.
In addition, make sure you have created the helper function on the primary, so its available for both primary and replicas:
CREATE OR REPLACE FUNCTION pganalyze.reset_stat_statements() RETURNS SETOF void AS
$$
/* pganalyze-collector */ SELECT * FROM public.pg_stat_statements_reset();
$$ LANGUAGE sql VOLATILE SECURITY DEFINER;
(to be created as a user that has superuser / rds_superuser or similar permissions)
from collector.
@lfittl it's happening again with all the mirrors. I already had the reset set to 24h for all mirrors. Should change the reset to every 12h, or maybe increase the timeout? The statement timeout is at 30s and the query text timeout is at 120s.
from collector.
@lfittl it's happening again with all the mirrors. I already had the reset set to 24h for all mirrors. Should change the reset to every 12h, or maybe increase the timeout? The statement timeout is at 30s and the query text timeout is at 120s.
Yeah, reducing the reset time interval sounds like a good next step. Also its worth double checking that you can run SELECT pganalyze.reset_stat_statements()
as the pganalyze user (to make sure there are no permission issues).
from collector.
Reducing the reset time interval didn't work 😢 I confirmed that I can run reset stat statements. Maybe I should try increasing Collector: Query Text Timeout
to 300s?
from collector.
@caleb15 Lets get together in a call to debug this further - I'll send you an email to follow up in a moment.
from collector.
Related Issues (20)
- Scheduler: Don't schedule runner intervals on fixed cron expressions HOT 5
- Documentation/support for Google Cloud SQL Auth Proxy or Connector HOT 2
- Update collector README to reference pg_monitor role for setup instructions HOT 3
- Handle read-only filesystem gracefully HOT 1
- Make IdentifierMap cache TTL configurable HOT 3
- Could not collect activity for server HOT 8
- Unable to DB_USE_IAM_AUTH with aws RDS instance HOT 4
- Error calling pg_stat_statements_reset() as requested intermittently HOT 2
- Regular collector oom-kill at midnight UTC on Sundays HOT 1
- Publish Docker image to ECR public registry HOT 1
- Error to send snapshots HOT 2
- Upgrade go.uuid module HOT 3
- Missing image for v0.50.1 HOT 2
- Additional security settings for pganalyze collector HOT 3
- Add rudimentary test suite for Helm chart
- Debian build uses EOL version HOT 1
- Error: Failed to reload collector HOT 1
- OTEL connector - can it support Datadog APM traces for PostgreSQL HOT 2
- OTEL connector - can it support Datadog?
- Feature request: Add DB_PASSWORD_FILE or similar HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from collector.