GithubHelp home page GithubHelp logo

Comments (11)

lfittl avatar lfittl commented on July 24, 2024 1

@caleb15 Yeah, 24h is a good place to start - in my experience that resolves the issues 90% of the time.

The reason the statement_timeout setting doesn't take effect is because the collector sets a session-local session timeout:

func SetQueryTextStatementTimeout(connection *sql.DB, logger *util.Logger, server *state.Server) {
queryTextStatementTimeoutMs := server.Grant.Config.Features.StatementTimeoutMsQueryText
if queryTextStatementTimeoutMs == 0 { // Default value
queryTextStatementTimeoutMs = 120000
}
SetStatementTimeout(connection, queryTextStatementTimeoutMs)
return
}
(120 seconds)

from collector.

caleb15 avatar caleb15 commented on July 24, 2024 1

This appears to have worked, thanks!

from collector.

caleb15 avatar caleb15 commented on July 24, 2024

The pganalyze container pointed at the heavy mirror is using a bunch of memory compared to the other instances for some reason:

CONTAINER ID        NAME                     CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
490ccc38d41d        pganalyze-mirror-heavy   0.00%               2.991GiB / 7.698GiB   38.85%              10.2GB / 6.79GB     26.1MB / 1.09GB     10
36389b04b5f7        pganalyze-mirror1        0.00%               1.342GiB / 7.698GiB   17.44%              13.7GB / 3.46GB     167MB / 9.24GB      10
8f4c79d8ea41        pganalyze-mirror         0.00%               1.472GiB / 7.698GiB   19.12%              13.8GB / 3.48GB     228MB / 9.1GB       10
246474c64fb0        pganalyze                0.00%               97.4MiB / 7.698GiB    1.24%               10.2GB / 2.33GB     157MB / 338MB       10

May be unrelated.

from collector.

lfittl avatar lfittl commented on July 24, 2024

@caleb15 Could you try doing a "SELECT pg_stat_statements_reset()" to see if that addresses the timeouts?

(when the Postgres query text file grows very large, retrieving the text sometimes takes too long)

from collector.

caleb15 avatar caleb15 commented on July 24, 2024

Sure, I'll try that out, although I still don't see how it can be timing out with statement timeouts disabled.

Unrelated question: when you recommended that I set Collector: Query Statistics Reset Frequency what value did you recommend again? 24h?

from collector.

caleb15 avatar caleb15 commented on July 24, 2024

I enabled it, I'll see if that fixes it. For the Collector: Query Statistics Reset Frequency should I just set that to 24h for the master or do it for the master and the mirrors?

from collector.

lfittl avatar lfittl commented on July 24, 2024

@caleb15 I would set it on all the servers where you are seeing timeouts.

In addition, make sure you have created the helper function on the primary, so its available for both primary and replicas:

CREATE OR REPLACE FUNCTION pganalyze.reset_stat_statements() RETURNS SETOF void AS
$$
  /* pganalyze-collector */ SELECT * FROM public.pg_stat_statements_reset();
$$ LANGUAGE sql VOLATILE SECURITY DEFINER;

(to be created as a user that has superuser / rds_superuser or similar permissions)

from collector.

caleb15 avatar caleb15 commented on July 24, 2024

@lfittl it's happening again with all the mirrors. I already had the reset set to 24h for all mirrors. Should change the reset to every 12h, or maybe increase the timeout? The statement timeout is at 30s and the query text timeout is at 120s.

from collector.

lfittl avatar lfittl commented on July 24, 2024

@lfittl it's happening again with all the mirrors. I already had the reset set to 24h for all mirrors. Should change the reset to every 12h, or maybe increase the timeout? The statement timeout is at 30s and the query text timeout is at 120s.

Yeah, reducing the reset time interval sounds like a good next step. Also its worth double checking that you can run SELECT pganalyze.reset_stat_statements() as the pganalyze user (to make sure there are no permission issues).

from collector.

caleb15 avatar caleb15 commented on July 24, 2024

Reducing the reset time interval didn't work 😢 I confirmed that I can run reset stat statements. Maybe I should try increasing Collector: Query Text Timeout to 300s?

from collector.

lfittl avatar lfittl commented on July 24, 2024

@caleb15 Lets get together in a call to debug this further - I'll send you an email to follow up in a moment.

from collector.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.