[Bug]: Incorrect bloat calculation in go.d.plugin, postgres module about netdata

[cuu508]

In my case, the dashboard was showing the outdated value for hours (and a warning was active too) .

Also, running the bloat calculation query manually did not return the calculated bloat value for that specific index at all (the row was absent). In summary:

• netdata dashboard (both local node and netdata cloud) was showing an outdated value, 72%
• a different bloat calculation query was showing 21%
• netdata's bloat calculation query was omitting that specific index from bloat calculation results

[cuu508]

PS PostgreSQL version:

select version();
                                                               version                                                               
-------------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 13.14 (Ubuntu 13.14-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit

[ilyam8]

@cuu508, hey 👋 While go.d.plugin gathers data every update_every, it collects and calculates bloat statistics less frequently, only every 5 minutes. In between calculations, go.d.plugin relies on the most recently calculated bloat statistics (cached data). This is because the bloat query can be resource-intensive. This 5-minute interval is not configurable at the moment.

[ilyam8]

@cuu508 I understood that, thanks! I think I fixed it in #17598

a different bloat calculation query was showing 21%

Netdata uses bloat query from bucardo/check_postgres. I am not a Postgres expert, so I can't say which query is better/more correct.

[cuu508]

Ah, sorry, I missed the referenced commit!

I compared bucardo's and netdata's bloat calculation queries. They are identical except the netdata version has at the very end:

WHERE sml.relpages - otta > 10 OR ipages - iotta > 10

I think this is the bit responsible for netdata missing bloat data for some of my indexes. I have one particular index for which bucardo's query calculates ibloat as 0.5:

-[ RECORD 91 ]---+-----------------------------------------------------
db               | ***
schemaname       | public
tablename        | api_ping
tups             | 108233496
pages            | 1469731
otta             | 1637553
tbloat           | 0.9
wastedpages      | 0
wastedbytes      | 0
wastedsize       | 0 bytes
iname            | api_ping_pkey
itups            | 107589248
ipages           | 522850
iotta            | 1000750
ibloat           | 0.5
wastedipages     | 0
wastedibytes     | 0
wastedisize      | 0 bytes
totalwastedbytes | 0

But since ipages is lower than iotta, it is omitted from the netdata version.

The other query calculates the bloat as 43% for this same index. (edit:) but some of the calculated bloat values looked wacky, -247% and such. In the bucardo query at least all ibloat values were zero or positive)

[ilyam8]

@cuu508 bucardos query has this otta/iotta limit (a bit bigger)

    ## Don't bother with tables or indexes unless they have at least this many bloated pages
    my $MINPAGES = 10;
    my $MINIPAGES = 15;

See:

• limits defined
• query updated