wolph / pg_query_analyser

A PostgreSQL query analyzer written in C++. Mimicks the output of PgFouine but _much_ faster.

pg_query_analyser -- PostgreSQL Slow Query Log parser

Overview

pg_query_analyser is a C++ clone of the PgFouine log analyser.

Processing logs with millions of lines only takes a few minutes with this parser while PgFouine chokes long before that.

Example output

The normal overview:

The overview with the examples expanded:

Requirements

Ubuntu (tested with 13.10):

apt-get install qt4-dev-tools

Ubuntu (tested with 16.04.2 LTS)

apt-get install qt4-dev-tools qt4-qmake libqt4-dev

Install

qmake
make
make test
sudo make install

Usage

Set Postgres to use this `log_line_prefix`:

log_line_prefix = '%t [%p]: [%l-1] host=%h,user=%u,db=%d,tx=%x,vtx=%v '

After that we can start parsing data.

From stdin:

cat /var/log/postgresql/postgresql.log | head -n 100000 | ./pg_query_analyser -i -

From file:

./pg_query_analyser --input-file=/var/log/postgresql/postgresql.log

For a full build+analyze on Ubuntu:

cd examples
fab -H <remote-ubuntu-postgres-server> build log_and_analyse

Help

# ./pg_query_analyser -h
Usage: ./pg_query_analyser [flags]
Options: 
  -h, --help=[false]
  -u, --users=[]
  -v, --verbose=[false]
  -i, --input-file=[/var/log/postgresql/postgresql-9.1-main.log]
  -d, --databases=[]
  -t, --query-types=[SELECT,UPDATE,INSERT,DELETE]
  -o, --output-file=[report.html]
  --top=[20]

pg_query_analyser's Issues

Whats supposed to be the log_line_prefix in PostgreSQL for pg_query_analyser ?

I am having formatting issues with the output html report generated by pg_query_analyser. The log_line_prefix used in postgresql.conf is '%m %r [%u@%d] [%l-1] '.

Please help !

Regards,
Venkata B N

Doesn't compile on Qt5

can't compile on ubuntu: QBool: No such file or directory

Hi,
Can you give hint which packages are needed to compile?

make

g++ -c -m64 -pipe -O2 -Wall -W -D_REENTRANT -fPIE -DQT_NO_DEBUG -DQT_SQL_LIB -DQT_CORE_LIB -I../../../share/qt5/mkspecs/linux-g++-64 -I. -I../../../include/qt5 -I../../../include/qt5/QtSql -I../../../include/qt5/QtCore -I. -o main.o main.cpp
In file included from args.h:10:0,
from main.h:12,
from main.cpp:1:
arg.h:6:17: fatal error: QBool: No such file or directory
#include
^
compilation terminated.
make: *** [main.o] Error 1

Duration not processed

Hey,
I have strange problem with your tool, everything works fine except zeros in every query "duration time".
Im not sure, problem is in our log format (configured like in documentation) or with something else, but i really need this information. Can you help me with it?
Below one query from log.

P.S. We use patroni over our pg cluster, i hope its not a problem.

BR
Bart

Dec 15 06:25:40 ip-172-26-100-121 patroni[26448]: 2017-12-15 06:25:40 UTC [26476]: [4087633-1] host=127.0.0.1,user=postgres,db=postgres,tx=0,vtx=2/2043829 LOG:  statement: SELECT to_char(pg_postmaster_start_time(), 'YYYY-MM-DD HH24:MI:SS.MS TZ'),
Dec 15 06:25:40 ip-172-26-100-121 patroni[26448]: #011                                       pg_is_in_recovery(),
Dec 15 06:25:40 ip-172-26-100-121 patroni[26448]: #011                                       CASE WHEN pg_is_in_recovery()
Dec 15 06:25:40 ip-172-26-100-121 patroni[26448]: #011                                            THEN 0
Dec 15 06:25:40 ip-172-26-100-121 patroni[26448]: #011                                            ELSE pg_xlog_location_diff(pg_current_xlog_location(), '0/0')::bigint
Dec 15 06:25:40 ip-172-26-100-121 patroni[26448]: #011                                       END,
Dec 15 06:25:40 ip-172-26-100-121 patroni[26448]: #011                                       pg_xlog_location_diff(pg_last_xlog_receive_location(), '0/0')::bigint,
Dec 15 06:25:40 ip-172-26-100-121 patroni[26448]: #011                                       pg_xlog_location_diff(pg_last_xlog_replay_location(), '0/0')::bigint,
Dec 15 06:25:40 ip-172-26-100-121 patroni[26448]: #011                                       to_char(pg_last_xact_replay_timestamp(), 'YYYY-MM-DD HH24:MI:SS.MS TZ'),
Dec 15 06:25:40 ip-172-26-100-121 patroni[26448]: #011                                       pg_is_in_recovery() AND pg_is_xlog_replay_paused()
Dec 15 06:25:40 ip-172-26-100-121 patroni[26448]: 2017-12-15 06:25:40 UTC [26476]: [4087634-1] host=127.0.0.1,user=postgres,db=postgres,tx=0,vtx=2/0 LOG:  duration: 0.251 ms

--output-file expects the output file to exist

at least on OSX :)