datadog / pg_tracing Goto Github PK

Distributed Tracing for PostgreSQL

License: MIT License

Dockerfile 0.81% Makefile 0.97% PLpgSQL 16.21% C 80.74% Perl 1.27%

pg_tracing's Issues

Only first statement in multi-statement is traced

Steps to reproduce the issue

$ psql postgres -c "select count(*) from pg_am; select count(*) from pg_class"
 count 
-------
     7
(1 row)

 count 
-------
   420
(1 row)

$ psql postgres -c "select trace_id, span_id, span_type, span_operation from pg_tracing_consume_spans"
             trace_id             |     span_id      |  span_type   |       span_operation        
----------------------------------+------------------+--------------+-----------------------------
 266759934ee85fe1100bf7a07bd43421 | 1b40b22bc5da4fa8 | Planner      | Planner
 266759934ee85fe1100bf7a07bd43421 | 88167138ea5945e5 | Executor     | ExecutorRun
 266759934ee85fe1100bf7a07bd43421 | d10fe19e7702ecd0 | SeqScan      | SeqScan on pg_am
 266759934ee85fe1100bf7a07bd43421 | 85e37a824bf3962b | Aggregate    | Aggregate
 266759934ee85fe1100bf7a07bd43421 | 7ffdd4d4338ba069 | Select query | select count(*) from pg_am;
(5 rows)

There are no spans for the second query, select count(*) from pg_class.

Make the service name configurable

I was looking for a way to do this and came across:

pg_tracing/src/pg_tracing_json.c

Line 139 in e9cc2d0

/* TODO: Make the service name configurable */

Has anyone implemented this yet, I had a stab at it, but didn't quite get it to work.

there are memory leak with current_trace_spans.

Environment details (Operating System, PostgreSQL version, pg_tracing version, etc)

PostgreSQL version: 16.2
pg_tracing version:latest
Operating System:centos 7

Steps to reproduce the issue

1.when the trace complete with a sql command, max_nested_level will be -1 again.
2.when max_nested_level is -1, current_trace_spans will malloc again.
3.but current_trace_spans only malloc, not free.

Describe the results you received

memory leak in every trace.

Describe the results you expected

memory leak in every trace.

Additional information you deem important (e.g. issue happens only occasionally)

Assertion failure with cursor

postgres=# create table foo (i int);
CREATE TABLE
postgres=# insert into foo select g from generate_series(1, 100000) g;
INSERT 0 100000
postgres=# set pg_tracing.sample_rate = 1.0;
SET
postgres=# begin;
BEGIN
postgres=*# declare foocur CURSOR FOR SELECT * from foo;
DECLARE CURSOR
postgres=*# fetch forward 10 from foocur;
server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.

TRAP: failed Assert("nested_level <= max_nested_level"), File: "src/pg_tracing.c", Line: 620, PID: 510625
postgres: heikki postgres [local] FETCH(ExceptionalCondition+0xaf)[0x55fb74f9651f]
/home/heikki/pgsql.master/lib/pg_tracing.so(+0x7242)[0x7f96b5563242]
/home/heikki/pgsql.master/lib/pg_tracing.so(+0x4940)[0x7f96b5560940]
postgres: heikki postgres [local] FETCH(ExecutorRun+0x45)[0x55fb74a77db5]
postgres: heikki postgres [local] FETCH(+0x720924)[0x55fb74d76924]
postgres: heikki postgres [local] FETCH(+0x7217ec)[0x55fb74d777ec]
postgres: heikki postgres [local] FETCH(PortalRunFetch+0x179)[0x55fb74d77149]
postgres: heikki postgres [local] FETCH(PerformPortalFetch+0x191)[0x55fb749d7f31]
postgres: heikki postgres [local] FETCH(standard_ProcessUtility+0x515)[0x55fb74d78b25]
/home/heikki/pgsql.master/lib/pg_tracing.so(+0x5366)[0x7f96b5561366]
postgres: heikki postgres [local] FETCH(ProcessUtility+0x132)[0x55fb74d785d2]
postgres: heikki postgres [local] FETCH(+0x721c7a)[0x55fb74d77c7a]
postgres: heikki postgres [local] FETCH(+0x720734)[0x55fb74d76734]
postgres: heikki postgres [local] FETCH(PortalRun+0x29f)[0x55fb74d7637f]
postgres: heikki postgres [local] FETCH(+0x71b727)[0x55fb74d71727]
postgres: heikki postgres [local] FETCH(PostgresMain+0x922)[0x55fb74d70992]
postgres: heikki postgres [local] FETCH(+0x713063)[0x55fb74d69063]
postgres: heikki postgres [local] FETCH(postmaster_child_launch+0xe7)[0x55fb74c69867]
postgres: heikki postgres [local] FETCH(+0x61a6fa)[0x55fb74c706fa]
postgres: heikki postgres [local] FETCH(+0x617b99)[0x55fb74c6db99]
postgres: heikki postgres [local] FETCH(PostmasterMain+0x18e7)[0x55fb74c6cd87]
postgres: heikki postgres [local] FETCH(+0x4bd7dd)[0x55fb74b137dd]
/lib/x86_64-linux-gnu/libc.so.6(+0x2724a)[0x7f96b484824a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7f96b4848305]
postgres: heikki postgres [local] FETCH(_start+0x21)[0x55fb74730b21]
2024-05-06 20:44:22.611 EEST [510617] LOG:  server process (PID 510625) was terminated by signal 6: Aborted
2024-05-06 20:44:22.611 EEST [510617] DETAIL:  Failed process was running: fetch forward 10 from foocur;
2024-05-06 20:44:22.611 EEST [510617] LOG:  terminating any other active server processes

Question: redundant computation of span_id in TracedPlanstate

Hi @bonnefoa , I'm recently reading through the code and would love to contribute to this project.

However, the logic in pg_tracing_planstate.c is quite complicated. I noticed that you override the ExecProcNode function to record the start time of each node. Besides a span_id is alse generated for each node in ExecProcNodeFirstPgTracing. But at the end of the query, when you call create_spans_from_planstate to walk through the planstate tree, you compute a random span id for each node again (line 439 in file pg_tracing_planstate.c).

I'm just wondering why don't we just reuse the span ids generated in traced_planstate?

Thanks for your time.

Discussion: Supporting OTLP protocol directly

Problem

Currently, you need another program to poll the pg_tracing_consume_spans view and push the spans to an OpenTelemetry Collector. See https://github.com/bonnefoa/pg-tracing-otel-forwarder. I wish the extension could connect directly to the OpenTelemetry Collector, using the OTLP protocol. That would simplify the deployment.

This was discussed on pgsql-hackers: https://www.postgresql.org/message-id/CAO6_Xqr5cNC-fK7kX4Pt9LkYSNjuZQjDfxbm_ckypM9LD8PT1Q%40mail.gmail.com. A pull model made sense for a patch for core PostgreSQL, to avoid having a dependency on HTTP or gRPC or other heavy libraries. But as an extension, that's less of an issue.

Alternatives

One approach is to implement the simplest OTLP/HTTP in JSON protocol using libcurl. Libcurl is widely available and doesn't require a lot of extra baggage.

Another approach would be to add a dependency to the OpenTelemetry C++ client library (https://github.com/open-telemetry/opentelemetry-cpp). That would come with more full support for different transports, but is a much bigger dependency.

Yet another approach would be to bundle a standalone binary like pg-tracing-otel-forwarder with this extensions. The extension could launch it on startup as a separate process, for example. Or just improve the binary to make it nicer to deploy, even though it needs to be deployed separately.

I'm leaning towards the libcurl approach myself. What do you think?

If we go down this route, we could simplify the rest of the code by removing the pull model. It's nice to have something to play with, though, even when you don't have OpenTelemetry colletor or something like Jaeger running. Maybe add an option to dump all the spans to a file instead of pushing them with OTLP?

Crash on SQL comments on explicit BEGIN

postgres=# /*traceparent='00-00000000000000000000000000000128-0000000000000128-01'*/begin; insert into foo values(1); commit;
BEGIN
server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.

TRAP: failed Assert("!traceid_zero(span->trace_id)"), File: "src/pg_tracing_span.c", Line: 127, PID: 556201
postgres: heikki postgres ::1(42026) INSERT(ExceptionalCondition+0xaf)[0x55900b67551f]
/home/heikki/pgsql.master/lib/pg_tracing.so(+0xc978)[0x7f045eeb0978]
/home/heikki/pgsql.master/lib/pg_tracing.so(+0x7527)[0x7f045eeab527]
/home/heikki/pgsql.master/lib/pg_tracing.so(+0x45f2)[0x7f045eea85f2]
postgres: heikki postgres ::1(42026) INSERT(planner+0x3d)[0x55900b2d09ad]
postgres: heikki postgres ::1(42026) INSERT(pg_plan_query+0x94)[0x55900b44d424]
postgres: heikki postgres ::1(42026) INSERT(pg_plan_queries+0x139)[0x55900b44d5d9]
postgres: heikki postgres ::1(42026) INSERT(+0x71b575)[0x55900b450575]
postgres: heikki postgres ::1(42026) INSERT(PostgresMain+0x922)[0x55900b44f992]
postgres: heikki postgres ::1(42026) INSERT(+0x713063)[0x55900b448063]
postgres: heikki postgres ::1(42026) INSERT(postmaster_child_launch+0xe7)[0x55900b348867]
postgres: heikki postgres ::1(42026) INSERT(+0x61a6fa)[0x55900b34f6fa]
postgres: heikki postgres ::1(42026) INSERT(+0x617b99)[0x55900b34cb99]
postgres: heikki postgres ::1(42026) INSERT(PostmasterMain+0x18e7)[0x55900b34bd87]
postgres: heikki postgres ::1(42026) INSERT(+0x4bd7dd)[0x55900b1f27dd]
/lib/x86_64-linux-gnu/libc.so.6(+0x2724a)[0x7f045e24824a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7f045e248305]
postgres: heikki postgres ::1(42026) INSERT(_start+0x21)[0x55900ae0fb21]
2024-05-07 14:14:26.263 EEST [556193] LOG:  server process (PID 556201) was terminated by signal 6: Aborted
2024-05-07 14:14:26.263 EEST [556193] DETAIL:  Failed process was running: insert into foo values(1);
2024-05-07 14:14:26.263 EEST [556193] LOG:  terminating any other active server processes
2024-05-07 14:14:26.264 EEST [556193] LOG:  all server processes terminated; reinitializing

visualize tracing information

Hi @bonnefoa I'd like to contribute the following code snippet for visualizing the tracing information in a hierarchy way. Hope this can help some users.

WITH RECURSIVE subspan AS (
   SELECT span_operation, parent_id, span_id, span_start, 1 AS level
   FROM   pg_tracing_spans(false)
   WHERE  trace_id = '00000000000000000000000000000002' and parent_id = '0000000000000002'

   UNION  ALL
   SELECT lpad(s.span_operation, length(s.span_operation) + (ss.level + 1) * 2 - 2, '_'), s.parent_id, s.span_id, s.span_start, ss.level + 1
   FROM   pg_tracing_spans(false) as s
   JOIN   subspan ss ON s.parent_id = ss.span_id
)
SELECT *
FROM   subspan
ORDER  BY span_start;

                                          span_operation                                           |    parent_id     |     span_id      |          span_start           | level 
---------------------------------------------------------------------------------------------------+------------------+------------------+-------------------------------+-------
 WITH cte_basic AS MATERIALIZED (SELECT $1 a, $2 b)                                               +| 0000000000000002 | f5b760b3dafe8617 | 2024-07-31 07:13:20.860781-06 |     1
 MERGE INTO m USING (select $3 k, $4 v offset $5) o ON m.k=o.k                                    +|                  |                  |                               | 
 WHEN MATCHED THEN UPDATE SET v = (SELECT b || $6 FROM cte_basic WHERE cte_basic.a = m.k LIMIT $7)+|                  |                  |                               | 
 WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v);                                                    |                  |                  |                               | 
 __Planner                                                                                         | f5b760b3dafe8617 | a43900f7a5c5ccec | 2024-07-31 07:13:20.860841-06 |     2
 __ExecutorRun                                                                                     | f5b760b3dafe8617 | 584dac0889104c61 | 2024-07-31 07:13:20.861172-06 |     2
 ____Merge on m                                                                                    | 584dac0889104c61 | 2d1637fe1c0f1a9e | 2024-07-31 07:13:20.861176-06 |     3
 ______Hash Right Join                                                                             | 2d1637fe1c0f1a9e | 82c89d52fda4ca70 | 2024-07-31 07:13:20.861178-06 |     4
 ________Hash                                                                                      | 82c89d52fda4ca70 | 1791f6d90d1b7a86 | 2024-07-31 07:13:20.861197-06 |     5
 __________SubqueryScan on o                                                                       | 1791f6d90d1b7a86 | 9d46a6e050ec12ba | 2024-07-31 07:13:20.861197-06 |     6
 ____________Result                                                                                | 9d46a6e050ec12ba | 8d9bbeb68360ec9b | 2024-07-31 07:13:20.861198-06 |     7
 ________SeqScan on m                                                                              | 82c89d52fda4ca70 | 6a557cb92647d63e | 2024-07-31 07:13:20.861238-06 |     5
 ______SubPlan 2                                                                                   | 2d1637fe1c0f1a9e | cf2ac26ad5b4d2e4 | 2024-07-31 07:13:20.861265-06 |     4
 ________Limit                                                                                     | cf2ac26ad5b4d2e4 | 7e0dff38fb4ba194 | 2024-07-31 07:13:20.861265-06 |     5
 __________CTEScan on cte_basic                                                                    | 7e0dff38fb4ba194 | 7ba294f4d5d648df | 2024-07-31 07:13:20.861266-06 |     6
 ________Result                                                                                    | 79529943ec0d0057 | 0414b9993d556107 | 2024-07-31 07:13:20.861267-06 |     5
 ______CTE cte_basic                                                                               | 2d1637fe1c0f1a9e | 79529943ec0d0057 | 2024-07-31 07:13:20.861267-06 |     4
 TransactionCommit                                                                                 | 0000000000000002 | 7c9bd44b2efe552d | 2024-07-31 07:13:20.861402-06 |     1
(15 rows)

measuring network time in pg tracing

Hi,

pg_tracing is a great work. I'm just wondering is it possible to record the span for network transmission between the psql client and database server because the network can be a bottleneck in some cases.

datadog / pg_tracing Goto Github PK

pg_tracing's Issues

Steps to reproduce the issue

Environment details (Operating System, PostgreSQL version, pg_tracing version, etc)

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

Additional information you deem important (e.g. issue happens only occasionally)

Problem

Alternatives

Recommend Projects

Recommend Topics

Recommend Org

Jobs