datadog / pg_tracing Goto Github PK
View Code? Open in Web Editor NEWDistributed Tracing for PostgreSQL
License: MIT License
Distributed Tracing for PostgreSQL
License: MIT License
$ psql postgres -c "select count(*) from pg_am; select count(*) from pg_class"
count
-------
7
(1 row)
count
-------
420
(1 row)
$ psql postgres -c "select trace_id, span_id, span_type, span_operation from pg_tracing_consume_spans"
trace_id | span_id | span_type | span_operation
----------------------------------+------------------+--------------+-----------------------------
266759934ee85fe1100bf7a07bd43421 | 1b40b22bc5da4fa8 | Planner | Planner
266759934ee85fe1100bf7a07bd43421 | 88167138ea5945e5 | Executor | ExecutorRun
266759934ee85fe1100bf7a07bd43421 | d10fe19e7702ecd0 | SeqScan | SeqScan on pg_am
266759934ee85fe1100bf7a07bd43421 | 85e37a824bf3962b | Aggregate | Aggregate
266759934ee85fe1100bf7a07bd43421 | 7ffdd4d4338ba069 | Select query | select count(*) from pg_am;
(5 rows)
There are no spans for the second query, select count(*) from pg_class
.
I was looking for a way to do this and came across:
pg_tracing/src/pg_tracing_json.c
Line 139 in e9cc2d0
Has anyone implemented this yet, I had a stab at it, but didn't quite get it to work.
1.when the trace complete with a sql command, max_nested_level will be -1 again.
2.when max_nested_level is -1, current_trace_spans will malloc again.
3.but current_trace_spans only malloc, not free.
memory leak in every trace.
memory leak in every trace.
postgres=# create table foo (i int);
CREATE TABLE
postgres=# insert into foo select g from generate_series(1, 100000) g;
INSERT 0 100000
postgres=# set pg_tracing.sample_rate = 1.0;
SET
postgres=# begin;
BEGIN
postgres=*# declare foocur CURSOR FOR SELECT * from foo;
DECLARE CURSOR
postgres=*# fetch forward 10 from foocur;
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
TRAP: failed Assert("nested_level <= max_nested_level"), File: "src/pg_tracing.c", Line: 620, PID: 510625
postgres: heikki postgres [local] FETCH(ExceptionalCondition+0xaf)[0x55fb74f9651f]
/home/heikki/pgsql.master/lib/pg_tracing.so(+0x7242)[0x7f96b5563242]
/home/heikki/pgsql.master/lib/pg_tracing.so(+0x4940)[0x7f96b5560940]
postgres: heikki postgres [local] FETCH(ExecutorRun+0x45)[0x55fb74a77db5]
postgres: heikki postgres [local] FETCH(+0x720924)[0x55fb74d76924]
postgres: heikki postgres [local] FETCH(+0x7217ec)[0x55fb74d777ec]
postgres: heikki postgres [local] FETCH(PortalRunFetch+0x179)[0x55fb74d77149]
postgres: heikki postgres [local] FETCH(PerformPortalFetch+0x191)[0x55fb749d7f31]
postgres: heikki postgres [local] FETCH(standard_ProcessUtility+0x515)[0x55fb74d78b25]
/home/heikki/pgsql.master/lib/pg_tracing.so(+0x5366)[0x7f96b5561366]
postgres: heikki postgres [local] FETCH(ProcessUtility+0x132)[0x55fb74d785d2]
postgres: heikki postgres [local] FETCH(+0x721c7a)[0x55fb74d77c7a]
postgres: heikki postgres [local] FETCH(+0x720734)[0x55fb74d76734]
postgres: heikki postgres [local] FETCH(PortalRun+0x29f)[0x55fb74d7637f]
postgres: heikki postgres [local] FETCH(+0x71b727)[0x55fb74d71727]
postgres: heikki postgres [local] FETCH(PostgresMain+0x922)[0x55fb74d70992]
postgres: heikki postgres [local] FETCH(+0x713063)[0x55fb74d69063]
postgres: heikki postgres [local] FETCH(postmaster_child_launch+0xe7)[0x55fb74c69867]
postgres: heikki postgres [local] FETCH(+0x61a6fa)[0x55fb74c706fa]
postgres: heikki postgres [local] FETCH(+0x617b99)[0x55fb74c6db99]
postgres: heikki postgres [local] FETCH(PostmasterMain+0x18e7)[0x55fb74c6cd87]
postgres: heikki postgres [local] FETCH(+0x4bd7dd)[0x55fb74b137dd]
/lib/x86_64-linux-gnu/libc.so.6(+0x2724a)[0x7f96b484824a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7f96b4848305]
postgres: heikki postgres [local] FETCH(_start+0x21)[0x55fb74730b21]
2024-05-06 20:44:22.611 EEST [510617] LOG: server process (PID 510625) was terminated by signal 6: Aborted
2024-05-06 20:44:22.611 EEST [510617] DETAIL: Failed process was running: fetch forward 10 from foocur;
2024-05-06 20:44:22.611 EEST [510617] LOG: terminating any other active server processes
Hi @bonnefoa , I'm recently reading through the code and would love to contribute to this project.
However, the logic in pg_tracing_planstate.c is quite complicated. I noticed that you override the ExecProcNode function to record the start time of each node. Besides a span_id is alse generated for each node in ExecProcNodeFirstPgTracing. But at the end of the query, when you call create_spans_from_planstate to walk through the planstate tree, you compute a random span id for each node again (line 439 in file pg_tracing_planstate.c).
I'm just wondering why don't we just reuse the span ids generated in traced_planstate?
Thanks for your time.
Currently, you need another program to poll the pg_tracing_consume_spans
view and push the spans to an OpenTelemetry Collector. See https://github.com/bonnefoa/pg-tracing-otel-forwarder. I wish the extension could connect directly to the OpenTelemetry Collector, using the OTLP protocol. That would simplify the deployment.
This was discussed on pgsql-hackers: https://www.postgresql.org/message-id/CAO6_Xqr5cNC-fK7kX4Pt9LkYSNjuZQjDfxbm_ckypM9LD8PT1Q%40mail.gmail.com. A pull model made sense for a patch for core PostgreSQL, to avoid having a dependency on HTTP or gRPC or other heavy libraries. But as an extension, that's less of an issue.
One approach is to implement the simplest OTLP/HTTP in JSON protocol using libcurl. Libcurl is widely available and doesn't require a lot of extra baggage.
Another approach would be to add a dependency to the OpenTelemetry C++ client library (https://github.com/open-telemetry/opentelemetry-cpp). That would come with more full support for different transports, but is a much bigger dependency.
Yet another approach would be to bundle a standalone binary like pg-tracing-otel-forwarder with this extensions. The extension could launch it on startup as a separate process, for example. Or just improve the binary to make it nicer to deploy, even though it needs to be deployed separately.
I'm leaning towards the libcurl approach myself. What do you think?
If we go down this route, we could simplify the rest of the code by removing the pull model. It's nice to have something to play with, though, even when you don't have OpenTelemetry colletor or something like Jaeger running. Maybe add an option to dump all the spans to a file instead of pushing them with OTLP?
postgres=# /*traceparent='00-00000000000000000000000000000128-0000000000000128-01'*/begin; insert into foo values(1); commit;
BEGIN
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
TRAP: failed Assert("!traceid_zero(span->trace_id)"), File: "src/pg_tracing_span.c", Line: 127, PID: 556201
postgres: heikki postgres ::1(42026) INSERT(ExceptionalCondition+0xaf)[0x55900b67551f]
/home/heikki/pgsql.master/lib/pg_tracing.so(+0xc978)[0x7f045eeb0978]
/home/heikki/pgsql.master/lib/pg_tracing.so(+0x7527)[0x7f045eeab527]
/home/heikki/pgsql.master/lib/pg_tracing.so(+0x45f2)[0x7f045eea85f2]
postgres: heikki postgres ::1(42026) INSERT(planner+0x3d)[0x55900b2d09ad]
postgres: heikki postgres ::1(42026) INSERT(pg_plan_query+0x94)[0x55900b44d424]
postgres: heikki postgres ::1(42026) INSERT(pg_plan_queries+0x139)[0x55900b44d5d9]
postgres: heikki postgres ::1(42026) INSERT(+0x71b575)[0x55900b450575]
postgres: heikki postgres ::1(42026) INSERT(PostgresMain+0x922)[0x55900b44f992]
postgres: heikki postgres ::1(42026) INSERT(+0x713063)[0x55900b448063]
postgres: heikki postgres ::1(42026) INSERT(postmaster_child_launch+0xe7)[0x55900b348867]
postgres: heikki postgres ::1(42026) INSERT(+0x61a6fa)[0x55900b34f6fa]
postgres: heikki postgres ::1(42026) INSERT(+0x617b99)[0x55900b34cb99]
postgres: heikki postgres ::1(42026) INSERT(PostmasterMain+0x18e7)[0x55900b34bd87]
postgres: heikki postgres ::1(42026) INSERT(+0x4bd7dd)[0x55900b1f27dd]
/lib/x86_64-linux-gnu/libc.so.6(+0x2724a)[0x7f045e24824a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7f045e248305]
postgres: heikki postgres ::1(42026) INSERT(_start+0x21)[0x55900ae0fb21]
2024-05-07 14:14:26.263 EEST [556193] LOG: server process (PID 556201) was terminated by signal 6: Aborted
2024-05-07 14:14:26.263 EEST [556193] DETAIL: Failed process was running: insert into foo values(1);
2024-05-07 14:14:26.263 EEST [556193] LOG: terminating any other active server processes
2024-05-07 14:14:26.264 EEST [556193] LOG: all server processes terminated; reinitializing
Hi @bonnefoa I'd like to contribute the following code snippet for visualizing the tracing information in a hierarchy way. Hope this can help some users.
WITH RECURSIVE subspan AS (
SELECT span_operation, parent_id, span_id, span_start, 1 AS level
FROM pg_tracing_spans(false)
WHERE trace_id = '00000000000000000000000000000002' and parent_id = '0000000000000002'
UNION ALL
SELECT lpad(s.span_operation, length(s.span_operation) + (ss.level + 1) * 2 - 2, '_'), s.parent_id, s.span_id, s.span_start, ss.level + 1
FROM pg_tracing_spans(false) as s
JOIN subspan ss ON s.parent_id = ss.span_id
)
SELECT *
FROM subspan
ORDER BY span_start;
span_operation | parent_id | span_id | span_start | level
---------------------------------------------------------------------------------------------------+------------------+------------------+-------------------------------+-------
WITH cte_basic AS MATERIALIZED (SELECT $1 a, $2 b) +| 0000000000000002 | f5b760b3dafe8617 | 2024-07-31 07:13:20.860781-06 | 1
MERGE INTO m USING (select $3 k, $4 v offset $5) o ON m.k=o.k +| | | |
WHEN MATCHED THEN UPDATE SET v = (SELECT b || $6 FROM cte_basic WHERE cte_basic.a = m.k LIMIT $7)+| | | |
WHEN NOT MATCHED THEN INSERT VALUES(o.k, o.v); | | | |
__Planner | f5b760b3dafe8617 | a43900f7a5c5ccec | 2024-07-31 07:13:20.860841-06 | 2
__ExecutorRun | f5b760b3dafe8617 | 584dac0889104c61 | 2024-07-31 07:13:20.861172-06 | 2
____Merge on m | 584dac0889104c61 | 2d1637fe1c0f1a9e | 2024-07-31 07:13:20.861176-06 | 3
______Hash Right Join | 2d1637fe1c0f1a9e | 82c89d52fda4ca70 | 2024-07-31 07:13:20.861178-06 | 4
________Hash | 82c89d52fda4ca70 | 1791f6d90d1b7a86 | 2024-07-31 07:13:20.861197-06 | 5
__________SubqueryScan on o | 1791f6d90d1b7a86 | 9d46a6e050ec12ba | 2024-07-31 07:13:20.861197-06 | 6
____________Result | 9d46a6e050ec12ba | 8d9bbeb68360ec9b | 2024-07-31 07:13:20.861198-06 | 7
________SeqScan on m | 82c89d52fda4ca70 | 6a557cb92647d63e | 2024-07-31 07:13:20.861238-06 | 5
______SubPlan 2 | 2d1637fe1c0f1a9e | cf2ac26ad5b4d2e4 | 2024-07-31 07:13:20.861265-06 | 4
________Limit | cf2ac26ad5b4d2e4 | 7e0dff38fb4ba194 | 2024-07-31 07:13:20.861265-06 | 5
__________CTEScan on cte_basic | 7e0dff38fb4ba194 | 7ba294f4d5d648df | 2024-07-31 07:13:20.861266-06 | 6
________Result | 79529943ec0d0057 | 0414b9993d556107 | 2024-07-31 07:13:20.861267-06 | 5
______CTE cte_basic | 2d1637fe1c0f1a9e | 79529943ec0d0057 | 2024-07-31 07:13:20.861267-06 | 4
TransactionCommit | 0000000000000002 | 7c9bd44b2efe552d | 2024-07-31 07:13:20.861402-06 | 1
(15 rows)
Hi,
pg_tracing is a great work. I'm just wondering is it possible to record the span for network transmission between the psql client and database server because the network can be a bottleneck in some cases.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.