puppetlabs / puppetdb

Centralized Puppet Storage

PuppetDB

PuppetDB is the fast, scalable, and reliable data warehouse for Puppet. It caches data generated by Puppet, and gives you advanced features at awesome speed with a powerful API.

For documentation on this product, consult the latest documentation.

Contributing

If you would like to contribute to PuppetDB, please take a look at our contributing doc.

Maintenance

Community Help: Puppet Users Mailing List, Puppet Community Portal

Tickets: https://tickets.puppet.com/browse/PDB

puppetdb's Issues

puppetdb should be aware of environments.

Per discussion with @nicklewis , puppetdb should be aware of environments. Opening this issue as a reminder.

Consider regularly resetting factset stable/volatile balances

In an attempt to decrease the steady-state Postgres write load PuppetDB currently stores each factset in two separate jsonb fragments, stable and volatile. Whenever a fact changes, the top-level subtree that contains it is moved from stable to volatile, if it wasn't already in volatile, and never moves it back. So if a leaf fact in a large subtree changes once, and then never changes again, the entire subtree containing that fact will always add to the write load when updating that factset, something the split is intended to avoid. (See the footnote in #3955 for additional information.)

To avoid that, and as a fairly simple improvement, consider moving all of a factset's volatile facts back into stable on some cadence. For example, perhaps make the move during any given factset update based on a percentage that would reset a fact every day/week/... assuming the default factset submission rate,

Thanks to Austin Blatt for the initial suggestion.

Add configuration option to disable sending of catalog edges from terminus

Profiling catalog submission has shown that catalog edges can take up to one third of the total time to submit a catalog. They are not the most used, or important piece of catalog data, so many users (especially those with many nodes) would benefit from not storing edges and having faster catalog submission times.

Consider avoiding the "stable/volatile facts merge" in puppetdb SQL queries

In an attempt to decrease the steady-state Postgres write load PuppetDB currently stores each factset in two separate jsonb fragments, stable and volatile¹. Whenever a fact changes, the top-level subtree that contains it is moved from stable to volatile, if it wasn't already in volatile, and never moves it back. (See also: #3956)

When generating queries, PuppetDB just refers to (stable || volatile) via || which reconstructs the full factset, and (it turns out) can be notably expensive. During some scale testing, a simple inventory query effectively filtering on "somefact = x" was observed to take about five seconds with 100k nodes. During investigation, it looked like most of the time was being spent in the examination of the fact value. Rewriting the SQL from roughly (stable || volatile)->somefact to stable->somefact or volatile->somefact decreased the execution time to less than half a second.
Charlie also determined via perf that much of the extra time for the original version was being spent in the stable/volatile merge.

Since the stable/volatile split may be worth preserving¹, consider adjusting PuppetDB to work with the stable and volatile fragments directly (as in the "or" conversion above). This will also require the creation of independent stable and volatile indexes, and may or may not allow dropping the existing, combined (stable||volatile) expresion index.

Footnotes

1. The stable/volatile split relies on an assumption that in typical installations there will be a substantial set of facts that never change. When true, those facts will end up in the stable jsonb factsets column, and if they're also large enough to be TOASTed then the value in the factset row should just be an integer TOAST table rowid, shrinking the size of the row with respect to future rewrites during factset updates. From the TOAST page linked above: "During an UPDATE operation, values of unchanged fields are normally preserved as-is; so an UPDATE of a row with out-of-line values incurs no TOAST costs if none of the out-of-line values change." ↩ ↩²

Abandon queries when client disconnects

Testing revealed that pdb streaming queries definitely do continue running for a while after client disconnection. Long after the client disconnected, jetty would continue to call the query-eng/generated-stream read to stream the encoded json bytes which in turn keep the database query running. The current assumption is that the reads are caused by buffering before the next network write. A stack trace from the read reveals that there's a gzip handler, but since (at least from a cursory investigation) that handler appears to rely on a Deflate instance with a 512 byte buffer (fixed?), it's probably not the key reason for the continuing reads.

In any case, after further investigation, we didn't find a straightforward way to detect a client disconnection in the ring or tk-jetty9 world, but since the jetty/servlet response handler is given a ServletResponse object, which for jetty is an org.eclipse.jetty.server.Response, I wondered if that might provide access to the client connection. It does.

Jetty (9 at least) handles the connection via a chain of "interceptors" attached to the HttpOutput, which is itself attached to the Response. For a trivial jetty app there's just one interceptor, while pdb includes others (the gzip "layer" at least). The jetty HttpOutput Interceptor comments say that the last link in the chain will be an HttpChannel, and for pdb, that's currently an HttpChannelOverHttp instance, and that allows direct access to the jdk SocketChannel connected to the client.

Fortunately the client channel is also in non-blocking mode, and so even though the client isn't (and shouldn't be) sending any data, an attempt to read from the client channel will return 0 as long as the client is connected and -1 after it disconnects (verified via curl against both the simple test app and pdb itself). It's then straightforward to add a check in pdb's query-eng/body-stream's stream-rows that will cause pdb to abandon the query whenever the read indicates that the client is gone.

This approach relies on some assumptions:

The jetty transport is non-blocking.

The read from the client always returns -1 after a disconnect.

By the time pdb is streaming, it's acceptable to attempt to read bytes from the client, i.e. either the client won't be sending any bytes, or it's OK to discard them.

It would also require a change to trapperkeeper-webserver-jetty9, because ring handlers (both in ring itself and in tk-jetty9) don't provide the response instance to the handler, just the request map. One option would be to add an :include-response? option to the add-ring-handler service method. When true the handler call would become (handle request-map response) instead of (handle request-map).

Multiple PuppetDB instances against a single PostgreSQL database

Describe the Bug

Hello and thanks for developing PuppetDB!

Based on this sentence:

Run multiple instances of PuppetDB on multiple servers, and use a reverse proxy or load balancer to distribute traffic between them.

we are testing a setup with two (or more) PuppetDB instances using the same PostgreSQL database.

So let's assume the following:

• pgsql001 hosts the PostgreSQL database named puppetdb
• puppet001 runs both puppetserver and puppetdb and points to pgsql001 database
• puppet002 runs both puppetserver and puppetdb and points to pgsql001 database

Both PuppetDB instances are employing the very same credentials against a single database. puppet001 and puppet002 are used by a different set of puppet-agents, for example hosts in two different datacenters.

puppetdb logs feature no error but postgresql logs do:

2024-02-06 14:01:33.459 UTC [812825] puppetdb@puppetdb ERROR:  could not serialize access due to concurrent update
2024-02-06 14:01:33.459 UTC [812825] puppetdb@puppetdb STATEMENT:  insert into certnames (certname) values ($1)  on conflict (certname) do update set deactivated=null, expired=null  where (certnames.deactivated < $2 or certnames.expired < $3)

This seems like the two PuppetDB instances are competing to update the same row.

So our questions are:

• is it supported to run multiple PuppetDB instances against the same PostgreSQL database?
• if yes, how can we avoid these database errors?

Expected Behavior

We would expect to be run two different PuppetDB instances without database errors.

Steps to Reproduce

Steps to reproduce the behavior:

1. Setup two PuppetDB instances
2. Point them to the same PostgreSQL database
3. Let puppet-agents run

Environment

• PuppetDB version 8.3.0-1jammy
• Puppetserver version 8.4.0-1jammy
• PostgreSQL version 14.10-1.pgdg22.04+1
• Platform Ubuntu 22.04.3

Additional Context

I will gladly provide more information or context on this topic if requested.

Avoid the puppet clj-http-client in benchmark to improve performance

Just use the JDK client. Some previous investigations have suggested that the puppet client has performance issues (at least the sync client), and recent pdb benchmarking has implied something similar -- it could only emulate about 80k hosts with a 30s run interval, no matter how many senders or similation threads it had (on a large 32-core server).

Since benchmark only needs to make synchronous POSTs from existing threads (the senders), the JDK's built in client should work fine, and preliminary tests have been promising.

We'll need to continue to support ssl-certs via pem files, but we can use puppet's ssl-utils for that.

Consider unifying query parameters/behaviors (monitoring/timeouts/etc.)

Right now the root query endpoint (and others) support timeouts (cf. wrap-typical-query) and their queries are monitored for disconnections, but that's not true for some of the other endpoints. Consider extending support to the other endpoints.

Negated regex on dotted path not the negation of a regex match

Consider 3 nodes with a single resource, their resource params for that resource might look like

# node 1
{"simple":"param"
 "nested" {"foo": "bar"}}

# node 2
{"simple":"foo"
 "nested" {"foo": "bar"}}

# node 3
{"nested" {"foo": "bar"}}

A query using regex on the parameter value would look like

# PQL
resources[] { parameters.simple ~ "par.m"}

# AST
["from", "resources", ["~" ["parameter", "simple"] "par.m"]]

and it would return the resource for node 1 which is the only matching resource parameter.

Confusingly, the negation of this query does not return node 2 & node 3. It will only return node 2

# PQL
resources[] { parameters.simple !~ "par.m"}

# AST
["from", "resources", ["not", ["~" ["parameter", "simple"] "par.m"]]]

This is because a regex match on a jsonb value expands into 2 clauses in a HoneySQL raw element (a string of SQL), a check that the regex matches and a check for key existence. When wrapped in a HoneySQL :not it does not wrap the internal clause in parentheses, and the check is that the regex does not match, but the key does exist.

NOT (parameters ->> 'simple') ~ ('"par.m"'::jsonb#>>'{}') AND parameters ? 'simple'

If the internal clauses were honeysql rather than raw SQL it would quote them, and the behavior does not line up to the behavior of = and != on jsonb fields, which are proper boolean negations of each other.

Internal issue: https://perforce.atlassian.net/browse/PDB-5713

Consider avoiding ubuquitous types in benchmark catalog queries

At the moment, benchmark generates two different flavors of an exported resources query when --catalog-query-pct is positive. Both of these filter on the resource type, without any consideration of the frequency of that type. In some testing with larger system (simulating 100k nodes), the Service type was frequently chosen for the query, a type that was extremely common, matching 11m rows in catalog_resources, causing an expensive full table scan, and a very large result.

Consider adjusting benchmark somehow, to prefer more selective exported resource queries.

Error: java.lang.String cannot be cast to class clojure.lang.Associative

I have been trying to query the PuppetDB with a Golang binary (Sensu monitoring) but the PuppetDB is not very happy about it and gives a error 500 back:

Puppetdb version: 7.15.0
Puppet version 7.27.0
Ubuntu 20.04 kernel 5.4.0-169

Sources of the Golang binary:
https://github.com/elfranne/http-checks/tree/master/cmd/http-json
https://github.com/elfranne/http-checks/releases

Command used:

http-json --url https://puppet.example.com:8081/pdb/query/v4/reports --mtls-cert-file /etc/puppetlabs/puppet/ssl/certs/puppet01.example.com.pem --mtls-key-file /etc/puppetlabs/puppet/ssl/private_keys/puppet01.example.com.pem --trusted-ca-file  /etc/puppetlabs/puppet/ssl/certs/ca.pem --method POST --post-data '{"query":["and",["=","latest_report?","true"],["=","certname","puppet01.example.com"]]}' --query '.[].status == "success" or .[].status == "unchanged" or .[].status == "changed"' --expression 'true' --header 'Content-Type:application/json' --debug

output:

DEBUG-REQUEST:
POST /pdb/query/v4/reports HTTP/1.1
Host: puppet.example.com:8081
Content-Type: application/json

"{\"query\":[\"and\",[\"=\",\"latest_report?\",\"true\"],[\"=\",\"certname\",\"puppet01.example.com\"]]}"
DEBUG-RESPONSE:
HTTP/1.1 500 Server Error
Content-Length: 198
Content-Type: text/plain;charset=utf-8
Date: Mon, 15 Jan 2024 14:30:37 GMT

class java.lang.String cannot be cast to class clojure.lang.Associative (java.lang.String is in module java.base of loader 'bootstrap'; clojure.lang.Associative is in unnamed module of loader 'app')Could not unmarshal response body into JSON: invalid character 'c' looking for beginning of value

PuppetDB log:

ERROR [p.p.middleware] #error {
 :cause class java.lang.String cannot be cast to class clojure.lang.Associative (java.lang.String is in module java.base of loader 'bootstrap'; clojure.lang.Associative is in unnamed module of loader 'app')
 :via
 [{:type java.lang.ClassCastException
   :message class java.lang.String cannot be cast to class clojure.lang.Associative (java.lang.String is in module java.base of loader 'bootstrap'; clojure.lang.Associative is in unnamed module of loader 'app')
   :at [clojure.lang.RT assoc RT.java 827]}]
 :trace
 [[clojure.lang.RT assoc RT.java 827]
  [clojure.core$assoc__5481 invokeStatic core.clj 193]
  [clojure.core$update invokeStatic core.clj 6231]
  [clojure.core$update invoke core.clj 6223]
  [puppetlabs.puppetdb.http.query$post_req__GT_query invokeStatic query.clj 353]
  [puppetlabs.puppetdb.http.query$post_req__GT_query invoke query.clj 344]
  [puppetlabs.puppetdb.http.query$fn__30571$create_query_map__30576$fn__30577 invoke query.clj 366]
  [puppetlabs.puppetdb.http.query$fn__30571$create_query_map__30576 invoke query.clj 358]
  [puppetlabs.puppetdb.http.query$extract_query$fn__30591 invoke query.clj 382]
  [bidi.ring$fn__35415 invokeStatic ring.cljc 25]
  [bidi.ring$fn__35415 invoke ring.cljc 21]
  [bidi.ring$fn__35400$G__35395__35409 invoke ring.cljc 16]
  [puppetlabs.puppetdb.middleware$fn__36708$make_pdb_handler__36717$fn__36720$fn__36722 invoke middleware.clj 419]
  [puppetlabs.puppetdb.middleware$wrap_with_illegal_argument_catch$fn__36551 invoke middleware.clj 99]
  [puppetlabs.puppetdb.middleware$verify_accepts_content_type$fn__36562 invoke middleware.clj 139]
  [puppetlabs.puppetdb.middleware$verify_content_type$fn__36574 invoke middleware.clj 169]
  [puppetlabs.puppetdb.middleware$verify_sync_version$fn__36658 invoke middleware.clj 354]
  [puppetlabs.puppetdb.middleware$wrap_with_metrics$fn__36619$fn__36628 invoke middleware.clj 272]
  [puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__26316$fn__26317$fn__26318 invoke metrics.clj 23]
  [puppetlabs.puppetdb.utils.metrics.proxy$java.lang.Object$Callable$7da976d4 call nil -1]
  [com.codahale.metrics.Timer time Timer.java 101]
  [puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__26316$fn__26317 invoke metrics.clj 23]
  [puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__26316$fn__26317$fn__26318 invoke metrics.clj 23]
  [puppetlabs.puppetdb.utils.metrics.proxy$java.lang.Object$Callable$7da976d4 call nil -1]
  [com.codahale.metrics.Timer time Timer.java 101]
  [puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__26316$fn__26317 invoke metrics.clj 23]
  [puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__26316$fn__26317$fn__26318 invoke metrics.clj 23]
  [puppetlabs.puppetdb.utils.metrics.proxy$java.lang.Object$Callable$7da976d4 call nil -1]
  [com.codahale.metrics.Timer time Timer.java 101]
  [puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__26316$fn__26317 invoke metrics.clj 23]
  [puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__26316$fn__26317$fn__26318 invoke metrics.clj 23]
  [puppetlabs.puppetdb.utils.metrics.proxy$java.lang.Object$Callable$7da976d4 call nil -1]
  [com.codahale.metrics.Timer time Timer.java 101]
  [puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__26316$fn__26317 invoke metrics.clj 23]
  [puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_ invokeStatic metrics.clj 26]
  [puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_ invoke metrics.clj 15]
  [puppetlabs.puppetdb.middleware$wrap_with_metrics$fn__36619 invoke middleware.clj 271]
  [puppetlabs.puppetdb.middleware$wrap_with_globals$fn__36546 invoke middleware.clj 93]
  [puppetlabs.puppetdb.middleware$wrap_with_exception_handling$fn__36555 invoke middleware.clj 114]
  [puppetlabs.puppetdb.http.server$build_app$fn__43107 invoke server.clj 80]
  [compojure.core$routing$fn__36019 invoke core.clj 151]
  [clojure.core$some invokeStatic core.clj 2718]
  [clojure.core$some invoke core.clj 2709]
  [compojure.core$routing invokeStatic core.clj 151]
  [compojure.core$routing doInvoke core.clj 148]
  [clojure.lang.RestFn invoke RestFn.java 423]
  [puppetlabs.puppetdb.pdb_routing$wrap_with_context$fn__46956 invoke pdb_routing.clj 32]
  [compojure.core$if_context$fn__36043 invoke core.clj 218]
  [compojure.core$routing$fn__36019 invoke core.clj 151]
  [clojure.core$some invokeStatic core.clj 2718]
  [clojure.core$some invoke core.clj 2709]
  [compojure.core$routing invokeStatic core.clj 151]
  [compojure.core$routing doInvoke core.clj 148]
  [clojure.lang.RestFn applyTo RestFn.java 139]
  [clojure.core$apply invokeStatic core.clj 669]
  [clojure.core$apply invoke core.clj 662]
  [compojure.core$routes$fn__36023 invoke core.clj 156]
  [compojure.core$routing$fn__36019 invoke core.clj 151]
  [clojure.core$some invokeStatic core.clj 2718]
  [clojure.core$some invoke core.clj 2709]
  [compojure.core$routing invokeStatic core.clj 151]
  [compojure.core$routing doInvoke core.clj 148]
  [clojure.lang.RestFn invoke RestFn.java 460]
  [puppetlabs.puppetdb.pdb_routing$pdb_app$fn__46967 invoke pdb_routing.clj 60]
  [compojure.core$if_context$fn__36043 invoke core.clj 218]
  [ring.middleware.params$wrap_params$fn__36514 invoke params.clj 67]
  [puppetlabs.puppetdb.middleware$wrap_with_certificate_cn$fn__36536 invoke middleware.clj 75]
  [puppetlabs.puppetdb.middleware$wrap_with_default_body$fn__36541 invoke middleware.clj 82]
  [puppetlabs.puppetdb.middleware$wrap_with_debug_logging$fn__36519 invoke middleware.clj 39]
  [puppetlabs.i18n.core$locale_negotiator$fn__272 invoke core.clj 361]
  [puppetlabs.trapperkeeper.services.webserver.jetty9_core$ring_handler$fn__48624 invoke jetty9_core.clj 460]
  [puppetlabs.trapperkeeper.services.webserver.jetty9_core.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a handle nil -1]
  [org.eclipse.jetty.server.handler.HandlerWrapper handle HandlerWrapper.java 127]
  [org.eclipse.jetty.server.handler.ScopedHandler nextHandle ScopedHandler.java 235]
  [org.eclipse.jetty.server.handler.ContextHandler doHandle ContextHandler.java 1440]
  [org.eclipse.jetty.server.handler.ScopedHandler nextScope ScopedHandler.java 190]
  [org.eclipse.jetty.server.handler.ContextHandler doScope ContextHandler.java 1355]
  [org.eclipse.jetty.server.handler.ScopedHandler handle ScopedHandler.java 141]
  [org.eclipse.jetty.server.handler.ContextHandlerCollection handle ContextHandlerCollection.java 234]
  [org.eclipse.jetty.server.handler.HandlerCollection handle HandlerCollection.java 146]
  [org.eclipse.jetty.server.handler.gzip.GzipHandler handle GzipHandler.java 772]
  [org.eclipse.jetty.server.handler.RequestLogHandler handle RequestLogHandler.java 54]
  [com.puppetlabs.trapperkeeper.services.webserver.jetty9.utils.MDCRequestLogHandler handle MDCRequestLogHandler.java 36]
  [org.eclipse.jetty.server.handler.StatisticsHandler handle StatisticsHandler.java 181]
  [org.eclipse.jetty.server.handler.HandlerWrapper handle HandlerWrapper.java 127]
  [org.eclipse.jetty.server.Server handle Server.java 516]
  [org.eclipse.jetty.server.HttpChannel lambda$handle$1 HttpChannel.java 487]
  [org.eclipse.jetty.server.HttpChannel dispatch HttpChannel.java 732]
  [org.eclipse.jetty.server.HttpChannel handle HttpChannel.java 479]
  [org.eclipse.jetty.server.HttpConnection onFillable HttpConnection.java 277]
  [org.eclipse.jetty.io.AbstractConnection$ReadCallback succeeded AbstractConnection.java 311]
  [org.eclipse.jetty.io.FillInterest fillable FillInterest.java 105]
  [org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint onFillable SslConnection.java 555]
  [org.eclipse.jetty.io.ssl.SslConnection onFillable SslConnection.java 410]
  [org.eclipse.jetty.io.ssl.SslConnection$2 succeeded SslConnection.java 164]
  [org.eclipse.jetty.io.FillInterest fillable FillInterest.java 105]
  [org.eclipse.jetty.io.ChannelEndPoint$1 run ChannelEndPoint.java 104]
  [org.eclipse.jetty.util.thread.strategy.EatWhatYouKill runTask EatWhatYouKill.java 338]
  [org.eclipse.jetty.util.thread.strategy.EatWhatYouKill doProduce EatWhatYouKill.java 315]
  [org.eclipse.jetty.util.thread.strategy.EatWhatYouKill tryProduce EatWhatYouKill.java 173]
  [org.eclipse.jetty.util.thread.strategy.EatWhatYouKill run EatWhatYouKill.java 131]
  [org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread run ReservedThreadExecutor.java 409]
  [org.eclipse.jetty.util.thread.QueuedThreadPool runJob QueuedThreadPool.java 883]
  [org.eclipse.jetty.util.thread.QueuedThreadPool$Runner run QueuedThreadPool.java 1034]
  [java.lang.Thread run Thread.java 829]]}

Doing a curl request does not trigger this.

Missing Puppet packages for Debian 12 (Bookworm)

Hello!

Debian 12 AIO packages for Puppet-Agent are now available (see puppetlabs/puppet#9149).

We now need PuppetServer (puppetlabs/puppetserver#2837) and PuppetDB (this issue) packages for this operating system 😁.

puppetdb 8.5.0 sql exception missing from clause for table certnames

Describe the Bug

Querying puppetdb 8.5.0 with following

[
"from", "node",
['extract', ['latest_report_status'],
['and',
['=', 'certname', 'SOMECERTNAME'],
]
]
]

Results now in a exception in puppetdb:

:cause ERROR: missing FROM-clause entry for table "certnames"
Position: 425
:via
[{:type org.postgresql.util.PSQLException
:message ERROR: missing FROM-clause entry for table "certnames"
Position: 425
:at [org.postgresql.core.v3.QueryExecutorImpl receiveErrorResponse QueryExecutorImpl.java 2725]}]
:trace
[[org.postgresql.core.v3.QueryExecutorImpl receiveErrorResponse QueryExecutorImpl.java 2725]
[org.postgresql.core.v3.QueryExecutorImpl processResults QueryExecutorImpl.java 2412]
[org.postgresql.core.v3.QueryExecutorImpl execute QueryExecutorImpl.java 371]
[org.postgresql.jdbc.PgStatement executeInternal PgStatement.java 502]
[org.postgresql.jdbc.PgStatement execute PgStatement.java 419]
[org.postgresql.jdbc.PgPreparedStatement executeWithFlags PgPreparedStatement.java 194]
[org.postgresql.jdbc.PgPreparedStatement executeQuery PgPreparedStatement.java 137]
[com.zaxxer.hikari.pool.ProxyPreparedStatement executeQuery ProxyPreparedStatement.java 52]
[com.zaxxer.hikari.pool.HikariProxyPreparedStatement executeQuery HikariProxyPreparedStatement.java -1]
[puppetlabs.puppetdb.jdbc$call_with_array_converted_query_rows$fn__21985 invoke jdbc.clj 330]
[clojure.java.jdbc$db_transaction_STAR_ invokeStatic jdbc.clj 860]
[clojure.java.jdbc$db_transaction_STAR_ invoke jdbc.clj 776]
[puppetlabs.puppetdb.jdbc$call_with_array_converted_query_rows invokeStatic jdbc.clj 322]
[puppetlabs.puppetdb.jdbc$call_with_array_converted_query_rows invoke jdbc.clj 304]
[puppetlabs.puppetdb.query_eng$body_stream$serialize_query_response__29427$fn__29477$fn__29494$fn__29495 invoke query_eng.clj 445]
[clojure.java.jdbc$db_transaction_STAR_ invokeStatic jdbc.clj 807]
[clojure.java.jdbc$db_transaction_STAR_ invoke jdbc.clj 776]
[puppetlabs.puppetdb.query_eng$body_stream$serialize_query_response__29427$fn__29477$fn__29494 invoke query_eng.clj 434]
[puppetlabs.puppetdb.query_eng$body_stream$serialize_query_response__29427$fn__29477 invoke query_eng.clj 433]
[puppetlabs.puppetdb.query_eng$body_stream$serialize_query_response__29427 invoke query_eng.clj 427]
[puppetlabs.puppetdb.query_eng$generated_stream$fn__29384 invoke query_eng.clj 334]
[ring.util.io$piped_input_stream$fn__20730 invoke io.clj 28]
[clojure.core$binding_conveyor_fn$fn__5823 invoke core.clj 2047]
[clojure.lang.AFn call AFn.java 18]
[java.util.concurrent.FutureTask run FutureTask.java 264]
[java.util.concurrent.ThreadPoolExecutor runWorker ThreadPoolExecutor.java 1136]
[java.util.concurrent.ThreadPoolExecutor$Worker run ThreadPoolExecutor.java 635]
[java.lang.Thread run Thread.java 840]]}

Expected Behavior

Returns last report status of certname

Environment

• Latest puppetdb 8.5.0
• Ubuntu 22.04
• puppet-agent 8.6.0

java.nio.channels.CancelledKeyException

I am attempting to troubleshoot the following problem and was hoping someone could point me in the right direction. Is this a problem in puppet dashboard or puppetdb?

Describe the Bug

Puppetdb is failing with the error below in a fresh installation of a 3-node setup of puppet master, db and db01 in a staging environment:

• puppetdb version: 8.1.0
• psql (PostgreSQL) 14.9 (Ubuntu 14.9-1.pgdg22.04+1)
• puppetserver version: 8.2.0
• Ubuntu 22

The same configuration is operating normally in my production environment:

• puppetdb version: 7.13.1
• psql (PostgreSQL) 11.21 (Ubuntu 11.21-1.pgdg20.04+1)
• puppetserver version: 7.12.0
• Ubuntu 20

This sometimes appears on puppet dashboard the following error. Other times, the 3 nodes are shown in "unreported" state (see below).

Internal Server Error
This error usually occurs because:
    We were unable to reach PuppetDB;
    The query to be executed was malformed resulting in an incorrectly encoded request.

Puppet server appears to work properly with no errors, so I am not sure if this defect is in puppet dashboard or puppetdb.

Steps to Reproduce

1. New 3-node installation of server, db01, puppetdb, puppetdashboard (versions above)
2. Accessing puppetdb web page fails

Additional Context

Unreported states

Stack Trace

2023-08-25T13:12:47.626-04:00 ERROR [p.p.middleware] #error {
 :cause nil
 :via
 [{:type java.nio.channels.CancelledKeyException
   :message nil
   :at [sun.nio.ch.SelectionKeyImpl ensureValid SelectionKeyImpl.java 75]}]
 :trace
 [[sun.nio.ch.SelectionKeyImpl ensureValid SelectionKeyImpl.java 75]
  [sun.nio.ch.SelectionKeyImpl interestOps SelectionKeyImpl.java 104]
  [java.nio.channels.spi.AbstractSelectableChannel register AbstractSelectableChannel.java 233]
  [java.nio.channels.SelectableChannel register SelectableChannel.java 260]
  [jdk.internal.reflect.GeneratedMethodAccessor19 invoke nil -1]
  [jdk.internal.reflect.DelegatingMethodAccessorImpl invoke DelegatingMethodAccessorImpl.java 43]
  [java.lang.reflect.Method invoke Method.java 568]
  [clojure.lang.Reflector invokeMatchingMethod Reflector.java 167]
  [clojure.lang.Reflector invokeInstanceMethod Reflector.java 102]
  [puppetlabs.puppetdb.query.monitor$stop_query_at_deadline_or_disconnect invokeStatic monitor.clj 328]
  [puppetlabs.puppetdb.query.monitor$stop_query_at_deadline_or_disconnect invoke monitor.clj 324]
  [puppetlabs.puppetdb.http.query$wrap_typical_query$fn__31076$fn__31078 invoke query.clj 434]
  [puppetlabs.puppetdb.http.query$wrap_typical_query$fn__31076 invoke query.clj 404]
  [bidi.ring$fn__35985 invokeStatic ring.cljc 25]
  [bidi.ring$fn__35985 invoke ring.cljc 21]
  [bidi.ring$fn__35970$G__35965__35979 invoke ring.cljc 16]
  [puppetlabs.puppetdb.middleware$fn__37279$make_pdb_handler__37288$fn__37291$fn__37293 invoke middleware.clj 429]
  [puppetlabs.puppetdb.middleware$wrap_with_illegal_argument_catch$fn__37121 invoke middleware.clj 100]
  [puppetlabs.puppetdb.middleware$verify_accepts_content_type$fn__37133 invoke middleware.clj 148]
  [puppetlabs.puppetdb.middleware$verify_content_type$fn__37145 invoke middleware.clj 181]
  [puppetlabs.puppetdb.middleware$verify_sync_version$fn__37229 invoke middleware.clj 363]
  [puppetlabs.puppetdb.middleware$wrap_with_metrics$fn__37190$fn__37199 invoke middleware.clj 281]
  [puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__26528$fn__26529$fn__26530 invoke metrics.clj 23]
  [puppetlabs.puppetdb.utils.metrics.proxy$java.lang.Object$Callable$7da976d4 call nil -1]
  [com.codahale.metrics.Timer time Timer.java 101]
  [puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__26528$fn__26529 invoke metrics.clj 23]
  [puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__26528$fn__26529$fn__26530 invoke metrics.clj 23]
  [puppetlabs.puppetdb.utils.metrics.proxy$java.lang.Object$Callable$7da976d4 call nil -1]
  [com.codahale.metrics.Timer time Timer.java 101]
  [puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__26528$fn__26529 invoke metrics.clj 23]
  [puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__26528$fn__26529$fn__26530 invoke metrics.clj 23]
  [puppetlabs.puppetdb.utils.metrics.proxy$java.lang.Object$Callable$7da976d4 call nil -1]
  [com.codahale.metrics.Timer time Timer.java 101]
  [puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__26528$fn__26529 invoke metrics.clj 23]
  [puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__26528$fn__26529$fn__26530 invoke metrics.clj 23]
  [puppetlabs.puppetdb.utils.metrics.proxy$java.lang.Object$Callable$7da976d4 call nil -1]
  [com.codahale.metrics.Timer time Timer.java 101]
  [puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__26528$fn__26529 invoke metrics.clj 23]
  [puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_ invokeStatic metrics.clj 26]
  [puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_ invoke metrics.clj 15]
  [puppetlabs.puppetdb.middleware$wrap_with_metrics$fn__37190 invoke middleware.clj 280]
  [puppetlabs.puppetdb.middleware$wrap_with_globals$fn__37116 invoke middleware.clj 94]
  [puppetlabs.puppetdb.middleware$wrap_with_exception_handling$fn__37125 invoke middleware.clj 115]
  [puppetlabs.puppetdb.http.server$build_app$fn__42708 invoke server.clj 80]
  [compojure.core$routing$fn__36589 invoke core.clj 151]
  [clojure.core$some invokeStatic core.clj 2718]
  [clojure.core$some invoke core.clj 2709]
  [compojure.core$routing invokeStatic core.clj 151]
  [compojure.core$routing doInvoke core.clj 148]
  [clojure.lang.RestFn invoke RestFn.java 423]
  [puppetlabs.puppetdb.pdb_routing$wrap_with_context$fn__46557 invoke pdb_routing.clj 32]
  [compojure.core$if_context$fn__36613 invoke core.clj 218]
  [compojure.core$routing$fn__36589 invoke core.clj 151]
  [clojure.core$some invokeStatic core.clj 2718]
  [clojure.core$some invoke core.clj 2709]
  [compojure.core$routing invokeStatic core.clj 151]
  [compojure.core$routing doInvoke core.clj 148]
  [clojure.lang.RestFn applyTo RestFn.java 139]
  [clojure.core$apply invokeStatic core.clj 669]
  [clojure.core$apply invoke core.clj 662]
  [compojure.core$routes$fn__36593 invoke core.clj 156]
  [compojure.core$routing$fn__36589 invoke core.clj 151]
  [clojure.core$some invokeStatic core.clj 2718]
  [clojure.core$some invoke core.clj 2709]
  [compojure.core$routing invokeStatic core.clj 151]
  [compojure.core$routing doInvoke core.clj 148]
  [clojure.lang.RestFn invoke RestFn.java 460]
  [puppetlabs.puppetdb.pdb_routing$pdb_app$fn__46568 invoke pdb_routing.clj 60]
  [compojure.core$if_context$fn__36613 invoke core.clj 218]
  [ring.middleware.params$wrap_params$fn__37084 invoke params.clj 67]
  [puppetlabs.puppetdb.middleware$wrap_with_certificate_cn$fn__37106 invoke middleware.clj 76]
  [puppetlabs.puppetdb.middleware$wrap_with_default_body$fn__37111 invoke middleware.clj 83]
  [puppetlabs.puppetdb.middleware$wrap_with_debug_logging$fn__37089 invoke middleware.clj 40]
  [puppetlabs.i18n.core$locale_negotiator$fn__272 invoke core.clj 361]
  [puppetlabs.trapperkeeper.services.webserver.jetty9_core$ring_handler$fn__48220 invoke jetty9_core.clj 460]
  [puppetlabs.trapperkeeper.services.webserver.jetty9_core.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a handle nil -1]
  [org.eclipse.jetty.server.handler.HandlerWrapper handle HandlerWrapper.java 127]
  [org.eclipse.jetty.server.handler.ScopedHandler nextHandle ScopedHandler.java 235]
  [org.eclipse.jetty.server.handler.ContextHandler doHandle ContextHandler.java 1440]
  [org.eclipse.jetty.server.handler.ScopedHandler nextScope ScopedHandler.java 190]
  [org.eclipse.jetty.server.handler.ContextHandler doScope ContextHandler.java 1355]
  [org.eclipse.jetty.server.handler.ScopedHandler handle ScopedHandler.java 141]
  [org.eclipse.jetty.server.handler.ContextHandlerCollection handle ContextHandlerCollection.java 234]
  [org.eclipse.jetty.server.handler.HandlerCollection handle HandlerCollection.java 146]
  [org.eclipse.jetty.server.handler.gzip.GzipHandler handle GzipHandler.java 772]
  [org.eclipse.jetty.server.handler.RequestLogHandler handle RequestLogHandler.java 54]
  [com.puppetlabs.trapperkeeper.services.webserver.jetty9.utils.MDCRequestLogHandler handle MDCRequestLogHandler.java 36]
  [org.eclipse.jetty.server.handler.StatisticsHandler handle StatisticsHandler.java 181]
  [org.eclipse.jetty.server.handler.HandlerWrapper handle HandlerWrapper.java 127]
  [org.eclipse.jetty.server.Server handle Server.java 516]
  [org.eclipse.jetty.server.HttpChannel lambda$handle$1 HttpChannel.java 487]
  [org.eclipse.jetty.server.HttpChannel dispatch HttpChannel.java 732]
  [org.eclipse.jetty.server.HttpChannel handle HttpChannel.java 479]
  [org.eclipse.jetty.server.HttpConnection onFillable HttpConnection.java 277]
  [org.eclipse.jetty.io.AbstractConnection$ReadCallback succeeded AbstractConnection.java 311]
  [org.eclipse.jetty.io.FillInterest fillable FillInterest.java 105]
  [org.eclipse.jetty.io.ChannelEndPoint$1 run ChannelEndPoint.java 104]
  [org.eclipse.jetty.util.thread.strategy.EatWhatYouKill runTask EatWhatYouKill.java 338]
  [org.eclipse.jetty.util.thread.strategy.EatWhatYouKill doProduce EatWhatYouKill.java 315]
  [org.eclipse.jetty.util.thread.strategy.EatWhatYouKill tryProduce EatWhatYouKill.java 173]
  [org.eclipse.jetty.util.thread.strategy.EatWhatYouKill run EatWhatYouKill.java 131]
  [org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread run ReservedThreadExecutor.java 409]
  [org.eclipse.jetty.util.thread.QueuedThreadPool runJob QueuedThreadPool.java 883]
  [org.eclipse.jetty.util.thread.QueuedThreadPool$Runner run QueuedThreadPool.java 1034]
  [java.lang.Thread run Thread.java 833]]}

Add documentation for using Postgresql

Fail better, or have nginx fail better

At the moment, when there's an issue with grayskull, nginx just hands back an ugly HTML page to Puppet. Which then causes the puppet client run to bork with

'''bash
Puppet::Parser::Compiler failed with error PSON::ParserError: source '

if grayskull returned something better in event of failure, then nginx could proxy that, or we could make the nginx side of it fail better.

PQL parsing of `or` clauses can exhaust memory

It looks like the current PQL parser, using instaparse, consumes exorbitant quantities of RAM when attempting to parse or clauses. In a recent test instance with 1G allocated to puppetdb's heap, this query was enough to exhaust the heap in ~3m:

reports {
  latest_report? = true and (
    certname = "foo22" or
    certname = "foo21" or
    certname = "foo20" or
    certname = "foo19" or
    certname = "foo18" or
    certname = "foo17" or
    certname = "foo16" or
    certname = "foo15" or
    certname = "foo14" or
    certname = "foo13" or
    certname = "foo12" or
    certname = "foo11" or
    certname = "foo10" or
    certname = "foo9" or
    certname = "foo8" or
    certname = "foo7" or
    certname = "foo6" or
    certname = "foo5" or
    certname = "foo4" or
    certname = "foo3" or
    certname = "foo2" or
    certname = "foo1"
  )
}

Calling it with 21 OR clauses, the query succeeded in 31s, with 20 in 7s, with 19 in 4s, and so on back down to <1 once you reach 16 clauses.

As a temporary measure, use in instead of a set of ors; or wrap the ors in parentheses, although in general an in will be faster on the postgres side once the query's been parsed and postgres is actually running it.

Add ability to easily update the hostname of the grayskull box

Puppetdb restarts because of java.lang.NoSuchMethodError

Describe the Bug

After the update from version 7.16.0 to version 7.17.1, Puppetdb restarts every minute.
The error we see in the logs is the following:

2024-02-28T08:00:29.413+01:00 ERROR [p.p.utils] Requesting shutdown: java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
2024-02-28T08:00:29.420+01:00 INFO  [p.t.internal] Beginning shutdown sequence
2024-02-28T08:00:29.814+01:00 ERROR [p.p.utils] Requesting shutdown: java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
2024-02-28T08:00:30.153+01:00 ERROR [p.p.utils] Requesting shutdown: java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
2024-02-28T08:00:30.478+01:00 ERROR [p.p.utils] Requesting shutdown: java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
2024-02-28T08:00:30.800+01:00 ERROR [p.p.utils] Requesting shutdown: java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
2024-02-28T08:00:30.814+01:00 ERROR [p.p.threadpool] Reporting unexpected error from thread cmd-proc-thread-2 to stderr and log
java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
	at org.postgresql.jdbc.ArrayDecoding.readBinaryArray(ArrayDecoding.java:523)
	at org.postgresql.jdbc.PgArray.readBinaryArray(PgArray.java:175)
	at org.postgresql.jdbc.PgArray.getArrayImpl(PgArray.java:150)
	at org.postgresql.jdbc.PgArray.getArray(PgArray.java:111)
	at puppetlabs.puppetdb.jdbc$convert_any_sql_array.invokeStatic(jdbc.clj:262)
	at puppetlabs.puppetdb.jdbc$convert_any_sql_array.invoke(jdbc.clj:259)
	at puppetlabs.puppetdb.scf.storage$fn__26869$catalog_resources__26874$fn__26875$fn__26876$iter__26879__26883$fn__26884$fn__26889.invoke(storage.clj:560)
	at puppetlabs.kitchensink.core$mapvals$iter__4426__4430$fn__4431.invoke(core.clj:362)
	at clojure.lang.LazySeq.sval(LazySeq.java:42)
	at clojure.lang.LazySeq.seq(LazySeq.java:51)
	at clojure.lang.Cons.next(Cons.java:39)
	at clojure.lang.RT.next(RT.java:713)
	at clojure.core$next__5451.invokeStatic(core.clj:64)
	at clojure.core.protocols$fn__8249.invokeStatic(protocols.clj:169)
	at clojure.core.protocols$fn__8249.invoke(protocols.clj:124)
	at clojure.core.protocols$fn__8204$G__8199__8213.invoke(protocols.clj:19)
	at clojure.core.protocols$seq_reduce.invokeStatic(protocols.clj:31)
	at clojure.core.protocols$fn__8236.invokeStatic(protocols.clj:75)
	at clojure.core.protocols$fn__8236.invoke(protocols.clj:75)
	at clojure.core.protocols$fn__8178$G__8173__8191.invoke(protocols.clj:13)
	at clojure.core$reduce.invokeStatic(core.clj:6886)
	at clojure.core$into.invokeStatic(core.clj:6958)
	at clojure.core$into.invoke(core.clj:6950)
	at puppetlabs.kitchensink.core$mapvals.invokeStatic(core.clj:362)
	at puppetlabs.kitchensink.core$mapvals.invoke(core.clj:356)
	at puppetlabs.puppetdb.scf.storage$fn__26869$catalog_resources__26874$fn__26875$fn__26876$iter__26879__26883$fn__26884.invoke(storage.clj:560)
	at clojure.lang.LazySeq.sval(LazySeq.java:42)
	at clojure.lang.LazySeq.seq(LazySeq.java:51)
	at clojure.lang.RT.seq(RT.java:535)
	at clojure.core$seq__5467.invokeStatic(core.clj:139)
	at clojure.core$zipmap.invokeStatic(core.clj:6619)
	at clojure.core$zipmap.invoke(core.clj:6619)
	at puppetlabs.puppetdb.scf.storage$fn__26869$catalog_resources__26874$fn__26875$fn__26876.invoke(storage.clj:558)
	at clojure.java.jdbc$execute_query_with_params.invokeStatic(jdbc.clj:1091)
	at clojure.java.jdbc$execute_query_with_params.invoke(jdbc.clj:1084)
	at clojure.java.jdbc$db_query_with_resultset_STAR_.invokeStatic(jdbc.clj:1106)
	at clojure.java.jdbc$db_query_with_resultset_STAR_.invoke(jdbc.clj:1093)
	at clojure.java.jdbc$db_query_with_resultset.invokeStatic(jdbc.clj:1140)
	at clojure.java.jdbc$db_query_with_resultset.invoke(jdbc.clj:1119)
	at clojure.java.jdbc$db_query_with_resultset.invokeStatic(jdbc.clj:1127)
	at clojure.java.jdbc$db_query_with_resultset.invoke(jdbc.clj:1119)
	at puppetlabs.puppetdb.jdbc$query_with_resultset.invokeStatic(jdbc.clj:236)
	at puppetlabs.puppetdb.jdbc$query_with_resultset.invoke(jdbc.clj:227)
	at puppetlabs.puppetdb.scf.storage$fn__26869$catalog_resources__26874$fn__26875.invoke(storage.clj:550)
	at puppetlabs.puppetdb.scf.storage$fn__26869$catalog_resources__26874.invoke(storage.clj:547)
	at puppetlabs.puppetdb.scf.storage$fn__27221$add_resources_BANG___27226$fn__27227.invoke(storage.clj:771)
	at puppetlabs.puppetdb.scf.storage$fn__27221$add_resources_BANG___27226.invoke(storage.clj:765)
	at puppetlabs.puppetdb.scf.storage$fn__27430$update_catalog_associations_BANG___27435$fn__27439$fn__27441.invoke(storage.clj:887)
	at puppetlabs.puppetdb.scf.storage.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
	at com.codahale.metrics.Timer.time(Timer.java:101)
	at puppetlabs.puppetdb.scf.storage$fn__27430$update_catalog_associations_BANG___27435$fn__27439.invoke(storage.clj:886)
	at puppetlabs.puppetdb.scf.storage$fn__27430$update_catalog_associations_BANG___27435.invoke(storage.clj:881)
	at puppetlabs.puppetdb.scf.storage$fn__27468$replace_existing_catalog__27473$fn__27474$fn__27475.invoke(storage.clj:905)
	at puppetlabs.puppetdb.scf.storage.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
	at com.codahale.metrics.Timer.time(Timer.java:101)
	at puppetlabs.puppetdb.scf.storage$fn__27468$replace_existing_catalog__27473$fn__27474.invoke(storage.clj:903)
	at puppetlabs.puppetdb.scf.storage$fn__27468$replace_existing_catalog__27473.invoke(storage.clj:891)
	at puppetlabs.puppetdb.scf.storage$fn__27529$replace_catalog_BANG___27538$fn__27544$fn__27546$fn__27547.invoke(storage.clj:945)
	at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:860)
	at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:776)
	at puppetlabs.puppetdb.scf.storage$fn__27529$replace_catalog_BANG___27538$fn__27544$fn__27546.invoke(storage.clj:928)
	at puppetlabs.puppetdb.scf.storage.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
	at com.codahale.metrics.Timer.time(Timer.java:101)
	at puppetlabs.puppetdb.scf.storage$fn__27529$replace_catalog_BANG___27538$fn__27544.invoke(storage.clj:927)
	at puppetlabs.puppetdb.scf.storage$fn__27529$replace_catalog_BANG___27538.invoke(storage.clj:920)
	at puppetlabs.puppetdb.command$do_replace_catalog.invokeStatic(command.clj:345)
	at puppetlabs.puppetdb.command$do_replace_catalog.invoke(command.clj:343)
	at puppetlabs.puppetdb.command$exec_replace_catalog$fn__32624.invoke(command.clj:353)
	at puppetlabs.puppetdb.jdbc$retry_with_monitored_connection$fn__22102$fn__22115.invoke(jdbc.clj:553)
	at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:807)
	at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:776)
	at puppetlabs.puppetdb.jdbc$retry_with_monitored_connection$fn__22102.invoke(jdbc.clj:549)
	at puppetlabs.puppetdb.jdbc$retry_sql$attempt__22082.invoke(jdbc.clj:497)
	at puppetlabs.puppetdb.jdbc$retry_sql.invokeStatic(jdbc.clj:507)
	at puppetlabs.puppetdb.jdbc$retry_sql.invoke(jdbc.clj:488)
	at puppetlabs.puppetdb.jdbc$retry_with_monitored_connection.invokeStatic(jdbc.clj:546)
	at puppetlabs.puppetdb.jdbc$retry_with_monitored_connection.invoke(jdbc.clj:541)
	at puppetlabs.puppetdb.command$exec_replace_catalog.invokeStatic(command.clj:350)
	at puppetlabs.puppetdb.command$exec_replace_catalog.invoke(command.clj:347)
	at puppetlabs.puppetdb.command$exec_command.invokeStatic(command.clj:525)
	at puppetlabs.puppetdb.command$exec_command.invoke(command.clj:510)
	at puppetlabs.puppetdb.command$attempt_exec_command$fn__32841.invoke(command.clj:611)
	at puppetlabs.puppetdb.command$call_with_quick_retry$fn__32830.invoke(command.clj:591)
	at puppetlabs.puppetdb.command$call_with_quick_retry.invokeStatic(command.clj:589)
	at puppetlabs.puppetdb.command$call_with_quick_retry.invoke(command.clj:587)
	at puppetlabs.puppetdb.command$attempt_exec_command.invokeStatic(command.clj:609)
	at puppetlabs.puppetdb.command$attempt_exec_command.invoke(command.clj:605)
	at puppetlabs.puppetdb.command$process_cmd$fn__32897$fn__32898.invoke(command.clj:732)
	at puppetlabs.puppetdb.command$process_cmd$fn__32897.invoke(command.clj:729)
	at puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__26314$fn__26315$fn__26316.invoke(metrics.clj:23)
	at puppetlabs.puppetdb.utils.metrics.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
	at com.codahale.metrics.Timer.time(Timer.java:101)
	at puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__26314$fn__26315.invoke(metrics.clj:23)
	at puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__26314$fn__26315$fn__26316.invoke(metrics.clj:23)
	at puppetlabs.puppetdb.utils.metrics.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
	at com.codahale.metrics.Timer.time(Timer.java:101)
	at puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__26314$fn__26315.invoke(metrics.clj:23)
	at puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_.invokeStatic(metrics.clj:26)
	at puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_.invoke(metrics.clj:15)
	at puppetlabs.puppetdb.command$process_cmd.invokeStatic(command.clj:726)
	at puppetlabs.puppetdb.command$process_cmd.invoke(command.clj:712)
	at puppetlabs.puppetdb.command$process_message.invokeStatic(command.clj:819)
	at puppetlabs.puppetdb.command$process_message.invoke(command.clj:775)
	at puppetlabs.puppetdb.command$message_handler$fn__32931.invoke(command.clj:853)
	at puppetlabs.puppetdb.threadpool$dochan$fn__32512$fn__32513.invoke(threadpool.clj:115)
	at puppetlabs.puppetdb.threadpool$gated_execute$fn__32473.invoke(threadpool.clj:68)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

Expected Behavior

Puppetdb does not restart every minute and runs without a problem

Steps to Reproduce

Steps to reproduce the behavior:
Happens after a minute.

Environment

• Version 7.17.1
• Platform RHEL 8.9
• Java Version (openjdk): 1.8.0_402

Additional Context

Our first hint what could be the issue was this change in clj-parent: https://github.com/puppetlabs/clj-parent/blob/5.6.11/project.clj#L100

Our workaround was a downgrade to version 7.16.0.

Remove duplicate storage of catalgo resource parameters

Currently resource parameters are stored flattened in resource_params and as jsonb in resource_params_cache. Both of these tables are deduplicated and contain all the parameter information. Most queries are already backed by resource_params_cache. A few summary queries, and the completely undocumented select_params, which is only valid as a subquery, use the resource_params table still.

Internal issue: https://perforce.atlassian.net/browse/PE-37294

Please make the packaging/build pipelines public and provide reproducible builds

Use Case

As a service delivery partner we want to engage into the development. This isn't really possible without at least read-only access to the pipelines. We don't know which jobs exist and what they do. That makes it impossible to add support for new platforms. And as we can see on the existing requests Puppet itself is quite slow for adding support for new distributions/architectures.

Describe the Solution You Would Like

use GitHub actions for supported platforms to build packages (preferred) or at least grant partners access to the Jenkins (which was public in the past).

Describe Alternatives You've Considered

While I hate it that an open source tool uses private pipelines, you could also workaround this by providing new packages faster. However I think reproducible builds are required in the future and that also requires logs.

Additional Context

Adjust benchmark to account for the factset stable/volatile arrangement

The current factset storage rests on the assumption that a nontrivial subset of top-level facts (including any subtree) will never change (though see #3956 for a promising, potential relaxation/improvement there). Given that, benchmark should probably attempt to support that assumption when mutating factsets.

See also: #3955