Sidekiq processes are stuck (frozen) regularly (they stop to process jobs) when trying to connect to PostgreSQL about ruby-pg

[sigmaray]

I forgot to mention. If I understood correctly, Sidekiq processes freeze on this line

19:39:59 jobs-2.1        | 2024-08-26T16:39:59.228Z pid=14 tid=4ge WARN:
[...]/activerecord-7.0.3.1/lib/active_record/connection_adapters/postgresql_adapter.rb:862:in `set_client_encoding'

It seems that set_client_encoding method doesn't respect timeot specified in database.yml

My guess could be wrong. I don't have experience of reading sidekiq traces.

[larskanis]

There was a previous issue regarding to encoding through a connection proxy: #368

It is resolved since pg-1.5.4. But looking into the ActiveRecord sources, it looks like you're actively triggering this command by setting :encoding in your database config. Can you disable this config option?

[drdrsh]

I ran into a similar issue where timeouts/interrupts are sometimes not honored by ruby-pg. I was never able to reproduce the issue in a controlled environment but it was happening in production.

The fix was to set keepalives and tcp_user_timeout on the sockets.

postgres://username:password@host:port/database?connect_timeout=2&keepalives=1&keepalives_idle=1&keepalives_count=1&tcp_user_timeout=1000

I don't know if this is related in anyway but using these settings we were able to eliminate the frozen thread occurrences