Grafana queries cause workers to stuck about trino

[davseitsev]

The fix in #21356 significantly improves the situation but it's still reproduced. Accoring to latest comments, there are still some cases when we create json factory with LockFreePool, so the fix in version 444 doesn't cover all the cases and it could be the reason.
We've applied additional fix for #21356 and the workaround from #21512 and we can't reproduce the issue anymore.
We were testing it during 2 days on our production env and it looks stable.

[davseitsev]

Vey likely the issue is cased by jackson serializer, it's described here #21356.
After changing recycler pool implementation we can't reproduce it.

[Akanksha-kedia]

@davseitsev The proposed solution involves modifying the HTTPServer code to introduce a timeout if a thread is in idle state for more than x minutes. This modification is done in a new module with the HTTPServer class from a specific version of the simple client http server used by the jmx prometheus java agent.?

[Akanksha-kedia]

i would like to work on this.

[davseitsev]

It might be related to this issue #21512.

[findepi]

It might be related to this issue #21512.

the other issue mentions a workaround (experimental.thread-per-driver-scheduler-enabled=false)
@davseitsev Once that is set, do you still observe the problem with workers being killed/stuck/unresponsive when you run your Grafana dashboard?