Jackson LockFreePool recycler pool causes memory issues about trino

[wendigo]

SORTED_MAPPER is using ObjectMapperProvider which configures the JsonFactoryBuilder to use JsonRecyclerPools.threadLocalPool(). if this is ineffective I guess we need to report Jackson issue.

Update: it was fixed in master so not yet been released (part of Airlift 244 update: airlift/airlift@6d36ba3 and 00d867b)

Please test with 445 once it's released.

[davseitsev]

Looks good, I can't reproduce the issue. I will keep testing and update if anything changes.

[marvin-roesch]

That's great to hear, thanks! I wasn't checking other dependencies' upstream when investigating this. Will report back once 445 is out.

[mariofusco]

I'm one of the author of the new Jackson pool and I would like to investigate the memory issue it caused, but at the moment I haven't been able to reproduce it on my side. Can anybody please explain how I could reproduce this problem? Thanks.

[hashhar]

should be now addressed with #21361.

Can you validate the change @davseitsev if possible since you seem to have a way to reproduce this?

[davseitsev]

@hashhar I've applied your fix and testing it right now. I will update later.

[wendigo]

Closing then. Release should happen this week. Thanks!

[marvin-roesch]

This issue appears to still be affecting us on Trino 444. We are running into massive memory issues with this. In our case, heap dumps indicate that JsonTypeUtil.parseJson is at fault.

Some quick debugging shows that the issue is due to the SORTED_MAPPER.writeValue call using the default JsonGenerator instance which in turn gets created by a default JsonFactory that uses the LockFreePool implementation. Until the upstream maybe changes back the default again, it might be needed to perform a more systematic survey of Jackson usages to make sure this doesn't negatively affect more places.

Ideally, all ObjectMapper instances would get created with the customized JsonFactory that contains the configuration.

Alternatively, maybe explicitly downgrading the Jackson libraries is an option?

[mariofusco]

Note that we're discussing this issue also here, so in case you have any suggestion on how to reproduce the problem it would be great to report it also there. In particular it would be very interesting if you could give a look at this comment and let me know if this could be relevant for your environment.

[wendigo]

@mariofusco have you seen how we are parsing JSON: https://github.com/trinodb/trino/blob/master/lib/trino-plugin-toolkit/src/main/java/io/trino/plugin/base/util/JsonTypeUtil.java#L46 ?

[cowtowncoder]

I am also hoping to work on figuring out the underlying issue. I created a test:

FasterXML/jackson-core#1265

which suggests some abnormal temporary growth in retention, but not quite the smoking gun. So hoping to learn if there is something specific that might trigger or amplify issues.

Another thing that would be good to know -- although I realize it may not be easy to try it out -- is whether use of one of alternate new pools, JsonRecyclerPools.newConcurrentDequePool() -- would show good performance without requiring use of ThreadLocal. The reason I am interested in this choice is that for Jackson 2.17.1 there is need to change the default and choices are reverting back to pre-2.17 ThreadLocal-pool or concurrentDequePool. And it would be great to base decision on observed performance from a real concurrency-heavy use case which I think Trino would qualify as.