Could Producer.produce() be blocking NodeJS thread? about node-rdkafka

[webmakersteve]

Awesome. I'll include the read-write locks in the next release.

[martinambrus]

Judging from this comment, which says:

Each broker has its own thread so the initial blocking connect() does not affect any other part of the library.

I assume those threads, while trying to connect initially, will block the execution in librdkafka itself. But I would still like to know your opinion, as this is just my assumption here.

[webmakersteve]

.produce queues up a poll to get information about deliveries, etc. from kafka. I think what is happening potentially is that you are producing a message, which then queues up a poll. Poll is waiting for the connection to finish to actually get the next message, so the subsequent poll is blocking.

Can you make a test case that reproduces this behavior? If it is happening reliably, it is definitely unwanted behavior and I would like to fix it.

[martinambrus]

librdkafka-blocking-pool-poc.zip

here it comes... since we're working in Docker on Windows, I've included a docker file with instructions (2 lines of comments on top of Dockerfile) if you'd like to build this from scratch. Otherwise, this should work fine if you have librdkafka with NodeJS 6 on your machine.

[webmakersteve]

It looks like this is a bug related to a deadlock. In the meantime, the Producer.produce method is not meant to be used outside of a closure for the on('ready') event or a callback from .connect. In this situation, it would throw anyway.

However, that doesn't mean I don't want to fix the deadlock!

[webmakersteve]

I implemented a read/write lock instead of the single mutex for connections. You can try running your application with the code at this branch here:

https://github.com/Blizzard/node-rdkafka/tree/deadlock-dr-cb

There are a few other fixes I need to put in before it is merge-able, but for the time being it will be there. Let me know how it works for you.

[martinambrus]

seems to work fine now, I'm getting info about brokers being offline without anything being blocked, thumbs up and thank you!