Potential deadlock about cl-rdkafka

[SahilKang]

Thanks for digging in and sharing the details 💯; I'm able to reproduce this some of the time with:

(ql:quickload '(cl-rdkafka babel))


(defparameter +topic+ "test-topic")

(defparameter +num-messages+ 100000)

(let ((producer (make-instance
                 'kf:producer
                 :conf '("bootstrap.servers" "127.0.0.1:9092")
                 :serde #'babel:string-to-octets)))
  (loop
     repeat +num-messages+
     do (kf:send producer +topic+ "test-message"))
  (kf:flush producer))


(defparameter +consumer+
  (make-instance
   'kf:consumer
   :conf '("bootstrap.servers" "127.0.0.1:9092"
           "group.id" "test-group-id"
           "enable.auto.commit" "false"
           "auto.offset.reset" "earliest"
           "offset.store.method" "broker"
           "enable.partition.eof" "false")
   :serde #'babel:octets-to-string))

(kf:subscribe +consumer+ +topic+)

(defparameter *count* 0)

(defparameter +lock+ (bt:make-lock "+lock+"))

(defparameter +channel+
  (let ((lparallel:*kernel*
         (lparallel:make-kernel 3 :name "test-kernel")))
    (lparallel:make-channel)))

(defun process ()
  (bt:with-lock-held (+lock+)
    (incf *count*)
    (kf:commit +consumer+ :asyncp t)))

(defun poll ()
  (let (message)
    (bt:with-lock-held (+lock+)
      (setf message (kf:poll +consumer+ 5)))
    (when message
      (lparallel:submit-task +channel+ #'process))))


(lparallel:submit-task
 +channel+
 (lambda ()
   (loop
      repeat +num-messages+
      do (poll))))

Running (bt:all-threads) will sometimes show that all the threads are running okay, but other times I'll see the deadlock you described:

#<SB-THREAD:THREAD "worker" waiting on:
      #<MUTEX "address->queue-lock"
          owner: #<SB-THREAD:THREAD "cl-rdkafka" RUNNING {100271A6E3}>>
    {1005DB8F93}>

The reason why poll-loop won't release the lock is because this lparallel.queue:pop-queue call will block indefinitely when the queue is empty. it's not a valid state to reach this call with an empty queue and the reason why it's occurring is because the commit call is not grabbing the mutex early enough: the mutex should be grabbed before the cl-rdkafka/ll:rd-kafka-commit-queue call a few lines above it.

I'll go into more details in the PR/commit that fixes this, but at a high level:

• this cl-rdkafka/ll:rd-kafka-commit-queue call adds more events to the rd-kafka-queue without grabbing the mutex
• this causes process-events to keep looping (process-events is called by poll-loop)
• this enqueue-payload will attempt to acquire the mutex but will be blocked because process-events is still looping with the lock acquired
• eventually this lparallel.queue:pop-queue call, which is called by process-events, blocks on an empty queue
• since enqueue-payload is the function which adds elements to this empty queue and since it's blocked while acquiring the mutex, we end up with a deadlock

For analogous reasons, a similar oversight also causes the same deadlock with the producer send method, so you get a double whammy 🏆