sahilkang / cl-rdkafka Goto Github PK

Hello, and thank you for the project! Our Kafka environment is as follows: v0.10.0.0 Broker -> v1.3.0 librdkafka -> cl-rdkafka-20200218-git (quicklisp).

We have a long-lived process which continually polls/commits. Occasionally, we see this message logged:

%4|1583796667.962|MAXPOLL|rdkafka#consumer-1| [thrd:main]: Application maximum poll interval (300000ms) exceeded by 226ms (adjust max.poll.interval.ms for long-running message processing): leaving group

(As an aside, adjusting the max poll interval does not cause this not to happen. As far as we can tell, the default is right-sized for our workload.)

Shouldn't this kind of thing be raised as a condition in cl-rdkafka?

Something like this signals a partition-error from the producer's send method:

(ql:quickload '(cl-rdkafka babel))

(let ((producer (make-instance
                 'kf:producer
                 :conf '("bootstrap.servers" "127.0.0.1:9092")
                 :serde #'babel:string-to-octets)))
  (loop
     repeat 500000
     do (kf:send producer "test-topic" "test-message"))
  (kf:flush producer))

The signaled condition looks something like:

Encountered error CL-RDKAFKA/LOW-LEVEL:RD-KAFKA-RESP-ERR--QUEUE-FULL for `test-topic:-1`: "Local: Queue full"

This is one of the expected errors that may occur from the rd_kafka_produce function and a restart should be provided in this case to recover.

I'm testing the latest release with these two functions:

(defun test-consume (topic-name)
  (let* ((string-serde (lambda (x)
                         (kf:bytes->object x 'string)))
         (conf (kf:conf
	        "bootstrap.servers" "127.0.0.1:9092"
	        "group.id" (write-to-string (get-universal-time))
	        "enable.auto.commit" "false"
	        "auto.offset.reset" "earliest"
	        "offset.store.method" "broker"
	        "enable.partition.eof"  "false"))
         (consumer (make-instance 'kf:consumer
			          :conf conf
                                  :key-serde string-serde
			          :value-serde string-serde))
         (topics (list topic-name)))
    (kf:subscribe consumer topics)
    (format t "Subscribed:  ~S, ~S.~%"
            (gethash "bootstrap.servers" conf)
            (kf:subscription consumer))
    (let ((message (kf:poll consumer 10000)))
      (format t "Message received or poll expired.~%")
      (when message
        (format t "Received message:  (~S) ~S~%"
                (kf:key message) (kf:value message))
        (kf:commit consumer)))
    (format t "Unsubscribing.~%")
    (kf:unsubscribe consumer)))

and

(defun test-produce (topic-name key message)
  (let ((producer (make-instance 'kf:producer
                                 :conf (kf:conf
                                        "bootstrap.servers" "127.0.0.1:9092")
                                 :key-serde #'kf:object->bytes
                                 :value-serde #'kf:object->bytes)))
    (kf:produce producer topic-name message :key key)
    (kf:flush producer 2000)))

When I monitor the topic with kafka-console-consumer, I see the messages produced by test-produce. When I run test-consumer and send messages with kafka-console-producer, it gets the messages. When I send messages with test-producer, test-consumer does not see them.
Please tell me I'm making a foolish mistake.

Maybe I don't know how to use it, but this fails for me:

(let ((config ("bootstrap.servers" "localhost:1234" "group.id" "test-lisp-consumer" "enable.auto.commit" "false"))
      (consumer (make-instance 'kf:consumer
                               :conf config
                               :serde #'babel:octets-to-string)))
  (kf:assign consumer (list (cons "my-topic" 0)))
  (kf:seek consumer "my-topic" 0 10 2000)
  consumer)

While doing this as separate steps in repl works, weirdly:

(setf consumer (make-instance 'kf:consumer
                               :conf config
                               :serde #'babel:octets-to-string))
(kf:assign consumer (list (cons "my-topic" 0)))
(kf:seek consumer "my-topic" 0 10 2000)
(kf:poll consumer 2000)

When I try the simple demo snippets in the README or run the test suite using docker I will get errors like this one:

test-test-1  | TEST/HIGH-LEVEL/ADMIN::CREATE-PARTITIONS-WITH-PRODUCER%3|1675427948.113|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.22.0.3:9092 failed: Connection refused (after 0ms in state CONNECT)
test-test-1  | %3|1675427948.113|ERROR|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: 1/1 brokers are down
test-test-1  | %3|1675427948.118|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.22.0.3:9092 failed: Connection refused (after 0ms in state CONNECT)
test-test-1  | Unhandled CL-RDKAFKA:KAFKA-ERROR in thread #<SB-THREAD:THREAD "cl-rdkafka" RUNNING
test-test-1  |                                               {1004234173}>:
test-test-1  |   Expected event-type `1`, not `8`
test-test-1  |
test-test-1  | Backtrace for: #<SB-THREAD:THREAD "cl-rdkafka" RUNNING {1004234173}>

I am working on an M1 Mac with librdkafka installed via homebrew. I can reach the kafka instance running inside of docker with command line tools such as kafka-topics --bootstrap-servers 127.0.0.1:9092 --list etc. But not via Common Lisp. Any ideas?

I have a consumer-only system which is utilizing lparallel to create a handful of threads. One of the threads calls kf:poll in a loop. Another thead is processing these messages, and when done is calling kf:commit. I noticed that -- seemingly randomly -- the pipeline I had established stops working.

I narrowed it down to determine it was blocking on the call to kf:commit, so I began placing format statements to narrow down where. This led me to the attempts to acquire a lock on +address->queue-lock+. The format statements seem to suggest a deadlock between poll-loop and enqueue-payload.

enqueue-payload: Attempting to hold lock.
poll-loop: Attempting to hold lock.
poll-loop: Lock held

(keep in mind output to stdio can arrive at different times)

enqueue-payload never reports that it's successfully acquired the lock, and poll-loop never reports that it's released the lock. I'm having trouble determining what shared resource the two stacks are sharing.

I am attempting to create a reproducible test-case, but I wanted to submit this information early to see if maybe anyone had some ideas.

librdkafka supports Windows so it's a shame that cl-rdkafka does not. Right now, the only thing preventing cl-rdkafka from running on Windows is the posix stuff in event-io.

Set client.software.name and client.software.version as described in the librdkafka language binding recommendations.

If you're going to put this in your readme, some newbie is going to come around and delete all of their docker images.

docker-compose down --rmi all
docker system prune -fa && docker volume prune -f

produce->consume.lisp is an invalid filename on Windows, so it's not even possible to clone this repo there. I suggest removing the >.

(SB-IMPL::FILE-PERROR #P"C:/Users/parallels/quicklisp/dists/quicklisp/software/cl-rdkafka-20200715-git/test/high-level/produce->consume.lisp" 123 SB-INT:SIMPLE-FILE-ERROR :FORMAT-CONTROL "Error opening ~S" :FORMAT-ARGUMENTS (#P"C:/Users/mkennedy/quicklisp/dists/quicklisp/software/cl-rdkafka-20200715-git/test/high-level/produce->consume.lisp"))