bulldog2011 / luxun Goto Github PK

1. Async Producer server
   AsyncProducer.send(...) : throw QueueFullException occurred
   -> Still in the message transfer request
2. Broker server die
3. Producer server
   java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
4. queue.addAll(events) execute

private void tryToHandle(List<QueueItem<T>> events) {
    if (logger.isDebugEnabled()) {
        logger.debug("handling " + events.size() + " events");
    }
    if (events.size() > 0) {
        try {
            this.eventHandler.handle(events, underlyingProducer, serializer);
        } catch (ConnectionRefusedException e) {
            List<QueueItem<T>> remainedItems = new ArrayList<QueueItem<T>>(events);
            while (queue.size() > 0) {
                remainedItems.add(queue.poll());
            }
            if (this.callbackHandler != null) {
                this.callbackHandler.connectionRefused(e.getMessage(), remainedItems);
            }
        } catch (RuntimeException e) {
            logger.error("Error in handling batch of " + events.size() + " events", e);
            queue.addAll(events);
        }
    }
}

Queue full & ProducerSendThread die.

java.lang.IllegalStateException: Queue full
        at java.util.AbstractQueue.add(AbstractQueue.java:98)
        at java.util.AbstractQueue.addAll(AbstractQueue.java:187)
        at com.leansoft.luxun.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.java:163)
        at com.leansoft.luxun.producer.async.ProducerSendThread.processEvents(ProducerSendThread.java:126)
        at com.leansoft.luxun.producer.async.ProducerSendThread.run(ProducerSendThread.java:63) 

solution

1. IllegalStateException occurred
2. tryToHandle(events) -> re execute
3. Thrift connection attempt
4. Connection attempt timeout.
5. ConnectionRefusedException occurred
6. CallbackHandler.connectionRefused(...) execute

Is it possible to supply acknowledge option and to remove the message from the queue after consumer acknowledged?
In case of producer or consumer crash, during recovery, we might not expect the consumed messages.
Otherwise the consumer application might be designed as idempotent.

How do you think about it(or I have misunderstand Luxun's architecture )

how many topics can handled by one node?

It is good to expose the low level interface - getSize : total number of items remaining in the queue - to high level api and JMX for stats or morning.

Steps:

1. 10K send by a producer with props.put("producer.type", "async");
2. try to consume by index simpleConsumer1.consume("test-topic", index, 1024); in bounded while loop
3. after the the first iteration, it failed with:
   Exception in thread "main" java.lang.IndexOutOfBoundsException
   at com.leansoft.luxun.common.exception.ErrorMapper.toException(ErrorMapper.java:34)
   at com.leansoft.luxun.consumer.SimpleConsumer.consume(SimpleConsumer.java:86)

Hi team,
I looked the design doc and saw consumer section, it will fetch messages in a blockingqueue first and up level consumer thread parallel consume. In case one up level consume one message failed, how we know it and let us consume again?

Hi

Just wanted to know if the project is still active.