Comments (5)
@yordis just wondering what kind of failure do you have in mind?
- Failure because you taged a message a failed by using
Broadway.Message.failed(message, "reason")
- Failure because there was an exception while the
handle_message
was running - Failure because the beam crashed?
I suppose you mean all the cases but I think if we break it down, we can produce a better doc!
So my understanding:
- If you tag a message as failed, this lib still sends an ack to Kafka for that message. It's due to the fact how messages are stored in Kafka and it's not as simple of let's say RabbitMQ to reprocess a message
- if there is an exception, again there will be an ack sent to Kafka
- if beam crashes, there wouldn't be an opportunity to send an ack. so the message will be re-read again
One thing to keep in mind about the acking is that ack is sent after batch of messages are processed. Each GenStage consumer gets 5 tasks(as default value) and after all 5 are processed(with or without error) the last offset is going to be committed to the Kafka broker.
from broadway_kafka.
@slashmili right, I was referring to the 3 cases you describe.
Ideally, the documentation could showcase how to do Dead-Letter-Queues or a strategy to re-consume the message in case of the processor fails when you have to guarantee at-least-once successful processing where it is critical to make sure the message was processed.
from broadway_kafka.
If we can validated my understandings, we can write them down in Handling failed messages section.
the documentation could showcase how to do Dead-Letter-Queues or a strategy to re-consume the message
This is complicated subject. While we can show that on handle_error
, you could put the message in a DLQ(or another topic). Showing how to re-consume is highly depends on the application and nature of failure. It also requires the app keeping track of amount of retries and stop retrying after X times....
I think the first step is to create a PR to add a section to explain the failure scenarios and consequences.
from broadway_kafka.
@slashmili yeah, I hear you, I think they are endless use cases on how we could do it, but there are also a few basic ways we could showcase since they are common enough, like DLQs after X failures.
Ideally, what is the basic example of doing at least one-processing guarantee without losing any message?
from broadway_kafka.
This is related to the discussion in #30. As you can see, there are many different approaches people may take. Documenting different strategies sound like a good call. 👍 Closing in favor of #30.
from broadway_kafka.
Related Issues (20)
- Offsets accumulating in the producer ack state HOT 5
- Support :query_api_versions brod option HOT 1
- Cut release 0.3.6 ? HOT 2
- Consumer Static Membership HOT 9
- No rejoin after "payload connection down :shutdown, :tcp_closed}" deadlock on race between assigments_revoked call and handle DOWN message HOT 16
- the table identifier does not refer to an existing ETS table HOT 5
- Deadlock on race between assigments_revoked call and handle DOWN message HOT 3
- drain_after_revoke failed due to killed process HOT 3
- Producers stuck in :assignments_revoked causing endless group rebalancing HOT 24
- Feature: Add option to set the starting offset for new consumer HOT 6
- Backoff strategy HOT 1
- Manual Partition Assignment HOT 4
- Allow to force consume the topic from the beginning or the end
- Undesirable resource usage related to producer concurrency HOT 8
- Add support for reseting offsets to a specific timestamp HOT 1
- Request for a new release HOT 1
- Offsets accumulating in the producer ack state (take 2) HOT 6
- Updates on the release date of the next version? HOT 5
- Fails to compile on otp 27 HOT 9
- Implementing offset lag telemetry HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from broadway_kafka.