Comments (7)
@kempt09 I’m not following. The implementation of KafkaEx.ConsumerGroup is per the Kafka consumer group spec - there’s nothing really hidden, that’s just how a consumer group is supposed to work. The Kafka broker itself handles all of the coordination and monitoring. It will automatically distribute the partitions across the consumer group and will automatically handle situations like crashes or scale up/down. KafkaEx basically just says “here I am, ready to consume zero or more partitions for topic X” and lets the broker tell it which partitions it should consume.
That said, there’s really nothing that would stop you from implementing something like you described (iiuc). You can basically just start a process for each partition and manage the assignments yourself. Having said that, that’s what we used to do before we implemented the consumer group spec and having used both I would never go back ;)
from kafka_ex.
@dantswain must have really overlooked something on my end, got it working as you mentioned. Sorry to bother y'all. Thanks for the help in pointing me in the right direction.
from kafka_ex.
If this is already achievable with the current library I would be interested in knowing or someone documenting this, as I was unable to determine from reading through the codebase and some playing around with the consumers.
from kafka_ex.
Hi @kempt09 I don’t believe any of the processes in the current implementation are global, so you should be able to accomplish this just by starting consumer group members on each node, unless I’m misunderstanding your request?
from kafka_ex.
Kafka handles coordination within a consumer group; the only thing needed to make this happen is using the same consumer group name (and topic and brokers of course) between servers.
from kafka_ex.
So I think that is where maybe I am just confused with the library on starting a bunch of consumers within the same group that each have their own partition they consume from. It looks like using the KafkaEx.ConsumerGroup
is probably hiding some of the ability for me to direct the consumers directly to be spread across some nodes? Wishing it was as simple as knowing this topic has n partitions and just start n consumers all registered to the same group and assign them a partition but allow them to be globally registered consumers so they can be spread out with something like Horde and using process rebalancing. I could have over looked it though in the code and docs. Maybe this is achievable without needing the KafkaEx.ConsumerGroup
.
from kafka_ex.
I see the convenience of the consumer group spec for sure. We have been using it and its nice and real easy to get up and running. We ran into an issue where our consumers are doing some intense work but as we add more data need some more cpu power. Imma play around for a bit in the code and see if I can pick up what your putting down :D
from kafka_ex.
Related Issues (20)
- snappy compatibility HOT 1
- Host gets hashed in metadata? HOT 2
- username and password support HOT 4
- Starting a standalone consumer HOT 7
- Deserializer error. Seems to be trying to deserialize not serialized data HOT 1
- How to gracefully shutdown consumer group?
- KafkaEx.ConsumerGroup starting suerpvisor tree - Supervisor.Spec.supervisor/2 is deprecated. HOT 1
- Error using Kayrock with SSL
- metadata cache does not get updated when producing event HOT 1
- encountered nonexistent topic when Kafka restarts HOT 1
- badmatch in NetworkClient HOT 2
- Share state for consumer group with multiple topics HOT 3
- Kafka_ex process exit without restart when request timeout HOT 2
- record headers does not work with fetch v3 HOT 1
- Create a new release with recent changes? HOT 3
- Consumers not showing up on strimzi kafka grafana dashboard
- :unsupported_version error HOT 7
- Typespec for KafkaEx.Protocol.Produce.Message with optional :timestamp
- `ssl_options` require configuration even when `use_ssl` is `false` in Erlang/OTP-26
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kafka_ex.