Comments (7)
Hi @arjun289, you should be able to just call fetch in a loop for each partition.
from kafka_ex.
Thanks for the offer @arjun289 I think your use case is not really outside the intended use case for consumer groups, so I’m not sure what value there would be in having another implementation. On the other hand, there are a few benefits to using the consumer group mechanism here. For example, if you accidentally start a duplicate consumer there will be no double-consuming of messages.
from kafka_ex.
Thanks for your suggestion @dantswain.
The real issue here is that there are stream processing chains implemented with flow-based programming. All these chains belong to a specific project which group these chains.
Each time the user creates a chain it subscribes to a specific topic in Kafka depending on the project. The topic is basically telemetry which is created per project.
All the chains in the project subscribe to this topic, hence they should be individual consumers without any group as every chain should receive every event coming to the topic and will process it in its own way.
As per the above suggestion, I will be creating a worker per chain which would call the fetch. However, will it take care of semantics that is supported for GenConsumer such as offset management?
from kafka_ex.
Ah. Maybe I didn’t understand your requirement. You can manually commit the offsets. However, I think you may be able to just use consumer groups with a single member. The cluster should just assign all partitions to that consumer, and then offset management should work as expected.
from kafka_ex.
Thanks, @dantswain I think I will go ahead with the consumer group
suggestion as it reduces efforts on my side.
from kafka_ex.
Will it be helpful if I raise a PR with implementation to handle standalone consumer?
from kafka_ex.
@arjun289 the consumer_group
in the config can be set to :no_consumer_group
Starting a individual consumer can be done like this:
KafkaEx.GenConsumer.start_link(
Cx.ExampleGenConsumer,
"foo",
"foo",
0,
commit_interval: 5000,
commit_threshold: 100
)
Not sure about the supervisor yet.
from kafka_ex.
Related Issues (20)
- snappy compatibility HOT 1
- Host gets hashed in metadata? HOT 2
- username and password support HOT 4
- Deserializer error. Seems to be trying to deserialize not serialized data HOT 1
- How to gracefully shutdown consumer group?
- KafkaEx.ConsumerGroup starting suerpvisor tree - Supervisor.Spec.supervisor/2 is deprecated. HOT 1
- Error using Kayrock with SSL
- metadata cache does not get updated when producing event HOT 1
- encountered nonexistent topic when Kafka restarts HOT 1
- badmatch in NetworkClient HOT 2
- Share state for consumer group with multiple topics HOT 3
- Kafka_ex process exit without restart when request timeout HOT 2
- record headers does not work with fetch v3 HOT 1
- Create a new release with recent changes? HOT 3
- Consumers not showing up on strimzi kafka grafana dashboard
- Feature Request: Distributed Consumers HOT 7
- :unsupported_version error HOT 7
- Typespec for KafkaEx.Protocol.Produce.Message with optional :timestamp
- `ssl_options` require configuration even when `use_ssl` is `false` in Erlang/OTP-26
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kafka_ex.