offset_reset_policy earliest resets offset to earliest on consumer reballance about broadway_kafka

[v0idpwn]

I'll open one in a few minutes, I have fixed it locally :)

[msaraiva]

Hi @victorolinasc!

I'll release a new version today.

Thanks for your contribution!

[josevalim]

Correct. Earliest is used only for the cases where you want to frequently start from the beginning. Latest is the one with the behavior you want. It is the latest committed.

[gerbal]

Then offset_reset_policy: :earliest has two effects

• start unknown kafka consumers at earliest offset
• reset all consumers to earliest offset any time brod receives any changes to partition assignments

While in offset_reset_policy: :latest only has one

• start unknown kafka consumers at latest offset

What should I do if I want new consumers to start at the earliest offset but not reset to the earliest offset on app restart?

[josevalim]

If you don't have a committed offset, then that's how latest behaves. If you have a committed offset, then it is not possible with the current options, but I believe Kafka has APIs that allow you to reset it manually.

[v0idpwn]

@josevalim I believe this is not the way :earliest is expected to behave. Offset reset policies should be used only when the offset is undefined or when the kafka current_offset is expired.

We can see this in Confluent docs: https://docs.confluent.io/current/clients/consumer.html#offset-management and I'd say even broadway_kafka documentation for offset_reset_policy implies this:

Defines the offset to be used when there's no initial offset in Kafka or if the current offset has expired. Possible values are :earliest or :latest. Default is :latest.

[josevalim]

You are correct. I misunderstood the docs. Thanks for the reference.

[josevalim]

In case someone wants to send a PR, it will be very appreciated too. :)

[victorolinasc]

Hey @josevalim ! Thanks for taking this in!

Can you release a minor version on hex? This would be greatly appreciated.

Thanks once again!

[msaraiva]

@v0idpwn thanks for the PR! ❤️