Optimize memory usage about kafka-large-message-serde

[philipp94831]

Hi @TomBillietKlarrio, I was not aware that Kafka serializers have access to the record headers (guess I should read the docs). Also Avro serializer does not use headers but rather magic bytes which did not lead us to questioning that decision...

Your idea sounds great, especially if it is a configurable feature. Feel free to submit a PR and we will review it as soon as possible

[philipp94831]

@TomBillietKlarrio The changes required for this are actually quite large. I think it will be better if we implement it ourselves. I expect it to be ready some time next week

[TomBillietKlarrio]

I expected it to be fairly small actually. But you'll be able to estimate better.
Fine for me too, I first had some other fixes I need to do anyway. Much appreciated!

[TomBillietKlarrio]

Cool, thanks for the effort.

[philipp94831]

Hi @TomBillietKlarrio, we just released version 2.2.0 which adds a new config large.message.use.headers. If you enable it, your problem should be solved

[TomBillietKlarrio]

Just tested it and works fine.

[philipp94831]

Hi @TomBillietKlarrio, we just had to release version 2.3.0 which uses different headers to distinguish key and values. Please update if possible as it is not compatible with the headers used in 2.2.0

[TomBillietKlarrio]

Thanks for the heads-up. Took in the new version.