Invalid value null for configuration key serializer exception about kafka-large-message-serde

[rajeshdasi]

Thanks for your quick support @philipp94831.

[philipp94831]

You are not configuring the producer correctly. You have to use

props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, Serdes.StringSerde.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, S3BackedSerializer.class);

[rajeshdasi]

After adding these properties I'm getting this error

Exception in thread "main" org.apache.kafka.common.KafkaException: Failed to construct kafka producer
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:465)
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:290)
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:317)
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:302)
	at com.example.KafkaLargeMessageSerdeProcessor.processLargeMessageSerde(KafkaLargeMessageSerdeProcessor.java:35)
	at com.example.Main.main(Main.java:14)
Caused by: org.apache.kafka.common.KafkaException: class org.apache.kafka.common.serialization.Serdes$StringSerde is not an instance of org.apache.kafka.common.serialization.Serializer
	at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:405)
	at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:436)
	at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:421)
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:380)
	... 5 more

[philipp94831]

Oh sorry. You have to put StringSerializer. Nevertheless, these are questions unrelated to this project but rather general Kafka API usage so please refer to the respective docs

[rajeshdasi]

Can you share a complete working example to use this library? I want to push the file to s3 using this library and get the metadata where the file is stored.

[philipp94831]

You have to interact with S3 yourself in that case. This libraries intend is to transparently offload large messages sent through Kafka to a blob storage.

[rajeshdasi]

Do we have any complete example for offloading the large message sent through Kafka to blob storage?

[philipp94831]

        final Properties props = new Properties();
        props.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, LargeMessageSerializer.class);
        props.setProperty(AbstractLargeMessageConfig.BASE_PATH_CONFIG, "s3://large-blob-data-test/");
        props.put(LargeMessageSerdeConfig.VALUE_SERDE_CLASS_CONFIG, Serdes.StringSerde.class);

        try (final Producer<String, String> producer = new KafkaProducer<>(props)) {
            final ProducerRecord<String, String> record = new ProducerRecord<>("topic", "key", "value");
            producer.send(record);
            producer.flush();
        }

[rajeshdasi]

I tried this way @philipp94831 but it's throwing an error.

Exception in thread "main" org.apache.kafka.common.KafkaException: Failed to construct kafka producer
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:465)
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:290)
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:317)
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:302)
	at com.example.KafkaLargeMessageSerdeProcessor.processLargeMessageSerde(KafkaLargeMessageSerdeProcessor.java:35)
	at com.example.Main.main(Main.java:14)
Caused by: java.lang.NoClassDefFoundError: io/confluent/common/config/AbstractConfig
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1012)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:862)
	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:760)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:681)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1012)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:862)
	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:760)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:681)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
	at com.bakdata.kafka.LargeMessageSerializer.configure(LargeMessageSerializer.java:58)
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:390)
	... 5 more
Caused by: java.lang.ClassNotFoundException: io.confluent.common.config.AbstractConfig
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
	... 25 more

These are my dependencies

    implementation 'org.apache.kafka:kafka-clients:3.4.0'
    implementation('com.bakdata.kafka:large-message-serde:2.5.1')
    implementation('org.apache.kafka:kafka-streams:3.4.0')
    implementation platform('software.amazon.awssdk:bom-internal:2.20.28')

    implementation('org.slf4j:slf4j-simple:2.0.5')
    implementation 'software.amazon.awssdk:s3'
    implementation('com.bakdata.kafka:s3-backed-serde:1.1.5')
    implementation('io.confluent:common-config:7.3.1')

[philipp94831]

Please have a look at the README of this project.

Make sure to also add Confluent Maven Repository to your build file.