new possible feature: toConfluentAvro with Dataset[T] for user provided T about abris

[Zhen-hao]

I think I am mixing up two things here.

1. I want a Dataset API.
2. This library doesn't seem to work with Structured Streaming even with Dataframe.

Exception in thread "main" org.apache.spark.sql.AnalysisException: 'write' can not be called on streaming Dataset/DataFrame;
	at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
	at org.apache.spark.sql.Dataset.write(Dataset.scala:3103)

[felipemmelo]

Hi Zhen-hao,

Foremost, thank you very much for your message.

Things are definitely getting mixed here, since working with structured APIs is exactly what this library is about. :)

Four points:

1. Have you tried to run the example presented here ?

2. Your message is telling that write cannot be called on Dataset/Dataframe, which means it is being called after the Dataframe/Dataframe has been returned from the library.

3. Can you attach your code?

4. Of course we're open to PRs. That will be a pleasure.

Best regards.

[Zhen-hao]

Hi Felip,
thanks for your fast response!
Yes, the error was gone when I replaced .save with .start. It seems you can't use save in structured streaming.
I think I was confused a lot by Structured Streaming, instead of your library. For example, I didn't see the normal Spark logs after I run the job. So I wasn't sure if the job started or not.
When I add logging and write to the console, sometimes I get the logs and sometimes not.

I will look into it further after my holiday and look forward to using this library in our production jobs.

[felipemmelo]

Great to read this, Zhen-hao. Just let me know if you stumble upon any issue.

[Zhen-hao]

Hi Felip,
After a deeper look at it, I realized that the Dataframe API is not working for me because after the .toDF operation the serializer picks up each row as bytes.
The schema generated will always be the following no matter what I put to Dataset; I tried case class and org.apache.avro.generic.GenericRecord.

{"subject":"topic_name","version":4,"id":2,"schema":"{\"type\":\"record\",\"name\":\"MyCaseClass\",\"namespace\":\"io.connecterra\"
,\"fields\":[{\"name\":\"value\",\"type\":[\"bytes\",\"null\"]}]}"}

I think I will have to implement something generic for Dataset[T].

[felipemmelo]

Hey Zhen-hao, thanks for pinging. May I have a look at your code? Luckily I've already stumbled upon something similar.

[Zhen-hao]

Essentially, my code is

val output =
    myDS
      .toDF
      .toConfluentAvro(TOPIC, "myTopic", "io.connecterra")(SchemaRegistryConfs)
      .writeStream
      .format("kafka")
      .option("kafka.bootstrap.servers", KAFKA_BOOTSTRAP_SERVERS)
      .option("topic", TOPIC)
      .option("checkpointLocation", "./checkpoint")
      .start()

output.awaitTermination()

[Zhen-hao]

I see that your library relies on the createConverterToAvro function from Databricks. I am not sure I want to build anything on that. I think I will try to use the avro4s library and some type-classes to make it work for Dataset[T].

[felipemmelo]

Hi Zhen-hao,

Not sure I got your issue. When you run myDS.toDF you should be able to retain the original schema of myDS. On library side, it will only be converted to Avro after toConfluentAvro.

Regarding createConverterToAvro from Databricks, what are the issues you see?

[Zhen-hao]

I am using Spark 2.3.
In general, I saw lots of anti-patterns and bad coding styles in Databricks code, in my standard of course.

[felipemmelo]

Ok, I see. thanks then.

[Zhen-hao]

my approach would be fairly simple. because I already have the transformation from Dataset[T] to Dataset[GenericRecard] for free thanks to the avro4s library, all I need is a module to talk to the schema registry and add the id bytes in the Array[Byte]. I think you already have that code ready for me to use :)

[felipemmelo]

Cool, happy it's solved :)

[felipemmelo]

Closing since it seems to be solved for now.