`internalCreateDataFrame` error about abris

[felipemmelo]

Hi @dantswain , thanks for the report. I'll investigate and come back to you asap. Cheers.

[hemanshupaliwa7]

Did this issue got resolved? Any Hint. My same code used to work before on my local and now it's not working when i moved the code to production env

[dantswain]

@hemanshupaliwa7 I ended up not using this library :/ I did, however, discover that my topic is compacted. I do not know if that was the cause of this error not.

[felipemmelo]

Hi guys, sorry for the late reply, have been quite busy.

@hemanshupaliwa7 , the error message seems to be related to some binary incompatibility. Can you share your Spark, Kafka and Kafka integration versions? Also, how are you producing your data, is it by using Confluent Kafka Producer?

[KenpachiRules]

@hemanshupaliwa7 For easy verification just check for spark.driver.extraClassPath and spark.executor.extraClassPath in the submitted spark application and see which version of spark-sql jar is getting loaded , it should be spark-sql-2.3 minimum because the signature of SparkSession.internalCreateDataFrame has changed between 2.2 and 2.3 spark , there is an additional boolean parameter.

[felipemmelo]

If any issues, please feel free to reopen it.