How to debug org.apache.kafka.common.errors.SerializationException about abris

[cerveada]

Hello, I am not sure if this is problem with Abris, since it's Kafka exception. Could you post the whole Exception?

If you try yo read from Kafka and just show the raw data without calling Abris is it working fine?

[suchitgupta01]

@cerveada Thanks for going through it.

Exception: Caused by: java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericRow cannot be cast to org.apache.avro.generic.GenericRecord

I do get byte array by doing: I was looking for a way by which I can skip schema validation and just print the incoming message(not as byte array)
kafkaDataFrame .writeStream .format("console") .option("truncate", "false") .start() .awaitTermination()

Batch: 1
|[00 00 00 01 33 BA 01]|[00 00 00 01 90 02 BA 01 48 39 39 61 35 64 31 37 32 2D 31 32 38 65 2D 34 65 37 39 2D 39 34 65 61 2D 38 65 30 63 64 30 62 31 37 36 36 32 48 61 32 35 62 35 37 66 66 2D 62 33 61 38 2D 34 64 64 32 2D 39 63 36 37 2D 65 32 37 37 31 35 62 32 39 33 34 63 02 40 30 63 61 36 64 35 35 37 39 30 36 63 34 30 66 30 33 63 30 38 39 32 39 65 35 34 36 66 37 66 37 61 00 12 63 61 73 74 6C 69 67 68 74 32 54 65 61 6D 20 4C 65 61 64 65 72 62 6F 61 72 64 20 50 72 6F 67 72 61 6D 73 02 50 54 68 69 73 20 69 73 20 61 20 63 6F 6D 70 65 74 69 74 69 76 65 20 77 61 6C 6B 69 6E 67 20 63 68 61 6C 6C 65 6E 67 65 2E CC 99 02 F0 99 02 CC 99 02 F0 99 02 0C 2D 30 37 3A 30 30 16 4C 45 41 44 45 52 42 4F 41 52 44 00 00 00 02 34 02 92 08 7B 22 6D 61 78 69 6D 75 6D 53 69 7A 65 22 3A 33 2C 22 6A 6F 69 6E 53 74 72 61 74 65 67 79 22 3A 7B 22 74 79 70 65 22 3A 22 55 73 65 72 4F 72 41 75 74 6F 22 2C 22 61 75 74 6F 4A 6F 69 6E 54 65 61 6D 53 74 72 61 74 65 67 79 43 6F 6E 66 69 67 75 72 61 74 69 6F 6E 22 3A 7B 22 74 79 70 65 22 3A 22 41 75 74 6F 22 2C 22 63 72 65 61 74 65 54 65 61 6D 49 66 4E 6F 74 45 78 69 73 74 73 22 3A 74 72 75 65 2C 22 74 65 61 6D 41 6C 6C 6F 63 61 74 69 6F 6E 41 6C 67 6F 72 69 74 68 6D 22 3A 22 53 45 51 55 45 4E 54 49 41 4C 22 7D 2C 22 75 73 65 72 53 65 6C 65 63 74 65 64 4A 6F 69 6E 54 65 61 6D 53 74 72 61 74 65 67 79 43 6F 6E 66 69 67 75 72 61 74 69 6F 6E 22 3A 7B 22 74 79 70 65 22 3A 22 55 73 65 72 22 7D 7D 2C 22 63 72 65 61 74 65 53 74 72 61 74 65 67 79 22 3A 7B 22 74 79 70 65 22 3A 22 55 73 65 72 4F 72 41 75 74 6F 22 2C 22 76 69 73 69 62 69 6C 69 74 69 65 73 22 3A 5B 22 50 55 42 4C 49 43 22 2C 22 50 52 49 56 41 54 45 22 5D 2C 22 61 75 74 6F 43 72 65 61 74 65 54 65 61 6D 53 74 72 61 74 65 67 79 43 6F 6E 66 69 67 75 72 61 74 69 6F 6E 22 3A 7B 22 74 79 70 65 22 3A 22 41 75 74 6F 22 2C 22 76 69 73 69 62 69 6C 69 74 69 65 73 22 3A 5B 22 50 55 42 4C 49 43 22 5D 2C 22 74 65 61 6D 4E 61 6D 65 50 72 6F 76 69 64 65 72 22 3A 22 52 41 4E 44 4F 4D 5F 4C 49 4E 45 22 7D 2C 22 75 73 65 72 53 70 65 63 69 66 69 65 64 43 72 65 61 74 65 54 65 61 6D 53 74 72 61 74 65 67 79 43 6F 6E 66 69 67 75 72 61 74 69 6F 6E 22 3A 7B 22 74 79 70 65 22 3A 22 55 73 65 72 22 2C 22 76 69 73 69 62 69 6C 69 74 69 65 73 22 3A 5B 22 50 55 42 4C 49 43 22 2C 22 50 52 49 56 41 54 45 22 5D 7D 7D 7D 02 70 7B 22 70 72 6F 67 72 61 6D 49 6E 73 74 61 6E 63 65 49 64 22 3A 22 37 31 66 65 66 35 34 32 37 39 32 38 34 30 31 61 62 35 61 38 38 63 63 66 61 39 37 66 38 62 63 65 22 7D 00 28 32 30 31 39 2D 30 35 2D 30 36 54 31 39 3A 30 35 3A 30 34 5A 28 32 30 31 39 2D 31 32 2D 31 37 54 30 37 3A 32 31 3A 34 33 5A 02 48 64 64 33 66 61 32 35 66 2D 61 34 32 66 2D 34 65 61 34 2D 38 34 33 65 2D 31 62 39 64 37 63 33 38 39 31 62 64 02 BA 01 48 39 39 61 35 64 31 37 32 2D 31 32 38 65 2D 34 65 37 39 2D 39 34 65 61 2D 38 65 30 63 64 30 62 31 37 36 36 32 48 61 32 35 62 35 37 66 66 2D 62 33 61 38 2D 34 64 64 32 2D 39 63 36 37 2D 65 32 37 37 31 35 62 32 39 33 34 63 02 40 30 63 61 36 64 35 35 37 39 30 36 63 34 30 66 30 33 63 30 38 39 32 39 65 35 34 36 66 37 66 37 61 00 12 63 61 73 74 6C 69 67 68 74 32 54 65 61 6D 20 4C 65 61 64 65 72 62 6F 61 72 64 20 50 72 6F 67 72 61 6D 73 02 52 54 68 69 73 20 69 73 20 61 20 63 6F 6D 70 65 74 69 74 69 76 65 20 77 61 6C 6B 69 6E 67 20 63 68 61 6C 6C 65 6E 67 65 2E 31 CC 99 02 F0 99 02 CC 99 02 F0 99 02 0C 2D 30 37 3A 30 30 16 4C 45 41 44 45 52 42 4F 41 52 44 00 00 00 02 34 02 92 08 7B 22 6D 61 78 69 6D 75 6D 53 69 7A 65 22 3A 33 2C 22 6A 6F 69 6E 53 74 72 61 74 65 67 79 22 3A 7B 22 74 79 70 65 22 3A 22 55 73 65 72 4F 72 41 75 74 6F 22 2C 22 61 75 74 6F 4A 6F 69 6E 54 65 61 6D 53 74 72 61 74 65 67 79 43 6F 6E 66 69 67 75 72 61 74 69 6F 6E 22 3A 7B 22 74 79 70 65 22 3A 22 41 75 74 6F 22 2C 22 63 72 65 61 74 65 54 65 61 6D 49 66 4E 6F 74 45 78 69 73 74 73 22 3A 74 72 75 65 2C 22 74 65 61 6D 41 6C 6C 6F 63 61 74 69 6F 6E 41 6C 67 6F 72 69 74 68 6D 22 3A 22 53 45 51 55 45 4E 54 49 41 4C 22 7D 2C 22 75 73 65 72 53 65 6C 65 63 74 65 64 4A 6F 69 6E 54 65 61 6D 53 74 72 61 74 65 67 79 43 6F 6E 66 69 67 75 72 61 74 69 6F 6E 22 3A 7B 22 74 79 70 65 22 3A 22 55 73 65 72 22 7D 7D 2C 22 63 72 65 61 74 65 53 74 72 61 74 65 67 79 22 3A 7B 22 74 79 70 65 22 3A 22 55 73 65 72 4F 72 41 75 74 6F 22 2C 22 76 69 73 69 62 69 6C 69 74 69 65 73 22 3A 5B 22 50 55 42 4C 49 43 22 2C 22 50 52 49 56 41 54 45 22 5D 2C 22 61 75 74 6F 43 72 65 61 74 65 54 65 61 6D 53 74 72 61 74 65 67 79 43 6F 6E 66 69 67 75 72 61 74 69 6F 6E 22 3A 7B 22 74 79 70 65 22 3A 22 41 75 74 6F 22 2C 22 76 69 73 69 62 69 6C 69 74 69 65 73 22 3A 5B 22 50 55 42 4C 49 43 22 5D 2C 22 74 65 61 6D 4E 61 6D 65 50 72 6F 76 69 64 65 72 22 3A 22 52 41 4E 44 4F 4D 5F 4C 49 4E 45 22 7D 2C 22 75 73 65 72 53 70 65 63 69 66 69 65 64 43 72 65 61 74 65 54 65 61 6D 53 74 72 61 74 65 67 79 43 6F 6E 66 69 67 75 72 61 74 69 6F 6E 22 3A 7B 22 74 79 70 65 22 3A 22 55 73 65 72 22 2C 22 76 69 73 69 62 69 6C 69 74 69 65 73 22 3A 5B 22 50 55 42 4C 49 43 22 2C 22 50 52 49 56 41 54 45 22 5D 7D 7D 7D 02 70 7B 22 70 72 6F 67 72 61 6D 49 6E 73 74 61 6E 63 65 49 64 22 3A 22 37 31 66 65 66 35 34 32 37 39 32 38 34 30 31 61 62 35 61 38 38 63 63 66 61 39 37 66 38 62 63 65 22 7D 00 28 32 30 31 39 2D 30 35 2D 30 36 54 31 39 3A 30 35 3A 30 34 5A 28 32 30 31 39 2D 31 32 2D 31 37 54 31 35 3A 31 38 3A 30 36 5A 02 48 64 64 33 66 61 32 35 66 2D 61 34 32 66 2D 34 65 61 34 2D 38 34 33 65 2D 31 62 39 64 37 63 33 38 39 31 62 64 02 16 30 2E 38 2E 33 2E 46 69 6E 61 6C 1A 64 65 6E 2D 71 61 30 31 2D 64 62 31 31 FE FF FF FF 1F DC C6 C7 DF 0B 00 20 6D 79 73 71 6C 2D 62 69 6E 2E 30 30 30 30 33 31 BA E4 B8 43 00 00 00 02 F8 FF 94 01 02 26 63 68 61 6C 6C 65 6E 67 65 73 32 5F 64 65 6E 71 61 30 31 02 14 63 68 61 6C 6C 65 6E 67 65 73 00 02 75 02 C6 C0 80 C7 E2 5B]|topic-name|0 |5212 |2019-12-17 07:18:06.412|0 |

[cerveada]

I'm not sure what exactly do you mean by validation. If the schema is not valid how do you want to deserialize the data? I would expect invalid schema to cause errors during deserialization.

When you use from_confluent_avro Abris work with two schemas. Reader schema loaded from schema registry according to configuration and writer schema also loaded froms chema registry, but this time by schema id that is on start of the payload (According to confluent specs). This two schemas must be compatible and there is no way to turn this off.

The other option is to use from_avro and provide the schema by yourself. This way you can specify any schema you want, but if there is this schema_id on start of the payload you need to trim it by yourself before handing the column to Abris.