Decode each individual payload using its respective schema id from registry about abris

[felipemmelo]

Hi @jquadrino , this is not possible with Spark as each Dataframe can only have one schema.

Cheers,

Felipe.

[jquadrino]

This is not quite compatible with confluent then, each payload has a schema id and using a different schema or global schema will introduce error for evolving or large schemas.

…

On Sun, Sep 22, 2019, 4:03 PM Felipe Melo ***@***.***> wrote: Hi @jquadrino <https://github.com/jquadrino> , this is not possible with Spark as each Dataframe can only have one schema. Cheers, Felipe. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#55?email_source=notifications&email_token=AASNGPJ5E26AHS5Y7VBMOLDQK7FRNA5CNFSM4IYZ62O2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7JOAQI#issuecomment-533913665>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASNGPIARFEAXX3K6ELSL5TQK7FRNANCNFSM4IYZ62OQ> .

[felipemmelo]

Well, that depends on how we approach compatibility and which tool we are talking about.. Taking into consideration Spark limitations in terms of Dataframe schemas, then yes, Spark itself is not compatible.

On the other hand, considering that vanilla Spark would break on Confluent Avro data and Abris won't, then Abris is compatible with that format, although there is no way to change the schema in a per record basis.

Anyway, since this is not an Abris issue, I'll close the issue. Feel free to reopen if needed.