When comparing record complex types the fields/symbols/unions must be in the same order to be compatible about avro

[nrwiersma]

In Kafka with a registry, no compatibility is needed as the schema id is in the message.

In either cause the compatibility function will not change and is not wrong. Should you wish to have one that considers field ordering I suggest you use your own function.

[nrwiersma]

If you read the rules dictating schema compatibility here: https://avro.apache.org/docs/1.11.1/specification/_print/#schema-resolution

if both are records:
the ordering of fields may be different: fields are matched by name.

This is generally the same for all the cases you mention.

[rmb938]

Right but when data is actually decoded from binary the order matters. If I have field A & B encoded to binary then accidentally decode it as B & A because compatibility says the schema is compatible, I will load the wrong data into the wrong fields. Same thing if I change the order of enums, ect... If I load data into a schema with a different order I'm going to get the wrong data.

…

On Mon, Apr 3, 2023, 11:46 PM Nicholas Wiersma ***@***.***> wrote: If you read the rules dictating schema compatibility here: https://avro.apache.org/docs/1.11.1/specification/_print/#schema-resolution if both are records: the ordering of fields may be different: fields are matched by name. This is generally the same for all the cases you mention. — Reply to this email directly, view it on GitHub <#249 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEE6IK57JYH232O6D6L6FTW7ORSXANCNFSM6AAAAAAWR5XR4Q> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[nrwiersma]

You need to make a compatibility schema between the reader and the writer, which avro does not yet support. But the compatibility function is correct.

[rmb938]

The function may be correct according to the spec but if I'm trying to read some data and I need to check compatibility before I do I'm going to end up with problems. Which means in an actual real world scenario where I need to be 100% sure I read the right data but I'm not 100% sure I have the same schema I can't actually rely on the compatibility function to make that decision.

…

On Tue, Apr 4, 2023, 2:02 AM Nicholas Wiersma ***@***.***> wrote: You need to make a compatibility schema between the reader and the writer, which avro does not yet support. But the compatibility function is correct. — Reply to this email directly, view it on GitHub <#249 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEE6II6BOAZCUPQTSXAJMLW7PBPJANCNFSM6AAAAAAWR5XR4Q> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[nrwiersma]

Cross schema reading is not supported in avro yet.

[rmb938]

Yea but while it may not be officially supported yet, it's a know practice and people have been doing it for many years. For example Kafka and confluent schema registry; they actively do it and document that it's possible. Even other than Kafka there are many blogs posts, stackoverflows, ect.. That point out that yes the spec says field ordering doesn't matter however, when you have to do a schema upgrade and temporary read and write with different schemas, it does matter.

…

On Tue, Apr 4, 2023, 2:23 AM Nicholas Wiersma ***@***.***> wrote: Cross schema reading is not supported in avro yet. — Reply to this email directly, view it on GitHub <#249 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEE6IOK7PQ7ZTWYIMT5AALW7PD57ANCNFSM6AAAAAAWR5XR4Q> . You are receiving this because you authored the thread.Message ID: ***@***.***>