Schema fields do not consider default in String() methods about avro

[nrwiersma]

Hi,

String() as noted in the docs returns the canonical form of the schema. Canonical forms are meant to determine what is considered the same schema from the PoV of the reader. As default has not effect for a reader (it is only used by the writer to fill missing data) it is not included in the canonical form.

So this is very intentional and noted in the spec here (look at [STRIP]).

Hope that answers your question.

[mpetronic]

Thanks for the quick response. However, I don't believe this is correct. Per the spec, the canonical form is derived from the "writer" (input schema) not the "reader" schema... Transforming into Parsing Canonical Form Assuming an input schema (in JSON form) that's already UTF-8 text for a *valid* Avro schema (including all quotes as required by JSON), the following transformations will produce its Parsing Canonical Form The default values definitely play into the behavior of a reader schema as they are required for the reader to know what to return when the datum being read does not contain a field defined in the reader schema. The default value will be returned, otherwise, the schema resolution will fail if a default is not defined for a field that is missing in the input datum. This supports backward compatibility. For writing data, the default plays no part as the application is responsible for setting default values into fields that support defaults or setting actual required values into fields that do not support defaults. In all cases, the application MUST provide a value for every field defined in the input schema to serialize a valid datum. We have used the default value as a "hint" as to what we should write as a default but that is no way prescribed in the spec - it's just our approach sometimes. In my case, I am attempting to build a reader schema from a BQ table schema. It's fine if your library is not intended to support doing this. I just wanted to be certain that I was not misusing or misunderstanding how to use it

…

On Fri, Sep 10, 2021 at 2:15 AM Nicholas Wiersma ***@***.***> wrote: Hi, String() as noted in the docs <https://pkg.go.dev/github.com/hamba/avro?utm_source=godoc#Schema> returns the canonical form of the schema. Canonical forms are meant to determine what is considered the same schema from the PoV of the reader. As default has not effect for a reader (it is only used by the writer to fill missing data) it is not included in the canonical form. So this is very intentional and noted in the spec here <https://avro.apache.org/docs/current/spec.html#Transforming+into+Parsing+Canonical+Form> (look at [STRIP]. Hope that answers your question. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#115 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSTHDU56BOWWJNF4XFDS5LUBGO7NANCNFSM5DYT45MQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[nrwiersma]

From the spec:

Parsing Canonical Form is a transformation of a writer's schema that let's us define what it means for two schemas to be "the same" for the purpose of reading data written against the schema

So the canonical is definitely from the PoV of a reader.

The default values definitely play into the behavior of a reader schema as
they are required for the reader to know what to return when the datum
being read does not contain a field defined in the reader schema.

This is simply not true. The default is only applied by the writer. The encoded data will always have a value for a field, it is simply not possible for it not to as it would become impossible to decode. So the default is only applied by the writer and as no bearing and is useless to the reader.

[nrwiersma]

So I think I now understand the misunderstanding here. I think you are talking about composable schemas when a reader schema is composed with a writer schema to get a schema to read a datum. This library does not yet support this form. It is expected you will decode with the writer schema at this point.

This is something I have looked at in the past, but it is very complex. I hope to have the time at some point to take a crack at this.

At any rate, this form does not require default to be in the canonical form, and this field property is specifically removed from the canonical form in the spec.

Hope this helps.

[mpetronic]

Yes, this is exactly what I am talking about. As an example, say I have a schema A1 and A2 (two versions of the same family) where schema A1 has fields a,b,c and schema A2 has fields a, b, c, d. Assume the BQ table currently has a table schema matching A1 (three fields a,b,c). I need to be able to load Avro files generated with A1 and A2 in the same batch of files. BQ has to pick one of those schemas to represent the data to load as it uses Avro schema resolution in an effort to support loading mixed schema versions in a single batch load job. But there are two versions so which one does it pick for the reader schema? It picks the one that is alphabetically last in sort order by file name. Our pipeline design is such that we can assume that A1 is forward compatible with A2 and A2 is backward compatible with A1. Therefore, we can change the schema of BQ to A2 at any time, even before any A2 data is showing up OR the data producers can start to produce A2 data BEFORE we alter the BQ table from A1 to A2. This decouples schema evolution deployments nicely. To make this work, I generate an Avro schema from the BQ table schema, then, using that schema, I generate an OCF with no data - just the header - and I name it zz.avro to ensure it sorts last and is chosen by BQ as the reader schema through which it then reads all files in the batch. In this example, that zz.avro file's schema reflects A1 and that means that, if some A2 data shows up for loading, ONLY fields a,b,c will make it into BQ. Yes, new data (field d) is lost on load but this is by design and agreement so as to not have stuff break completely when some data producer starts producing new data before the BQ table schema is updated and this approach allows flexibility of schema deployments. So, given this design, yes, I am generating the zz.avro schema to be used as a READER schema by BQ under the hood and relying on Avro schema resolution to handle multiple WRITER versions. If I were to deploy A2 to BQ and some application was still producing against A1, that would work, too, because zz.avro would now be based on A2 (which would have a default value defined for new field d) so it would simply provide that default to fill in the missing field reflected in the BQ schema for any A1 data being loaded as part of a load job batch. I hope this explanation helps to explain why I was asking my question and why I needed defaults in the String(). I totally agree on the meaning of the PCF and its purpose that allows proper fingerprinting of schema to compare equality of the schemas as well as your decision to generate the schema via String() as you do. At any rate, I am changing my code to just generate my JSON schemas directly in custom code (not that hard) so as to handle my need for default values. I was just hoping to leverage someone else's kind contribution to OSS to address my need. :) I appreciate your work, nonetheless. We have some other Avro producing applications for which it may very well be suited for use. You can close this issue as I am satisfied we understand the problem now and again, thank you for your responses and inputs.

…

On Sat, Sep 11, 2021 at 12:20 AM Nicholas Wiersma ***@***.***> wrote: So I think I now understand the misunderstanding here. I think you are talking about composable schemas when a reader schema is composed with a writer schema to get a schema to read a datum. This library does not yet support this form. It is expected you will decode with the writer schema at this point. This is something I have looked at in the past, but it is very complex. I hope to have the time at some point to take a crack at this. At any rate, this form does not require default to be in the canonical form, and this field property is specifically removed from the canonical form in the spec. Hope this helps. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#115 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSTHDTWEDE3NN46BOCKVKTUBLKKFANCNFSM5DYT45MQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[nrwiersma]

Yeah, in your case your want to generate a full schema rather than a canonical schema. In reality it would not be a massive addition to the lib if you would be interested in making the contribution. I think the "correct" approach would be to handle it as a json.Marshaler, so you would json encode the schema. This would provide the best separation between canonical String() or Stringer and full.

Is this something you would be interested in?

[mpetronic]

Let me ponder it and I will get back to you. My plate is pretty full with this current project and another one just got added to my plate on top of that. :( I will review the code more to consider the approach I would use based on your suggestion, which makes good sense to me, and that will help me scope the effort.

…

On Sat, Sep 11, 2021 at 2:04 AM Nicholas Wiersma ***@***.***> wrote: Yeah, in your case your want to generate a full schema rather than a canonical schema. In reality it would not be a massive addition to the lib if you would be interested in making the contribution. I think the "correct" approach would be to handle it as a json.Marshaler, so you would json encode the schema. This would provide the best separation between canonical String() or Stringer and full. Is this something you would be interested in? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#115 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSTHDUEVH6SLNNWO65NFPDUBLWQPANCNFSM5DYT45MQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[nrwiersma]

No stress if you dont have time. I have some time coming up, so I might take a stab at it in a week or so.

[mpetronic]

Thanks!

…

On Mon, Sep 13, 2021 at 12:31 PM Nicholas Wiersma ***@***.***> wrote: No stress if you dont have time. I have some time coming up, so I might take a stab at it in a week or so. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#115 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSTHDUXEQHRESR2NSS4Q4TUBYRPRANCNFSM5DYT45MQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.