Comments (7)
@hhromic In general that is correct.
from avro.
Its not that you cannot generate the schema from objects, but that the schema is the source of truth for encoding and decoding data in Avro. There are of course use cases where the objects make up a central definition of data exchange (a good example of this is Kubernetes) but this is not the general use case. And even in these cases, the object is only used when working in the same language as API Server. For example, were Kubernetes to support Avro in its API, the object is only useful in Go, in any other language I must depend instead of the fixed Schema generated from those objects.
from avro.
What is the question/proposal here?
from avro.
I thinks this is a duplicate of #263.
@longbai , here is my advise after our own experience.
We tried to think in this way in the past and realized this is not the correct way to think about it.
With Avro, you should always start with the schema first. Then, for convenience, you can generate Go structs from that schema so you can create typed objects to serialize/deserialize for that schema in particular. But this is technically optional as you can also serialize/deserialize to/from generic maps as well without Go structs.
If using structs, your program should always have a copy of the schema in the binary to use for serialization/deserialization (that is, you need a parsed schema object). If using generic maps, you can have the schemas external to the binary, e.g. from a schema registry or on disk and read them in runtime.
In our experience, an easy way to manage this in Go is to use the go embed tooling. In this way you can have the schema on disk and make sure the binary always contains the correct schema for which it was designed to work.
The biggest reason to embed the schema into the binary, is that it does not make much sense to change the schema and not the binary. If you change the schema, is highly likely that you have to change something in your program as well. In other words, Avro schemas are highly coupled to the program using them.
@nrwiersma please correct me if the above is still not the right way to think about it. This is the way we are using this great library with lot of success in our software components.
from avro.
In my current project, I chose to use Avro mainly because of its flexibility regarding schema generation.
I tend to disagree and consider Avro a friendly format when following a code-first approach compared to other formats such as Protocol buffer (because of the absence of tag numbers)
The Encoding and Evolution chapter in the Designing Data-intensive applications book (by Martin Kleppmann) highlights this point.
from avro.
👍 thanks @nrwiersma @hhromic
from avro.
Related Issues (20)
- Support tinygo HOT 6
- Are cross schema references supported? HOT 2
- PropertySchema properties are not JSON marshalled/unmarshalled HOT 5
- Referencing defined type throw error "unknown type" HOT 5
- Encoding Fixed Type In A Union
- AvroGen: generates duplicate structs due to a name resolution conflict HOT 7
- Schema with duplicate names should throw error
- Encoding to JSON does not reference reused types HOT 1
- Unmarshaling Interface Types in a Union HOT 6
- Reset the schema cache? HOT 4
- AVRO schema refs with Hamba avro HOT 8
- Support original name for structs
- Add protocol original schema string HOT 4
- Infinite loop parsing recursive schema HOT 1
- Encoding a struct with a sub-record field default value failed HOT 2
- avrogen: does not support logicalType: "uuid" in a union
- Support TextMarshaler/TextUnmarshaler for map keys HOT 2
- Bug for tree typed schema HOT 2
- [BUG] array schema cannot be correctly parsed HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from avro.