Comments (11)
Fixed in 3.0.2
from abris.
I got the exact same error with 2_11, 3.0.1 when calling from_confluent_avro with a schema string.
Not sure if it's a bug or if I'm not using this correctly, but since this issue has been created, I'll hitch a ride on it
from abris.
Hi, thanks for repporting the error. Could any of you please provide more information? What schema were you using? How have you serialized the data?
from abris.
My schema is very complex. Not sure about serialization. Just reading from a Kafka topic.
I'm just investigating what happens for a simpler schema first before coming back with more details...
from abris.
Using from_confluent_avro, I have managed to read from a different Kafka topic with a different avro schema. This schema is much simpler than the schema that fails.
I amn't sure which serialization is being used, but I know that both the simple and complex schema are written in the same way.
Here is the complex schema that is failing
root
|-- from_avro(value): struct (nullable = true)
| |-- a: string (nullable = false)
| |-- b: string (nullable = false)
| |-- c: string (nullable = false)
| |-- d: long (nullable = false)
| |-- e: integer (nullable = false)
| |-- f: struct (nullable = false)
| | |-- g: struct (nullable = false)
| | | |-- value: double (nullable = true)
| | | |-- tsMs: long (nullable = false)
| | | |-- status: integer (nullable = false)
| | | |-- changed: boolean (nullable = false)
| | |-- h: struct (nullable = false)
| | | |-- value: boolean (nullable = true)
| | | |-- tsMs: long (nullable = false)
| | | |-- status: integer (nullable = false)
| | | |-- changed: boolean (nullable = false)
| | |-- i: struct (nullable = false)
| | | |-- value: string (nullable = true)
| | | |-- tsMs: long (nullable = false)
| | | |-- status: integer (nullable = false)
| | | |-- changed: boolean (nullable = false)
| | |-- j: struct (nullable = false)
| | | |-- value: boolean (nullable = true)
| | | |-- tsMs: long (nullable = false)
| | | |-- status: integer (nullable = false)
| | | |-- changed: boolean (nullable = false)
| | |-- k: struct (nullable = false)
| | | |-- value: integer (nullable = false)
| | | |-- tsMs: long (nullable = false)
| | | |-- status: integer (nullable = false)
| | | |-- changed: boolean (nullable = false)
| | |-- l: struct (nullable = false)
| | | |-- value: boolean (nullable = true)
| | | |-- tsMs: long (nullable = false)
| | | |-- status: integer (nullable = false)
| | | |-- changed: boolean (nullable = false)
| | |-- m: struct (nullable = false)
| | | |-- value: boolean (nullable = true)
| | | |-- tsMs: long (nullable = false)
| | | |-- status: integer (nullable = false)
| | | |-- changed: boolean (nullable = false)
| | |-- n: struct (nullable = false)
| | | |-- value: boolean (nullable = true)
| | | |-- tsMs: long (nullable = false)
| | | |-- status: integer (nullable = false)
| | | |-- changed: boolean (nullable = false)
| | |-- o: struct (nullable = false)
| | | |-- value: boolean (nullable = true)
| | | |-- tsMs: long (nullable = false)
| | | |-- status: integer (nullable = false)
| | | |-- changed: boolean (nullable = false)
| | |-- p: struct (nullable = false)
| | | |-- value: boolean (nullable = true)
| | | |-- tsMs: long (nullable = false)
| | | |-- status: integer (nullable = false)
| | | |-- changed: boolean (nullable = false)
| | |-- q: struct (nullable = false)
| | | |-- value: integer (nullable = true)
| | | |-- tsMs: long (nullable = false)
| | | |-- status: integer (nullable = false)
| | | |-- changed: boolean (nullable = false)
| | |-- r: struct (nullable = false)
| | | |-- value: integer (nullable = true)
| | | |-- tsMs: long (nullable = false)
| | | |-- status: integer (nullable = false)
| | | |-- changed: boolean (nullable = false)
| | |-- s: struct (nullable = false)
| | | |-- value: float (nullable = true)
| | | |-- tsMs: long (nullable = false)
| | | |-- status: integer (nullable = false)
| | | |-- changed: boolean (nullable = false)
| | |-- t: struct (nullable = false)
| | | |-- value: double (nullable = true)
| | | |-- tsMs: long (nullable = false)
| | | |-- status: integer (nullable = false)
| | | |-- changed: boolean (nullable = false)
| | |-- u: struct (nullable = false)
| | | |-- value: integer (nullable = true)
| | | |-- tsMs: long (nullable = false)
| | | |-- status: integer (nullable = false)
| | | |-- changed: boolean (nullable = false)
I can confirm that the kafka topic containing the complex schema is being read by a microservice without any problems, so the data itself is not corrupt.
from abris.
Also, we are using Confluent's schema registry for encoding, however I am still using from_confluent_avro with a jsonSchema string derived by directly reading and parsing the avsc file. In other words, the jsonSchema I am using does not include any additional information that Confluent adds.
At the moment I am still doing the proof of concept and as the schema registry is stuck behind some layers of security, I am passing in a schema string instead to make things easier
However, as I was able to parse data from the kafka topic with the simpler schema, I don't believe that this is a problem
from abris.
we use confluent schema registration
our code originally run on databricks notebook and we use databricks avro.functions.from_avro.
now we migrate the code off databricks and replace that part with ABRiS from_confluent_avro with schema registry signature, same producer , topic and messages and got this error
from abris.
I've just seen on the wiki that abris does not support batch processing which is what I am trying to do (but I did get it to work for the simple schema!)
Interested to know if that could be the problem? Even if it is, heng2018 still has a problem here.
from abris.
I've just seen on the wiki that abris does not support batch processing which is what I am trying to do (but I did get it to work for the simple schema!)
The WIKI.wiki
file is outdated. It is about the old API from Abris 2.x.x. The current API should work anywhere where you can do select.
Good news is that I was able to simulate this exception. The fix will be part of next release.
from abris.
great news!
when will the the next release?
from abris.
Most probably this week.
from abris.
Related Issues (20)
- from_avro converts `\uFFFD` to a question mark HOT 1
- schema registry being called with http instead of https HOT 2
- Improve code-coverage & add GH check action HOT 1
- Fix JaCoCo CI for PRs from forked repos HOT 1
- update madrapps/jacoco-report
- Detect different schema versions in batch HOT 5
- Revert pull_request action back HOT 3
- TopicNameStrategy issue HOT 1
- Split GitHub actions for tests and test coverage
- Multiple schemas in one topic example HOT 1
- Spark 3.4 Support HOT 13
- malformed records to topic HOT 2
- foreach batch download by schem id HOT 3
- Container exited with a non-zero exit code 137 | Out of memory HOT 5
- Issues running inside Scala notebook on databricks HOT 1
- Fix tests for Spark 3.5.0
- Fix NoSuchMethodException in Spark 3.5.x
- get key from avro message HOT 3
- Compatibility with Spark 3.5 HOT 3
- Version 6.4.0 failing for Spark 3.5.0 HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from abris.