Comments (11)
@felipemmelo could you please have a look at this?
from abris.
Hi there, sure thing.
It usually happens (to the best of my experience) in one of two situations.
- Schema does not match the payload
- The payload is being produced by a Confluent-compliant producer - which includes the id of the schema to the message - and consumed by a standard Avro parser.
How is your message being produced? Is there any chance your schema does not match the data?
from abris.
Hi @felipemmelo , thanks for your advise.
I checked the schema and the payload found they are matched.
Message is produced by go client https://github.com/Shopify/sarama.
Manually consume from topic:
bin/kafka-avro-console-consumer --zookeeper localhost:2181 --topic VIP_E3AVAIcYWzLHS5YM_0000000000 --from-beginning
{
"uri": {
"string": "/js/vendor/modernizr-2.8.3.min.js"
},
"protocol": {
"string": "HTTP/1.1"
},
"code": {
"long": 200
},
"ip": {
"string": "10.128.2.1"
},
"timestamp": {
"string": "29/Jan/2018:20:29:23"
},
"method": {
"string": "GET"
},
"raw_text": {
"string": "\"10.128.2.1\",\"[29/Jan/2018:20:29:23\",\"GET /js/vendor/modernizr-2.8.3.min.js HTTP/1.1\",\"200\""
},
"_appId": "1380542074",
"_repo": "weblog",
"weblog_1380542074_timestamp": "2018-08-16T10:08:07.469125479+08:00"
}
Schema:
{
"type": "record",
"name": "VIP_E3AVAIcYWzLHS5YM_0000000000",
"fields": [{
"name": "uri",
"type": ["null", "string"]
}, {
"name": "protocol",
"type": ["null", "string"]
}, {
"name": "code",
"type": ["null", "long"]
}, {
"name": "ip",
"type": ["null", "string"]
}, {
"name": "timestamp",
"type": ["null", "string"]
}, {
"name": "method",
"type": ["null", "string"]
}, {
"name": "raw_text",
"type": ["null", "string"]
}, {
"name": "_appId",
"type": "string"
}, {
"name": "_repo",
"type": "string"
}, {
"name": "weblog_1380542074_timestamp",
"type": "string"
}]
}
Is there anything wrong with my usage?
from abris.
Hi @OopsOutOfMemory , thanks for the reply.
The usage seems to be alright. It may be related to how Sarama is generating the Avro record. I'll look into that library and try to replicate the error. Would there be any open/public topic I could connect to in order to test it?
Also, could you let me know which naming strategy you're using for Schema Registry? Currently ABRiS only supports topic subject strategy.
Finally, is bin/kafka-avro-console-consumer
also provided by Sarama?
Thanks in advance.
from abris.
It usually happens (to the best of my experience) in one of two situations.
...
2. The payload is being produced by a Confluent-compliant producer - which includes the id of the schema to the message - and consumed by a standard Avro parser.
In this case where we are using a Confluent compliant producer, can you confirm for me how to get around this issue?
from abris.
It usually happens (to the best of my experience) in one of two situations.
...
2. The payload is being produced by a Confluent-compliant producer - which includes the id of the schema to the message - and consumed by a standard Avro parser.In this case where we are using a Confluent compliant producer, can you confirm for me how to get around this issue?
Hi ... did you get the resolution of this issue. .. I am facing the same issue in my code, my schema is being generated with 'id' field which is causing outofbound error.
Can you please let me know the resolution if any?
from abris.
Hi @NitinRamola ,
Which API are you using fromAvro
or fromConfluentAvro
? If you have the id attached to the top of the payload, the latter should be used.
from abris.
Hi @NitinRamola ,
Which API are you using
fromAvro
orfromConfluentAvro
? If you have the id attached to the top of the payload, the latter should be used.
Hi Felipemmelo, Thanks for your suggestion, yes i m using fromAvro right now, let me check with fromConfluentAvro and will reply back to you...
from abris.
Hi @NitinRamola ,
Which API are you usingfromAvro
orfromConfluentAvro
? If you have the id attached to the top of the payload, the latter should be used.Hi Felipemmelo, Thanks for your suggestion, yes i m using fromAvro right now, let me check with fromConfluentAvro and will reply back to you...
HI Felipemmelo, I can't use 3rd party packages currently although your solution looks good, is there any official API suggested to resolve this error ?
from abris.
Hi @NitinRamola , not that I know, this is why we've developed this one. You can try to use KafkaStreams which is also provided by Confluent, or, if you don't want to use Confluent-compliant payload (i.e. schema id on top of payload), there is a built-in Avro module in Spark 2.4.
from abris.
It usually happens (to the best of my experience) in one of two situations.
...
2. The payload is being produced by a Confluent-compliant producer - which includes the id of the schema to the message - and consumed by a standard Avro parser.In this case where we are using a Confluent compliant producer, can you confirm for me how to get around this issue?
Hi ... did you get the resolution of this issue. .. I am facing the same issue in my code, my schema is being generated with 'id' field which is causing outofbound error.
Can you please let me know the resolution if any?
Sorry, genuinely can't remember.
from abris.
Related Issues (20)
- from_avro converts `\uFFFD` to a question mark HOT 1
- schema registry being called with http instead of https HOT 2
- Improve code-coverage & add GH check action HOT 1
- Fix JaCoCo CI for PRs from forked repos HOT 1
- update madrapps/jacoco-report
- Detect different schema versions in batch HOT 5
- Revert pull_request action back HOT 3
- TopicNameStrategy issue HOT 1
- Split GitHub actions for tests and test coverage
- Multiple schemas in one topic example HOT 1
- Spark 3.4 Support HOT 13
- malformed records to topic HOT 2
- foreach batch download by schem id HOT 3
- Container exited with a non-zero exit code 137 | Out of memory HOT 5
- Issues running inside Scala notebook on databricks HOT 1
- Fix tests for Spark 3.5.0
- Fix NoSuchMethodException in Spark 3.5.x
- get key from avro message HOT 3
- Compatibility with Spark 3.5 HOT 3
- Version 6.4.0 failing for Spark 3.5.0 HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from abris.