Comments (13)
Hi there. For now you're right, we are currently only supporting the value column. The key will be supported in about 2 weeks. You'll come back here as soon as it's committed. Thanks for question.
from abris.
Hi there, pinging to let you know that the key column can now be preserved after consuming from Kafka. News about using it to send messages to Kafka are coming soon.
from abris.
Hi there, full support now provided for Avro serde on keys an values from Dataframes retrieved from Kafka.
https://github.com/AbsaOSS/ABRiS#writingreading-keys-and-values-as-avro-from-kafka
from abris.
Thanks. Appreciate it a lot.
One more question. As far as I'm able to read nofluent avro value+plain key. Is it possible to write confluent avro record in such way?
from abris.
Do you mean having plain key and Avro payload in Confluent format?
from abris.
from abris.
Not yet, but since you're asking about it we now have a use case. Will add and let you know. Thanks for help.
from abris.
Hi @pateusz , pinging to let you know about the plain key feature. You can check more about it here: https://github.com/AbsaOSS/ABRiS#writingreading-values-as-avro-and-plain-keys-as-string-tofrom-kafka
Regards.
from abris.
Rather than make new methods for each combination of key+value type (for example, say I have avro keys and integer values), what about a UDF that you can just at-will do like
df.select(from_confluent_avro(col("key")), col("value").cast("int"))
Similar to the existing from_json
function
from abris.
Hi @Cricket007 , sorry for the late reply. Yep, that is definitely coming since Spark itself had these methods added. We'll be updating the whole API to comply with that standard.
from abris.
@felipemmelo Would you like me to create a new issue for tracking that request?
from abris.
Hi @Cricket007 , if you can, please, so that we keep it documented. Thanks a lot!
from abris.
@felipemmelo Done! #16
from abris.
Related Issues (20)
- from_avro converts `\uFFFD` to a question mark HOT 1
- schema registry being called with http instead of https HOT 2
- Improve code-coverage & add GH check action HOT 1
- Fix JaCoCo CI for PRs from forked repos HOT 1
- update madrapps/jacoco-report
- Detect different schema versions in batch HOT 5
- Revert pull_request action back HOT 3
- TopicNameStrategy issue HOT 1
- Split GitHub actions for tests and test coverage
- Multiple schemas in one topic example HOT 1
- Spark 3.4 Support HOT 13
- malformed records to topic HOT 2
- foreach batch download by schem id HOT 3
- Container exited with a non-zero exit code 137 | Out of memory HOT 5
- Issues running inside Scala notebook on databricks HOT 1
- Fix tests for Spark 3.5.0
- Fix NoSuchMethodException in Spark 3.5.x
- get key from avro message HOT 3
- Compatibility with Spark 3.5 HOT 3
- Version 6.4.0 failing for Spark 3.5.0 HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from abris.