Comments (7)
SchemaManager
class should be available inside Abris jar, but it seems in your case s3://bucket/jars/abris.jar
doesn't contain it. I don't know why that would happen.
Can you manually open the jar and check the class is there?
from abris.
Hi @cerveada,
I extracted the code and I can see the class in above bath. Could the error be related to something else? Can this be because the library versions are not aligned?
Thank you
from abris.
Can you provide the whole exception with a stack trace? Or maybe the whole log of that run? You can upload files and it may help to find the issue here.
from abris.
Hi @cerveada,
Unfortunately, there is not much info in the stack trace:
INFO Client:
client token: N/A
diagnostics: User class threw exception: java.lang.NoClassDefFoundError: za/co/absa/abris/avro/read/confluent/SchemaManager$
at scala.collection.immutable.List.foreach(List.scala:431)
at scala.Function0.apply$mcV$sp(Function0.scala:39)
at scala.Function0.apply$mcV$sp$(Function0.scala:39)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
at scala.App.$anonfun$main$1$adapted(App.scala:80)
at scala.collection.immutable.List.foreach(List.scala:431)
at scala.App.main(App.scala:80)
at scala.App.main$(App.scala:78)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:740)
Caused by: java.lang.ClassNotFoundException: za.co.absa.abris.avro.read.confluent.SchemaManager$
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 23 more
from abris.
I opened the jar from maven https://mvnrepository.com/artifact/za.co.absa/abris_2.12/6.3.0
za/co/absa/abris/avro/read/confluent % ls
ConfluentConstants$.class
ConfluentConstants.class
SchemaManager.class
SchemaManagerFactory$$anonfun$$nestedInanonfun$getOrCreateRegistryClient$1$1.class
SchemaManagerFactory$.class
SchemaManagerFactory.class
There is no SchemaManager$
class. Only SchemaManager
. The $
is important here.
In Scala there can be class and object both having the same name. In Java this is not possible so when compiling the object names have the $
added at the end.
In Abris SchemaManager object was last present in version 3.2
https://github.com/AbsaOSS/ABRiS/blob/branch-3.2/src/main/scala/za/co/absa/abris/avro/read/confluent/SchemaManager.scala
In newer version it was removed.
So it looks like your code is using the SchemaManager object somewhere and that is causing the issue.
from abris.
Hi @cerveada,
Oh wow, I thought I saw it when I extracted the code. Hmm...
My code does indeed uses that class. Not sure why the class was removed. However, when I check the path under master branch I do see it:
Isn't that the same one? Or does it get excluded from the library?
Thank you
from abris.
As I was trying to explain in the previous message, there are two classes:
SchemaManager.class
for SchemaManager scala class
SchemaManager$.class
for SchemaManager scala object
The object was removed in Abris 4.
when I check the path under master branch I do see it:
https://github.com/AbsaOSS/ABRiS/blob/master/src/main/scala/za/co/absa/abris/avro/read/confluent/SchemaManager.scala
That is the scala class, but you can see that in that file there is no scala object defined.
This may explain better what object I am talking about:
https://docs.scala-lang.org/overviews/scala-book/companion-objects.html
from abris.
Related Issues (20)
- from_avro converts `\uFFFD` to a question mark HOT 1
- schema registry being called with http instead of https HOT 2
- Improve code-coverage & add GH check action HOT 1
- Fix JaCoCo CI for PRs from forked repos HOT 1
- update madrapps/jacoco-report
- Detect different schema versions in batch HOT 5
- Revert pull_request action back HOT 3
- TopicNameStrategy issue HOT 1
- Split GitHub actions for tests and test coverage
- Multiple schemas in one topic example HOT 1
- Spark 3.4 Support HOT 13
- malformed records to topic HOT 2
- foreach batch download by schem id HOT 3
- Container exited with a non-zero exit code 137 | Out of memory HOT 5
- Issues running inside Scala notebook on databricks HOT 1
- Fix tests for Spark 3.5.0
- Fix NoSuchMethodException in Spark 3.5.x
- get key from avro message HOT 3
- Compatibility with Spark 3.5 HOT 3
- Version 6.4.0 failing for Spark 3.5.0 HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from abris.