Comments (4)
Thanks @Zejnilovic for reporting it
from spline-spark-agent.
The compile scope dependency tree for bundle-3.3:
![image](https://private-user-images.githubusercontent.com/795479/249571310-63f03ee9-8f00-44f6-8342-ce432a1ee22a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM4NjE2ODMsIm5iZiI6MTcyMzg2MTM4MywicGF0aCI6Ii83OTU0NzkvMjQ5NTcxMzEwLTYzZjAzZWU5LThmMDAtNDRmNi04MzQyLWNlNDMyYTFlZTIyYS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODE3JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgxN1QwMjIzMDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0yN2Y4MDY1YmY2YzU4MTc4Y2UwYjRhMDI0ODY5YTc5MTU2OThlMjgwNzY2OWY3NTBhN2FjYjlmNWQ1ZjRhNDRhJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.Tkhl3A6AzOvg1_fjrBRgAteUxWw8DITggIAisRq1Rys)
and for the bundle-2.4 for comparison:
![image](https://private-user-images.githubusercontent.com/795479/249571676-0f556c3b-3580-4ca3-9a04-e9a33c22d3b5.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM4NjE2ODMsIm5iZiI6MTcyMzg2MTM4MywicGF0aCI6Ii83OTU0NzkvMjQ5NTcxNjc2LTBmNTU2YzNiLTM1ODAtNGNhMy05YTA0LWU5YTMzYzIyZDNiNS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODE3JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgxN1QwMjIzMDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0wZDFkNWIzZmFiMjYyODIxZmJiYzhlZGU0OTgyY2Y2OWRmZGQ3MjM0ZWE1NDRjMjVkZDAyNzIxZjZlNjQwMTE5JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.ouQ3j9HJhcApS0p8JWFm7Bmz-QxkW6S0VsdAoOLVo8Y)
The only spotted difference - the commons-configuration:1.6
was missing in 2.4 but present in 3.3 due to the difference in Spark provided environment. This dependency is a part of Spline public API (appears in Plugin API), so it has to stay unshaded anyway.
The graph-core
contains packages scala.collection
and scalax
which makes it dangerous to relocate due to too broad pattern.
The absa-shaded
is already shaded under za.co.absa.shaded
The rest dependencies are to be relocated to za.co.absa.spline.shaded
from spline-spark-agent.
As a result of a further discussion with Adam we decided to leave scalaj.http
library unshaded, The reasons were:
- Even though it is not officially a part of Spline Agent public API, but many people's work could depend on it because they could develop extensions by coping code from Spline embedded plugins, modify it and make it theirs, and thus inherit some internal dependencies used in our internal plugins.
Scalaj.http
is among of them. Scalaj.http
has been stable for a long time and is now deprecated, so we expect really small collision probability on that library.
from spline-spark-agent.
We also discussed another option - in addition to every agent bundle jar produce another jar with the classifier shaded
(e.g spark-3.2-spline-agent-bundle_2.12-1.2.1-shaded.jar
). The difference would be that the original jar would have dependencies unshaded as before, but the new one would have them shaded. This would allow the users to decide if they want deps shaded or not depending on their use-case. The downside of this approach is that for majority of users it would be difficult to make a decision which jar to use, or their use-case might change, so the decision they make could be often wrong. So having two variants of jar publicly available could quickly lead to even more confusion around managing Spline dependencies. So in the end, we decided not to go down that route.
from spline-spark-agent.
Related Issues (20)
- Cant track org.apache.hadoop.fs.rename
- use splineAgent as a Jar file HOT 12
- Spline Agent required for spark 3.4 version and Scala 2.12 HOT 2
- Facing issues while sending Databricks Lineage (Spline) to kafka HOT 1
- Support for spark 3.5 HOT 1
- Does spline solution support snowpark
- Support of setting the arangoDB name on the configuration HOT 7
- Need to deploy on Synapse environment in order to track lineage for Synapse notebooks HOT 1
- Unity Catalog Support to get the Notebook details
- Support for AWS Glue DynamicFrame HOT 2
- Error when running spark application with agent-core as queryExecutionListeners in java 17
- Spline Support of Statment Create table as select HOT 11
- Using Spline in Spark Scala project HOT 1
- Can Spline support lineage for AWS glue Spark dynamic frames? HOT 1
- Databricks - Support of Delta live tables
- Spline doesn't track some in memory operations in pyspark HOT 5
- Customized Listener HOT 1
- Spline doesn't support MaxCompute table storage? HOT 1
- Incompatible version between Spline and Spark BigQuery connector HOT 3
- Spline Agent in AWS Glue 4.0 HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spline-spark-agent.