GithubHelp home page GithubHelp logo

Comments (12)

cerveada avatar cerveada commented on July 18, 2024

Hi, share the error, what jar are you uploading?

from spline-spark-agent.

zacayd avatar zacayd commented on July 18, 2024

HI
i have attached the screen shot of the installation of the jar
the jar is the artifact of build the project
https://github.com/AbsaOSS/spline-spark-agent
Screenshot 2023-11-01 130220

the errors are

command-4378807525288898:2: error: not found: value za
import za.co.absa.spline.harvester.SparkLineageInitializer._
^
command-4378807525288898:3: error: not found: value za
import za.co.absa.spline.agent.AgentConfig
^
command-4378807525288898:4: error: not found: value za
import za.co.absa.spline.harvester.postprocessing.AbstractPostProcessingFilter
^
command-4378807525288898:5: error: not found: value za
import za.co.absa.spline.harvester.postprocessing.PostProcessingFilter
^
command-4378807525288898:6: error: object configuration is not a member of package org.apache.commons
import org.apache.commons.configuration.Configuration
^
command-4378807525288898:7: error: not found: value za
import za.co.absa.spline.harvester.conf.StandardSplineConfigurationStack
^
command-4378807525288898:8: error: not found: value za
import za.co.absa.spline.harvester.HarvestingContext
^
command-4378807525288898:9: error: not found: value za
import za.co.absa.spline.producer.model.ExecutionPlan
^
command-4378807525288898:10: error: not found: value za
import za.co.absa.spline.producer.model.ExecutionEvent
^
command-4378807525288898:11: error: not found: value za
import za.co.absa.spline.producer.model.ReadOperation
^
command-4378807525288898:12: error: not found: value za
import za.co.absa.spline.producer.model.WriteOperation
^
command-4378807525288898:13: error: not found: value za
import za.co.absa.spline.producer.model.DataOperation
^
command-4378807525288898:14: error: not found: value za
import za.co.absa.spline.harvester.ExtraMetadataImplicits._
^
command-4378807525288898:15: error: not found: value za
import za.co.absa.spline.harvester.SparkLineageInitializer._
^

from spline-spark-agent.

cerveada avatar cerveada commented on July 18, 2024

If you are modifying the code, you must build and use your own spark-3.3-spline-agent-bundle.jar not the one from maven.

from spline-spark-agent.

wajda avatar wajda commented on July 18, 2024

error: not found: value za

It looks like the Jar is either not seen by the Spark, or is empty/does not contain any Spline related classes. Please check your Jar content and the way you attach it to Spark.

from spline-spark-agent.

zacayd avatar zacayd commented on July 18, 2024

the jar is 5K - it was created when i made a build on the Intelij
the file is called spline-spark-agent_2.12.jar
i took the content of the POM.xml from Bundle 3.3
I then uploaded the jar as "Install New" on the Databricks Cluster
Do you have a working solution of spline for Databricks which is not from Maven Central ?

from spline-spark-agent.

wajda avatar wajda commented on July 18, 2024

Please follow the building procedure from this section in the README - https://github.com/AbsaOSS/spline-spark-agent#building
Then take the bundle jar from the bundle-3.3/target folder.

from spline-spark-agent.

zacayd avatar zacayd commented on July 18, 2024

from spline-spark-agent.

wajda avatar wajda commented on July 18, 2024

I've never seen this error. Something might be wrong with your POM files. Try it on a clean project. call the Maven command in the project root directory.

Here's how it should look like:

 ~/Projects/spline-spark-agent │ develop                                                                                                                                                      
❯ mvn scala-cross-build:change-version -Pscala-2.11
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Spline Spark Agent                                                 [pom]
[INFO] agent-commons_2.12                                                 [jar]
[INFO] agent-core_2.12                                                    [jar]
[INFO] examples_2.12                                                      [jar]
[INFO] integration-tests_2.12                                             [jar]
[INFO] spark-2.2-spline-agent-bundle_2.11                                 [jar]
[INFO] spark-2.3-spline-agent-bundle_2.11                                 [jar]
[INFO] spark-2.4-spline-agent-bundle_2.12                                 [jar]
[INFO]
[INFO] -------< za.co.absa.spline.agent.spark:spline-spark-agent_2.12 >--------
[INFO] Building Spline Spark Agent 2.1.0-SNAPSHOT                         [1/8]
[INFO]   from pom.xml
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] --- scala-cross-build:0.2.2:change-version (default-cli) @ spline-spark-agent_2.12 ---
[INFO]
[INFO] ----------< za.co.absa.spline.agent.spark:agent-commons_2.12 >----------
[INFO] Building agent-commons_2.12 2.1.0-SNAPSHOT                         [2/8]
[INFO]   from commons/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- scala-cross-build:0.2.2:change-version (default-cli) @ agent-commons_2.12 ---
[INFO]
[INFO] -----------< za.co.absa.spline.agent.spark:agent-core_2.12 >------------
[INFO] Building agent-core_2.12 2.1.0-SNAPSHOT                            [3/8]
[INFO]   from core/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- scala-cross-build:0.2.2:change-version (default-cli) @ agent-core_2.12 ---
[INFO]
[INFO] ------------< za.co.absa.spline.agent.spark:examples_2.12 >-------------
[INFO] Building examples_2.12 2.1.0-SNAPSHOT                              [4/8]
[INFO]   from examples/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- scala-cross-build:0.2.2:change-version (default-cli) @ examples_2.12 ---
[INFO]
[INFO] --------< za.co.absa.spline.agent.spark:integration-tests_2.12 >--------
[INFO] Building integration-tests_2.12 2.1.0-SNAPSHOT                     [5/8]
[INFO]   from integration-tests/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- scala-cross-build:0.2.2:change-version (default-cli) @ integration-tests_2.12 ---
[INFO]
[INFO] --< za.co.absa.spline.agent.spark:spark-2.2-spline-agent-bundle_2.11 >--
[INFO] Building spark-2.2-spline-agent-bundle_2.11 2.1.0-SNAPSHOT         [6/8]
[INFO]   from bundle-2.2/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- scala-cross-build:0.2.2:change-version (default-cli) @ spark-2.2-spline-agent-bundle_2.11 ---
[INFO]
[INFO] --< za.co.absa.spline.agent.spark:spark-2.3-spline-agent-bundle_2.11 >--
[INFO] Building spark-2.3-spline-agent-bundle_2.11 2.1.0-SNAPSHOT         [7/8]
[INFO]   from bundle-2.3/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- scala-cross-build:0.2.2:change-version (default-cli) @ spark-2.3-spline-agent-bundle_2.11 ---
[INFO]
[INFO] --< za.co.absa.spline.agent.spark:spark-2.4-spline-agent-bundle_2.12 >--
[INFO] Building spark-2.4-spline-agent-bundle_2.12 2.1.0-SNAPSHOT         [8/8]
[INFO]   from bundle-2.4/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- scala-cross-build:0.2.2:change-version (default-cli) @ spark-2.4-spline-agent-bundle_2.12 ---
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spline Spark Agent 2.1.0-SNAPSHOT:
[INFO]
[INFO] Spline Spark Agent ................................. SUCCESS [  0.078 s]
[INFO] agent-commons_2.12 ................................. SUCCESS [  0.009 s]
[INFO] agent-core_2.12 .................................... SUCCESS [  0.009 s]
[INFO] examples_2.12 ...................................... SUCCESS [  0.009 s]
[INFO] integration-tests_2.12 ............................. SUCCESS [  0.009 s]
[INFO] spark-2.2-spline-agent-bundle_2.11 ................. SUCCESS [  0.007 s]
[INFO] spark-2.3-spline-agent-bundle_2.11 ................. SUCCESS [  0.008 s]
[INFO] spark-2.4-spline-agent-bundle_2.12 ................. SUCCESS [  0.009 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  0.279 s
[INFO] Finished at: 2023-11-06T10:58:25+01:00
[INFO] ------------------------------------------------------------------------

from spline-spark-agent.

zacayd avatar zacayd commented on July 18, 2024

from spline-spark-agent.

wajda avatar wajda commented on July 18, 2024

There seems to be a compilation error. Are you compiling the original source code, or you changed something in it?

from spline-spark-agent.

zacayd avatar zacayd commented on July 18, 2024

the origanl code

from spline-spark-agent.

wajda avatar wajda commented on July 18, 2024

Sorry I cannot reproduce it. It builds fine on all our environments including CI and personal PCs. Please try to build it on another env. It must work.

from spline-spark-agent.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.