NoClassDefFoundError: org/apache/spark/SparkContext when starting Jetty about spark-jetty-server

[cjnolet]

I'll try and reproduce this if I can. What version of Hadoop? I'm assuming Spark 1.2?

[cjnolet]

nm, I see 2.4 and spark 1.2 in your comment above

[cjnolet]

I was able to reproduce this.

[cjnolet]

Let me know if you still have issues with this.

[kulkeez]

Yes, fixed. Able to start jetty but the Spring application is not reachable. I get the following errors:

[INFO] Configuring Jetty for project: war
[INFO] Webapp source directory = /home/tbda/spark-jetty-server/war/src/main/webapp
[INFO] web.xml file = /home/tbda/spark-jetty-server/war/src/main/webapp/WEB-INF/web.xml
[INFO] Classes = /home/tbda/spark-jetty-server/war/target/classes
[INFO] Property spark.home=/opt/spark-1.2.0-bin-hadoop2.4 was set
[INFO] Property spark.scheduler.allocation.file=/home/tbda/spark-jetty-server/war/src/main/resources/defaultPoolConfig.xml was set
[INFO] Property spark.ui.port=4040 was set
[INFO] Logging to org.slf4j.impl.SimpleLogger(org.mortbay.log) via org.mortbay.log.Slf4jLog
[INFO] Context path = /test
[INFO] Tmp directory = determined at runtime
[INFO] Web defaults = org/mortbay/jetty/webapp/webdefault.xml
[INFO] Web overrides = none
[INFO] Webapp directory = /home/tbda/spark-jetty-server/war/src/main/webapp
[INFO] Starting jetty 6.1.10 ...
[INFO] jetty-6.1.10
[INFO] No Transaction manager found - if your webapp requires one, please configure one.
[INFO] Initializing Spring root WebApplicationContext
[INFO] Root WebApplicationContext: initialization started
[INFO] Refreshing Root WebApplicationContext: startup date [Wed Jan 21 13:52:51 IST 2015]; root of context hierarchy
[INFO] Loading XML bean definitions from ServletContext resource [/WEB-INF/applicationContext.xml]
[INFO] Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@56c4c5fd: defining beans [sparkContext,sparkSqlContext]; root of factory hierarchy
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
[WARNING] Your hostname, localhost.localdomain resolves to a loopback address: 127.0.0.1; using 192.168.226.130 instead (on interface eth0)
[WARNING] Set SPARK_LOCAL_IP if you need to bind to another address
[INFO] Changing view acls to: tbda
[INFO] Changing modify acls to: tbda
[INFO] SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(tbda); users with modify permissions: Set(tbda)
[INFO] Slf4jLogger started
[INFO] Starting remoting
[INFO] Remoting started; listening on addresses :[akka.tcp://[email protected]:53230]
[INFO] Successfully started service 'sparkDriver' on port 53230.
[INFO] Registering MapOutputTracker
[INFO] Registering BlockManagerMaster
[INFO] Created local directory at /tmp/spark-local-20150121135253-8b6e
[INFO] MemoryStore started with capacity 224.6 MB
[WARNING] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[INFO] HTTP File server directory is /tmp/spark-a6f1b166-b599-4477-989a-4d4946b14f42
[INFO] Starting HTTP Server
[INFO] jetty-8.y.z-SNAPSHOT
[INFO] Started [email protected]:51578
[INFO] Successfully started service 'HTTP file server' on port 51578.
[INFO] jetty-8.y.z-SNAPSHOT
[INFO] Started [email protected]:4040
[INFO] Successfully started service 'SparkUI' on port 4040.
[INFO] Started SparkUI at http://192.168.226.130:4040
[INFO] Added JAR /home/tbda/spark-jetty-server/services/target/services-0.0.1-SNAPSHOT.jar at http://192.168.226.130:51578/jars/services-0.0.1-SNAPSHOT.jar with timestamp 1421828574525
[INFO] Created pool default, schedulingMode: FAIR, minShare: 2, weight: 1
[INFO] Connecting to ResourceManager at /0.0.0.0:8032
[INFO] Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
[INFO] Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
[INFO] Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
[INFO] Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

Does this application require Hadoop daemons (specifically YARN) to be started?

[cjnolet]

Yes it does and this is something I should probably add to the documentation. The purpose of this application is to submit a yarn job in Spark's yarn-client mode and expose the resulting SparkContext as a web service.

[kulkeez]

Thanks ! Have issues with my Hadoop environment and will get back on how I progressed.