Package for saagie : sbt clean assembly and get the package in target.
Usage in local :
- sbt clean assembly
- spark-submit --class=io.saagie.example.hdfs.Main example-spark-scala-read-and-write-from-hive-assembly-1.0.jar --hdfsHost "hdfs://hdfshost:8020/"
Usage in Saagie :
- sbt clean assembly (in local, to generate jar file)
- create new Spark Job
- upload the jar (target/scala-2.10/example-spark-scala-read-and-write-from-hive-assembly-1.0.jar)
- Replace MyClass with the full class name (ex : io.saagie.example.hive.Main)
- copy URL from HDFS connection details panel and put it as argument after --hdfsHost
- choose Spark 1.6.1
- choose resources you want to allocate to the job
- create and launch.