Spark on YARN not working : Examples about emr-bootstrap-actions

[mkanchwala]

I am using Spark 1.3.0 with AMI 3.5.0 EMR and Scala 2.10.4

[christopherbozeman]

Just use the JavaKinesisWordCountASL example from the stock build as it now picks up the master dynamically.

Like:

MASTER=yarn-client run-example ...

Also, as for credentials use roles (http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-iam-roles-defaultroles.html) in order to easily pass creds to executors. I'll write up a doc on this more detail soon.

[mkanchwala]

Hi @christopherbozeman

The JavaKinesisWordCountASL is working fine, I am looking into more to run a Yarn Spark job

1. This is working fine :
   MASTER=yarn-client run-example org.apache.spark.examples.streaming.JavaKinesisWordCountASL mySparkStream https://kinesis.us-east-1.amazonaws.com

2. But still the same problem in this one :
   MASTER=yarn-client run-example org.apache.spark.examples.streaming.JavaKinesisWordCountASLYARN mySparkStream https://kinesis.us-east-1.amazonaws.com

Also I've provided Administrator Access and AmazonKinesisFullAccess on EMR_EC2_DefaultRole

Please let me know for more details.

Thanks

[christopherbozeman]

The JavaKinesisWordCountASLYARN example was added with 1.1.0 to provide an example of Streaming with Kinesis on YARN, but it has not kept pace with other changes to the stock examples of Spark. Thus, this example is no longer valid. The stock JavaKinesisWordCountASL now supports dynamic master selection so follow this example. I have removed the outdated example in the EMR build as can be seen with pull #83 in 1.3.0.d.

[mkanchwala]

Thanks for the Update @christopherbozeman.... Closing the Issue.