EMR Step: Submit Spark JAR from S3 about emr-bootstrap-actions

[m1racoli]

First of all THX for your examples and your general efforts.

It is true that the spark-submit script retrieves the jar from S3, when the master is set to yarn-cluster.

Unfortunately I use some steps in Java code, which require AWS credentials and thus failes when running in cluster mode, since the credentials are not available from within the yarn containers (please correct me, if I am wrong).

My solution for now is to deploy the jar locally on the master node and run as yarn-client.

Best

[davidski]

👍

[christopherbozeman]

This can be done using the existing EMR framework and Spark's spark-submit. Detailed at https://forums.aws.amazon.com/thread.jspa?messageID=591612&#591612

Will add examples to this repo under spark/examples/.

[christopherbozeman]

Example of EMR step with Spark at https://github.com/awslabs/emr-bootstrap-actions/blob/master/spark/examples/spark-submit-via-step.md

[christopherbozeman]

@m1racoli - AWS credentials can be retrieved from within the yarn containers. If the cluster is running with EC2 role it can retrieve a temporary credential from EC2 metadata (http://docs.aws.amazon.com/AmazonS3/latest/dev/AuthUsingTempSessionTokenJava.html) or if using root/IAM account could just retrieve it from the Hadoop config (core-site.xml). The preferred method is using EC2 role and then requesting temporary credentials.