install-spark with -x for 1.2.0.a does not set spark.default.parallelism correctly about emr-bootstrap-actions

[dunajevas]

https://github.com/awslabs/emr-bootstrap-actions/blob/master/spark/maximize-spark-default-config script does grep statements wrong for example
NUM_CORE_NODES=$(grep /mnt/var/lib/info/job-flow.json -e "CORE" -A 4 | grep -e requestedInstanceCount | cut -d':' -f2 | sed 's/\s\+//g')
does not work since Core is not in camel case, at least in ami 3.3.1

[jonchase]

This works on 3.3.1 for NUM_CORE_NODES:

"instanceGroupName": "core",
"instanceRole": "Core",
"marketType": "Spot",
"instanceType": "m3.2xlarge",
"requestedInstanceCount": 10,
"bidPrice": ".25"

grep /mnt/var/lib/info/job-flow.json -i -e "CORE" -A 4 | grep -e requestedInstanceCount | cut -d':' -f2 | sed 's/\s+//g' | cut -d',' -f1
10

[dunajevas]

that is strange.

[jonchase]

I started a new cluster and set the instanceGroupName for the core group to 'CORE' and that worked better, but it's still messed up. Then it grabs the trailing comma when getting the requestedInstanceCount (like "1,") - which is why I added the "cut -d',' -f1" to the end of my command.

[christopherbozeman]

Thank you for the report and discussion. This issue is addressed in pull #40. These changes to script maximize-spark-default-config are now live at s3://support.elasticmapreduce/spark/.

Changed the script use instance count from json file and to key off the slaveInstanceType property instead to avoid issues with names.

There is still room for more improvement such that this script will can be called later while running to pick up resizes.

[jonchase]

Changes look good to me. thx!!

[christopherbozeman]

jonchase - please close issue if resolved. Thanks!

[jonchase]

Looks like someone beat me to closing it. Works as expected now, thanks.