install-spark doesn't extract number of instances properly about emr-bootstrap-actions

[christopherbozeman]

Part of issue #38.

[christopherbozeman]

Derrick... is this still an issue?

[derrickburns]

I stopped using install-spark -x, but I will try again.

On Fri, Feb 13, 2015 at 2:07 PM, Christopher Bozeman <
[email protected]> wrote:

Derrick... is this still an issue?

—
Reply to this email directly or view it on GitHub
#43 (comment)
.

[derrickburns]

I tested your code, abc while it appears to set the Spark parameters correctly, I am NOT getting all the executors that are allocated. This is consistent behavior.

Sent from my iPhone

On Feb 13, 2015, at 2:07 PM, Christopher Bozeman [email protected] wrote:

Derrick... is this still an issue?

—
Reply to this email directly or view it on GitHub.

[christopherbozeman]

@derrickburns - Can I get a cluster id and application id to reference demonstrating the issue? Also, is there sufficient memory to fulfill all the executors requested when considering the amount of driver memory being requested?

[derrickburns]

Try this one: j-8YQLFX9G9PV5. There is only one application.

It appears to me that there are 11 non-driver executors running and 1
driver executor. However, I requested 13 core and 1 master. So I am
confused.

On Mon, Feb 16, 2015 at 9:34 AM, Christopher Bozeman <
[email protected]> wrote:

@derrickburns https://github.com/derrickburns - Can I get a cluster id
and application id to reference demonstrating the issue? Also, is there
sufficient memory to fulfill all the executors requested when considering
the amount of driver memory being requested?

—
Reply to this email directly or view it on GitHub
#43 (comment)
.

[derrickburns]

Also, the -x option only works in yarn-client mode.

On Mon, Feb 16, 2015 at 1:08 PM, Derrick Burns [email protected]
wrote:

Try this one: j-8YQLFX9G9PV5. There is only one application.

It appears to me that there are 11 non-driver executors running and 1
driver executor. However, I requested 13 core and 1 master. So I am
confused.

On Mon, Feb 16, 2015 at 9:34 AM, Christopher Bozeman <
[email protected]> wrote:

@derrickburns https://github.com/derrickburns - Can I get a cluster id
and application id to reference demonstrating the issue? Also, is there
sufficient memory to fulfill all the executors requested when considering
the amount of driver memory being requested?

—
Reply to this email directly or view it on GitHub
#43 (comment)
.

[derrickburns]

How do you specify the memory of the Application Master separate from the
memory for the driver when running in yarn-client mode?

On Mon, Feb 16, 2015 at 10:56 PM, Derrick Burns [email protected]
wrote:

Also, the -x option only works in yarn-client mode.

On Mon, Feb 16, 2015 at 1:08 PM, Derrick Burns [email protected]
wrote:

Try this one: j-8YQLFX9G9PV5. There is only one application.

It appears to me that there are 11 non-driver executors running and 1
driver executor. However, I requested 13 core and 1 master. So I am
confused.

On Mon, Feb 16, 2015 at 9:34 AM, Christopher Bozeman <
[email protected]> wrote:

@derrickburns https://github.com/derrickburns - Can I get a cluster
id and application id to reference demonstrating the issue? Also, is there
sufficient memory to fulfill all the executors requested when considering
the amount of driver memory being requested?

—
Reply to this email directly or view it on GitHub
#43 (comment)
.

[christopherbozeman]

In regards to specifying memory of the AM separate from driver, see spark.yarn.am.memory which is available in Spark 1.3.