ipython kernel busy about initialization-actions

[nehalecky]

Hi @tuanvu216, I've also encountered the same issue, and it typically happens when one tries to open a second notebook. I haven't confirmed exactly why this is, however, I suspect that it's due to the way the IPython init action is configured to instantiate a SparkContext, and in particular the port settings.

See more, StackOverflow: Start multiple SparkContext Instances in Ipython. Are you able to check any logs?

[tuanavu]

Hi @nehalecky,

This is what I see in the Spark cluster:

As you can see the UNDEFINED indicates the notebook that has busy kernel. Checking it, this is what I see:

[karth295]

For posterity: you can only have a limited number of applications running concurrently -- specified as a fraction of the cluster size. See yarn.scheduler.capacity.maximum-applications and yarn.scheduler.capacity.maximum-am-resource-percent. We set max applications to 10000 (virtually infinite), and max percentage of app master resources to 25% of the cluster.

The latter means that the number of applications you can run concurrently is limited to the size of your cluster. You likely don't want to increase that percentage -- the other 75% is used for executors (or map/reduce containers when running MapReduce jobs).