Feature proposal: Launch ipcluster from python about ipyparallel

[jakirkham]

Also, in favor of this proposal, currently I find myself having to shell out the launch and shutdown the cluster. It would be great not to have to do that. Plus, am a big fan of being able to run multiple instances of the cluster at once.

[minrk]

Agreed. For a summary of the plans I have in my head:

• make ipcluster a service, rather than a script
• ipcluster would manage many instances of controllers and engine groups, not just one at a time
• talk to it via REST API
• add a Python wrapper for the service API, so you can start/stop clusters

This would alleviate a lot of the annoyances of ipcluster, I think.

In the meantime, there's ipython-cluster-helper, which provides much of what you describe, I think.

[twiecki]

Thanks. I think ipython-cluster-helper really only works with cluster schedulers, rather than locally.

[twiecki]

@minrk just wondering if perhaps there's a WIP version I could try.

[minrk]

Sorry, not yet.

[BitMechanic]

Hey @minrk, would it not work just to start a cluster from a multiprocess.process?

[BitMechanic]

So;

def worker():
      call("ipcluster start -n 4")

def main():

    initipy = multiprocessing.Process(target=worker)
    initipy.start()
  
    rc = ipp.Client()

if __name__ == "__main__":
    main()

[minrk]

Yes, that definitely works. It wouldn't be much more than that, though I would probably invoke Launches directly in process, rather than shelling out to ipcluster, which in turn shells out to ipengine, etc. I'm mainly thinking of a nice, clean API that allows automatic cleanup, etc., so you could do things like:

with ipp.cluster(n=4) as rc:
    view = rc.load_balanced_view()
...

[BitMechanic]

Is there a way to query the engines etc in python, so that you can check when an engine has finished booting up, what its currently doing? The booting up signal is particularly important from python, because we don't want someone using a python based GUI to try and use an engine that isn't yet there.

[minrk]

The client doesn't know that engines are booting up, so the only way to do it is to communicate (somehow) how many engines are coming and wait for them to register. The test suite has a utility that does this:

ipyparallel/ipyparallel/tests/__init__.py

Lines 78 to 108 in fbe5be5

	def add_engines(n=1, profile='iptest', total=False):
	"""add a number of engines to a given profile.
	If total is True, then already running engines are counted, and only
	the additional engines necessary (if any) are started.
	"""
	rc = Client(profile=profile)
	base = len(rc)
	if total:
	n = max(n - base, 0)
	eps = []
	for i in range(n):
	ep = TestProcessLauncher()
	ep.cmd_and_args = ipengine_cmd_argv + [
	'--profile=%s' % profile,
	'--InteractiveShell.colors=nocolor'
	]
	ep.start()
	launchers.append(ep)
	eps.append(ep)
	tic = time.time()
	while len(rc) < base+n:
	if any([ ep.poll() is not None for ep in eps ]):
	raise RuntimeError("A test engine failed to start.")
	elif time.time()-tic > 15:
	raise RuntimeError("Timeout waiting for engines to connect.")
	time.sleep(.1)
	rc.close()
	return eps

(my first use of embedded code snippets, yay!)

[BitMechanic]

Great, thanks @minrk