concurrent.futures compatibility (IPEP 19) about ipyparallel

[minrk]

@aarchiba I really want this. I even started writing it a long time ago, but had to move on to other things. Hopefully I, or someone else, will get a chance to do it soon.

[aarchiba]

Well, since I kind of need it now and I just wrote an MPI-based Executor, maybe the thing to do is write a standalone wrapper module; it can go into ipyparallel when it's ready. I don't actually need the callback thing (yet? actually it would be nice), which is the only thing that wants modifications to ipyparallel.

[minrk]

Yup, a simple wrapper exposing the right API would be the place to start. If you are working on that, feel free to link to it here.

[aarchiba]

See https://github.com/aarchiba/futuretools . Needs more testing (e.g. workers appearing and disappearing, cleanliness of shutdown).

[aarchiba]

In terms of what would be needed to improve/simplify the wrapper, the main thing is that right now it has to poll - check the AsyncResults repeatedly to see if any have finished yet. Client.wait() almost does the right thing, except that I want to wait until any result is available - including one from a request that hadn't been submitted when I start waiting. But since Client.wait() has to poll anyway there's not too much to be gained here.

The other thing is that it's not so easy to tell when a node goes idle - queue_status turns out to require a round-trip to the node (and wait for the currently-executing job to complete) so that's no good. Right now I just keep track of how many jobs are in flight and how many workers I have and send more jobs if there are fewer jobs than workers.

concurrent.futures never interupts a Future that is in progress, so I don't actually use abort(). It probably makes sense to make the AsyncResult object available from the Future (beware race conditions; maybe just an abort method) so that users can do this if they want to.

[minrk]

That's awesome!

What I would do is probably subclass Client and hook up ZMQStreams on the zmq sockets and run a tornado IOLoop in a thread. That way, callbacks will fire as zmq messages arrive, rather than ever calling Client.wait. I'm sure that upon subclassing I would find a few methods to split up differently for re-use.

Are you interested in tackling that? This sounds pretty fun to me, but I'm not sure when I'll get a chance to put it together.

[aarchiba]

If tornado IOLoop gives you a poll-free way to wait on messages, then it should be used to implement Client.wait(). In fact the thing to do would be to generalize Client.wait() to permit waiting for any message, as well as all messages. I haven't used tornado or zmq directly at all, so it'd take me a little while to get up to speed. For the moment I need to do some tests to see how nestle (python nested sampling) behaves with larger numbers of cores.

[aarchiba]

I think it makes sense to create a PR to include this code. I'm having trouble getting ipyparallel working from git, though. What are its dependencies? pyzmq, obviously, but I'm getting complaints about a missing ipykernel module.

[minrk]

pip install -e . should get all the dependencies. I think it's just ipykernel, though.

[minrk]

@aarchiba do you think you may have a chance to work on this? I added some Future support in pyzmq recently, so now I have some ideas for how this might work.

[minrk]

I might take a stab at making Client's IO use IOLoop and Futures internally this weekend. That should clear much of the path for wrapping a view in an Executor interface.