Create multi-threaded, blocking process-watcher about pypeflow

[mhsieh]

(updated: talked to Chris in slack)

[pb-cdunn]

Progress is on the blocking branch:

• https://github.com/PacificBiosciences/pypeFLOW/blob/blocking/pwatcher/blocking.py

[pb-cdunn]

MJ, the multiprocessing module is usually unwise, for a variety of reasons. I used to like it, but I've learned. There are very few cases when it is actually a good solution, except for something quick-and-dirty. I don't have time to give you the full lecture right now.

In my Python implementation last night, I used a thread per call. That's not ideal, but I think it's ok since we already limit concurrency from the workflow side. (Different FALCON stages have different max_concurrency.)

(In my opinion, GoLang is a much better choice than Python for this, since each blocking call could be in its own goroutine, which is cheap. The caveats are suddenly far fewer. But this is a tiny amount of code, only a few hundred lines, so we can re-write in a couple hours down the road.)

The main problem I have with my current version is that termination is a bit tricky. We could share a repo with my little test-case, and you could experiment with different cases if you want. Ctrl-C at least seems to do the right thing, though I haven't tried it yet with a real qsub job.

The other difficulty is that this blocking "pwatcher" is ipso facto not actually restartable. So it's not really "watching" anything. That's fine, but we have to change slightly how this thing is called. Two choices:

1. Currently, the fs_based.py pwatcher is in-process with FALCON. If we want blocking.py also in-process, then we need to stop using the process_watcher context-manager repeatedly (or we would need global state).
2. Alternatively, blocking.py could be run as a separate process, either using IPC or using repeated socket reconnections. But then we have to consider how to kill it.

(Talked with MJ in Slack.)