Comments (3)
(updated: talked to Chris in slack)
from pypeflow.
Progress is on the blocking
branch:
from pypeflow.
MJ, the multiprocessing
module is usually unwise, for a variety of reasons. I used to like it, but I've learned. There are very few cases when it is actually a good solution, except for something quick-and-dirty. I don't have time to give you the full lecture right now.
In my Python implementation last night, I used a thread per call. That's not ideal, but I think it's ok since we already limit concurrency from the workflow side. (Different FALCON stages have different max_concurrency
.)
(In my opinion, GoLang is a much better choice than Python for this, since each blocking call could be in its own goroutine, which is cheap. The caveats are suddenly far fewer. But this is a tiny amount of code, only a few hundred lines, so we can re-write in a couple hours down the road.)
The main problem I have with my current version is that termination is a bit tricky. We could share a repo with my little test-case, and you could experiment with different cases if you want. Ctrl-C at least seems to do the right thing, though I haven't tried it yet with a real qsub job.
The other difficulty is that this blocking "pwatcher" is ipso facto not actually restartable. So it's not really "watching" anything. That's fine, but we have to change slightly how this thing is called. Two choices:
- Currently, the
fs_based.py
pwatcher is in-process with FALCON. If we wantblocking.py
also in-process, then we need to stop using theprocess_watcher
context-manager repeatedly (or we would need global state). - Alternatively,
blocking.py
could be run as a separate process, either using IPC or using repeated socket reconnections. But then we have to consider how to kill it.
(Talked with MJ in Slack.)
from pypeflow.
Related Issues (20)
- Ensure that directories exist before qsub HOT 5
- FALCON qsub issues (part II) HOT 6
- job_start.py must be "executable" HOT 1
- Problem with network_based pwatcher on OSX HOT 20
- network_based slow for job_type=local HOT 4
- Bad argument to sbatch with SLURM HOT 8
- Heartbeat files not being created HOT 5
- Job completion not being detected HOT 12
- gc3pie pwatcher adaptor
- Update out of date example in pypeFLOW/example/PypeTest.py HOT 1
- Consider running autopep8 to improve readablity
- Not able to detect any error HOT 5
- newly created directories not always available fro remote job HOT 1
- add support for SLURM job dependencies to pypeFLOW HOT 5
- No git tags for each version HOT 3
- Problem with HPC pro and Falcon HOT 4
- PBS needs jobid for qdel HOT 7
- qsub jobs submission error HOT 1
- slurm command does not use MB, NPROC from config file
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pypeflow.