pb-assembly needs a mechanism for the user to control the environment when any task is executed. about pbbioconda

[armintoepfer]

Thank you, I forwarded it internally and we are looking into it.

[pb-cdunn]

We'll have a quick solution in the next pb-falcon update. You will set

export PYPEFLOW_PRE='source /home/user/myenv.sh'

E.g. that file can export anything you need in your remote environment.

The specified command will be prepended to task.sh and user_script.sh, which are generally run on the remote machines (unless you run "locally").

(Someday, this will be available via fc_run.cfg instead of via your shell environment, but that will require a larger change to FALCON/etc.)

Stay tuned for an update...

[bruc]

I'm a little confused -- will this option not work if the job_type is set to 'local'? For testing, it's valuable to separate the control flow from the queueing process.
Thanks.

[pb-cdunn]

It would work always, since it will get into the task.sh wrapper, which is used with any process-watcher configuration.

I don't know when we will update pb-falcon in bioconda. Maybe a week? Until then, you can pull the develop branch of pypeFLOW and try it.

[pb-cdunn]

Btw, we still rely on some environment variables being passed through qsub: PYPEFLOW_JOB_START_SCRIPT and PYPEFLOW_JOB_START_TIMEOUT. That's because we want to pass a pre-existing script to qsub, not a generated script. I've seen qsub (or equivalent) fail on some users' systems when a generated script is not yet available.

But I suspect your environment is being passed through qsub. I think your problem is that your installation is not on a network disk:

+ /bin/bash task.sh
/db/congenomics/local6/binaries/python-2.7.10/bin/python2.7: No module named pypeflow

local6? That sounds like a local disk.

If I'm right, then this new solution should work for you. The qsub script itself can be on a local disk because qsub actually copies it to the remote machine for us. E.g.

qsub -S /bin/bash -V foo.sh

In that case, /bin/bash obviously needs to exist on the remote machine, but foo.sh does not, since qsub will copy it. -V ensures that our PYPEFLOW_* environment variables will be available on the remote machine.

Does that make sense?

[pb-cdunn]

export PATH=/db/congenomics/local6/binaries/conda/bin:$PATH

If you're sure that's the PATH needed remotely, then I do not understand how job_start.sh is working for you. Possibly you are not using the default pwatcher_type=fs_based. We really need Issue submitters to be more clear about their configuration.

And if job_start.sh does not work for you, then we will have to think more deeply on how to satisfy both you and other users simultaneously, since filesystem latency is a real problem that other users have.

Job-distrubution is a very difficult problem. Satisfying the workflow graph is the only thing most authors of workflow systems seem to care about, but that is a trivial problem. I wish we could simply rely on a third-party job-distribution system which would satisfy all users, but I have not found something.

[bruc]

I'm somewhat confused by the above discussion -- all I need is the ability to include a command at the beginning of any script generated by Pypeflow. Fundamentally, it's a string substitution. File system latency is not the issue -- rather, I need to set the environment because our qsub commands doesn't pass it in, even though Pypeflow specifies the copying of the environment. (qsub -A doesn't work in our environment.)

BTW, "local" refers to local to our environment, like /usr/local.

[pb-cdunn]

export PYPEFLOW_PRE='source /my/bashrc'