nellore / rail Goto Github PK

"ideas for counters we'd like to have (a) how many reads/readlets align, (b) number of introns called, (c) number of introns & nucleotides that go into intron index"

Add specification of expected precision and recall on dataset as command-line parameters of line.py and return failure of they don't agree with test output; this way, changes to the underlying alignment algorithms must be "approved" by editing .travis.yml.

While trying to run the Drosophila example from the Tutorial, I've noticed that providing the relative paths to the bowtie indices throws a runtime error. I'm calling the following command from an empty directory railtests, and the Drosophila_melanogaster directory containing the bowtie indices is one level above railtests:

[root@ip-10-5-163-113 railtests]# rail-rna go local -x ../Drosophila_melanogaster/UCSC/dm3/Sequence/BowtieIndex/genome ../Drosophila_melanogaster/UCSC/dm3/Sequence/Bowtie2Index/genome -m https://raw.githubusercontent.com/nellore/rail/master/ex/dm3_example.manifest

The runtime error which can be found in /rail-rna_logs/precoverage/dp.reduce.log/0.0.log:

Traceback (most recent call last):
  File "app_main.py", line 75, in run_toplevel
  File "/home/ec2-user/raildotbio/rail-rna/rna/steps/coverage_pre.py", line 130, in <module>
    os.path.expandvars(args.bowtie_idx)
  File "/home/ec2-user/raildotbio/rail-rna/rna/utils/bowtie_index.py", line 27, in __init__
    raise RuntimeError('No Bowtie index files with prefix "%s"' % idx_prefix)
RuntimeError: No Bowtie index files with prefix "../Drosophila_melanogaster/UCSC/dm3/Sequence/BowtieIndex/genome"

While the issue can be fixed simply by providing the full paths to the bowtie indices, I feel it would be best to support relative paths in the future, or to note this requirement in the documentation for now.

Hi,

To be consistent with recount, it'll be great if Rail-RNA includes in counts.tsv.gz the AUC for each sample.

Best,
Leo

Sometimes I am running the tool in elastic mode just to get a json file (with --json) and not to actually start a job. In that case, I'd like to suppress the uploading of the software to S3, which can be slow over a slow connection.

This Python module seems to be a reasonably easy-to-use and portable way to slurp a stream of reads from the SRA: https://github.com/ncbi/ngs/tree/master/ngs-python.

It depends on a library that they will dynamically download and install in ~/.ncbi upon first use of the LibManager module:

https://github.com/ncbi/ngs/blob/master/ngs-python/ngs/LibManager.py

There's many-second delay when this happens. Overall, I don't see anything terribly wrong with that way of doing things.

It's written by US govt employees and as such isn't copyrighted. I'm pretty sure, but not 100% positive, that makes it OK to package in an open source project.

Final note: I thought that DNANexus mirrored the SRA, but it seems they only mirror the SRA metadata. So this library is only for accessing reads at NCBI, and there is no mirror that I know of.

Hi,

I believe that my manifest file is correct. That is, I have:

<local path pair1><tab><0><tab><local path pair2><tab><0><tab><sample name>

But when I run the rail prep local step, I get errors like this:

Rail-RNA v0.1.9c
Loading...
Traceback (most recent call last):
  File "/jhpce/shared/community/compiler/gcc/4.4.7/python/2.7.9/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/jhpce/shared/community/compiler/gcc/4.4.7/python/2.7.9/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/bst/student/lcollado/raildotbio/rail-rna/__main__.py", line 725, in <module>
    scratch=args.scratch
  File "/home/bst/student/lcollado/raildotbio/rail-rna/rna/driver/rna_config.py", line 4610, in __init__
    raise_runtime_error(base)
  File "/home/bst/student/lcollado/raildotbio/rail-rna/rna/driver/rna_config.py", line 1189, in raise_runtime_error
    ) if len(bases.errors) > 1 else bases.errors[0]
RuntimeError: 1) The following line from the manifest file /dcs01/ajaffe/Brain/derRuns/derSupplement/simulation/rail/rail-manifest.txt has an invalid sample label:
/dcs01/ajaffe/Brain/derRuns/derSupplement/simulation/simulated_reads/sample_01_1.fasta.gz   0   /dcs01/ajaffe/Brain/derRuns/derSupplement/simulation/simulated_reads/sample_01_2.fasta.gz   0   sample1G1R1
A valid sample label takes the following form:
<Group ID>-<BioRep ID>-<TechRep ID>
2) The following line from the manifest file /dcs01/ajaffe/Brain/derRuns/derSupplement/simulation/rail/rail-manifest.txt has an invalid sample label:
/dcs01/ajaffe/Brain/derRuns/derSupplement/simulation/simulated_reads/sample_02_1.fasta.gz   0   /dcs01/ajaffe/Brain/derRuns/derSupplement/simulation/simulated_reads/sample_02_2.fasta.gz   0   sample2G1R1
A valid sample label takes the following form:
<Group ID>-<BioRep ID>-<TechRep ID>

From the error I can see that rail is thinking that I'm on single-end mode instead of paired-end. Also, I don't think that my manifest file is different from https://github.com/nellore/rail/blob/master/eval/GEUVADIS_28_local.manifest so I'm not sure what's wrong.

Details

manifest file which gets created by this script
script for local rail currently failing at rail prep local step.

I'm using v0.1.9c which is the latest non-devel version. Should I switch to the devel one?

In the coverage_bigwigs hundreds of bw chromosome files are generated, provide an option to report only final bw file.

I am trying to run make_it_rail.sh per the manual (http://docs.rail.bio/installation/) but it needs [email protected]'s password:

$ sh make_it_rail.sh
  adding: __main__.py (deflated 89%)
  adding: cloudformation/ (stored 0%)
  adding: cloudformation/dbgap.template (deflated 80%)
  (snip)
  adding: rna/utils/tempdel.py (deflated 55%)
  adding: version.py (deflated 20%)
Installer created at /Users/langmead/git/rail/releases/install_rail-rna-devel .
Copying to webfactional... .
[email protected]'s password:

https://github.com/nellore/rail/blob/master/src/rna/steps/preprocess.py#L656
https://github.com/nellore/rail/blob/master/src/rna/steps/preprocess.py#L737

Hi,

To be consistent with recount, it'll be great if Rail-RNA uses the AUC as provided in #47 for normalizing the samples before calculating the mean bigwig. In recount we are normalizing to a 40 million 100 bp SE reads library using the AUC.

Best,
Leo

https://github.com/russss/Herd

I recently tried a local rail-RNA run on one of our compute nodes here. I specified "-r 1000000" for the sort-memory-cap and got the following error:

Loading...
Detected Bowtie 1 v1.1.1, Bowtie 2 v2.2.5, and SAMTools v1.2.1.
Launching Dooplicity runner with PyPy...
ningal.cluster.lifesci.dundee.ac.uk > tail -f rail-rna.e1288159
Detected Bowtie 1 v1.1.1, Bowtie 2 v2.2.5, and SAMTools v1.2.1.
Launching Dooplicity runner with PyPy...
usage: emr_simulator.py [-h] [-m MEMCAP] [-p NUM_PROCESSES] [-t MAX_ATTEMPTS]
[-s SEPARATOR] [-k] [--keep-last-output]
[--gzip-outputs] [--gzip-level GZIP_LEVEL] [--ipy]
[--ipcontroller-json IPCONTROLLER_JSON]
[--ipy-profile IPY_PROFILE] [--scratch SCRATCH]
[--common COMMON] [--sort SORT] [-b BRANDING]
[-j JSON_CONFIG] [-f] [-l LOG]
emr_simulator.py: error: argument -m/--memcap: invalid int value: '1000000000.0'

It looks like the command line input is being read as a float when an int is expected. Removing the -r parameter removes the issue.

python rail/releases/install_rail-rna-devel --no-dependencies
â Rail-RNA vdevel Installer
Rail-RNA can be installed for all users or just the current user.
* Install for all users? [y/n]: n
Traceback (most recent call last):.
File "/cm/shared/apps/python/2.7.9/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/cm/shared/apps/python/2.7.9/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "rail/releases/install_rail-rna-devel/main.py", line 90, in
File "/home-1/[email protected]/rail/releases/install_rail-rna-devel/rna/driver/rna_installer.py", line 381, in install
File "/cm/shared/apps/python/2.7.9/lib/python2.7/contextlib.py", line 17, in enter
return self.gen.next()
File "rail/releases/install_rail-rna-devel/dooplicity/tools.py", line 63, in cd
OSError: [Errno 2] No such file or directory: '/tmp/tmpxPjmQ4/samtools-1.2'

Not sure if this is a bug or specific for my use case.

When running rail in parallel mode using ipcluster with Slurm I get a RuntimeError that isofrags.tar.gz does not exist. If I restart from that point everything finishes cleanly.

If I run rail in parallel on a single node with ipcluster (i.e. local instead of slurm) everything runs cleanly.

I am guessing it has something to do with using slurm. Probably not your problem, only bring it up because there is a mention of this in a commit log on the parallel branch. Please let me know if you have a known fix or a suggestion what might be going on.

Thanks
Justin

Load balance would be improved if task were assigned according to key % task_count

Currently the prep_merge.R and prep_sample.R are a laborious second step. I would love to have the rse_exon.Rdata and rse_gene.Rdata ect, provided at the end of the rail run.

next_report_line & report_multiplier unresolved:

https://github.com/nellore/rail/blob/master/src/rna/steps/align_reads_delegate.py#L173

E.g., this output for pypy --version results in Rail-RNA complaining I don't have the right version:

$ pypy --version
Python 2.7.10 (5f8302b8bf9f53056e40426f10c72151564e5b19, Feb 11 2016, 20:38:54)
[PyPy 4.0.1 with GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)]

Because the check is too restrictive:

If there is a series of Rail-RNA steps (alignment-related ones, probably) that achieve good CPU utilization and load balance, then high-cpu instances are cost effective & well suited. If a series of steps is not like this (poor load balance, mostly I/O), then high-cpu instances probably a waste of money, as most CPUs are idle.

If we partition the pipeline into stretches that either do or don't have these properties, then we could launch separate clusters (with different instance types) for those stretches. This could reduce costs relative to a pipeline that runs end-to-end on a big high-cpu cluster.

Downside: we pay the cost of bootstrapping multiple times for a given dataset. But we might also be able to simplify the bootstrapping for any given cluster, since a given cluster is running only a portion of the overall pipeline. E.g. if there's no alignment involved then you don't have to download and install the index. If samtools isn't involve, you don't have to install samtools.

Thanks to elasticity, there's no reason this would have to come at the expense of overall throughput.

should be super-easy -- just read the introns and insert them as a distinct sample before intron_config

In addition to being cool (and downloading a bunch of stuff automatically), might expand user base?

Cheers!

Ben

This might be a problem with just my machine, but I just did a fresh install of Ubuntu onto a new SSD so it's possible others are experiencing this as well.

I tried to install Rail-RNA using the downloaded installer as well as the command line script and run into this error with both. It seems to occur right after all the dependencies are downloaded and unpacked:

Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/var/tmp/e5618b82c6dd032102ed3aa1d2c2e502/main.py", line 90, in
File "/var/tmp/e5618b82c6dd032102ed3aa1d2c2e502/rna/driver/rna_installer.py", line 416, in install
File "/usr/lib/python2.7/subprocess.py", line 212, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 390, in init
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1025, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

I've tried to do some Googling on this error and it looks like a potential Python 2 problem. I've tried reinstalling the most recent Python 2.7 which hasn't helped. Any ideas on what I might be doing wrong?

Thank you!

David

Scenario: I have a bucket blah and my EMR job is set to output to a subdirectory of blah. When I try to run Rail starting from json:

cat hg19.json | python src/dooplicity/emr_runner.py --region us-east-1

I get an error:

RuntimeError: Bucket `blah` already exists on S3. Change affected output directories in job flow and try again. The more distinctive the name chosen, the less likely it's already a bucket on S3. It may be easier to create a bucket first using the web interface and use its name + a subdirectory as the output directory.

I don't know why the bucket already existing should pose a problem here.

http://res.illumina.com/documents/products/whitepapers/whitepaper_datacompression.pdf

I find that when I create a new bucket from the AWS CLI, it doesn't have lifecycle enabled by default. This causes an error here:

But seems like it shouldn't. @nellore commented that he things the check should be a not in errors rather than an in errors, but I defer on what the best solution is.

In local mode, L1750 of emr_simulator.py creates a temporary directory in /tmp when it should do this in the scratch directory specified by the user

• Resolve multimapping reads in a way that makes the most sense for downstream analyses. (How? Push coverage distribution toward more uniformity like MMR?)
• Add option to run job flow with Spark on EMR
• Permit automated sample clustering for collective analysis and report clusters?
• Add more normalization methods
• Play even nicer with derfinder; compute region matrix?
• Investigate --bind-to-socket in MPI to constrain processor usage on cluster
• Code robustness to losing and gaining ipengines in emr_simulator.py

https://github.com/nellore/rail/blob/master/src/rna/steps/preprocess.py#L559 -- Unresolved reference 'k'

Looks like output_dir and mover aren't defined before being used in the stream_and_upload function:

Since people may do multiple large, expensive runs with Rail, like we do, it's nice to be able to tag the EMR cluster so that the various projects can be distinguished on the bill.