Comments (14)
I am experiencing the slowness at the moment. Do you know how long it should take for a 1-2 million reads?
from marginalign.
It could take a couple of hours, if the reads are long (~10 kb). I will fix a patch for this in a few days. Sorry for the hassle.
from marginalign.
Thank you. Actually it has been running for 16 hours already. Is there anyway to use multiple threads? I may need to devise a divide-and-conquer strategy.
from marginalign.
If I run it on 50-100 reads, it seems to suggest it will take 400 hours to finish 2 million reads. Does that sound correct?
from marginalign.
Are you doing both training and alignment on the reads?
Benedict
(from phone, forgive extra typos)
On Nov 9, 2016 9:07 AM, "John Urban" [email protected] wrote:
If I run it on 50-100 reads, it seems to suggest it will take 400 hours to
finish 2 million reads. Does that sound correct?—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#16 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAkOIIrmM_NTzpA1VEFFEr0GQyIaRZZ9ks5q8f3BgaJpZM4EEC_3
.
from marginalign.
For the big job I am running (2 million reads), I am running marginStats (to get pct identity values) on a set of nanopore reads previously aligned by BWA (independent of marginAlign). These reads are mapped to an ~300 Mb assembly.
For the 50-100 reads I checked the speed with, it was on lambda nanopore reads previously aligned with marginAlign. Of course, these reads are mapped to the 48.5 kb lambda genome.
So to answer your question -- I am past alignment/training stages -- just trying to use marginStats.
from marginalign.
400 hours seems odd, esp. if you are just running stats. I have done stats on ~125000 reads in around 5.5 hours (parallelizing on 32 cpu's).
There are ways we can speed this up further. I am out of town till tomorrow, but will look into this ASAP.
from marginalign.
But are you parallelizing by splitting up the SAM files or is there a parallelization option?
I do massively appreciate you looking into this and responding so quickly. Feel free to ignore this until you get back in town.
from marginalign.
Did you make any progress on this? Either way, can you let me know your strategy for parallelizing to 32 CPUs. Was it just breaking it up and executing into 32 separate BAM files? Is it a little more sophisticated than that? There is a potential learning moment to be had here.
from marginalign.
Sorry, haven't gotten to it yet.
There are two ways to parallelize, one requires using the jobTree framework (this is being upgraded now) where you can specify cpu and memory. The simplest way to do it is as you mentioned, with multiple SAMs and running using parallel (I have done that too in the past).
I will confirm once I have looked at the stats code for speeding it up.
from marginalign.
Did that help? I am going to push a possible fix later today/early tomorrow that should help with this.
Apologies for the delay.
from marginalign.
No worries. I appreciate your help. Thanks for clarifying your parallelization strategy. I will break the BAM up into a ton of smaller pieces. Also, I will wait to try your possible fix. Thanks for looking into this.
from marginalign.
Hello, is there an option for using multiple CPUs (threads) with marginStats ?
from marginalign.
The easiest way to speed it up is to break the samfiles and run in parallel. We are working on a newer design of the code that will parallelize this. This should be available very soon.
from marginalign.
Related Issues (20)
- Make test fails HOT 4
- make error HOT 1
- marginAlign & marginCaller errors HOT 4
- input fasta files HOT 4
- marginAlign: command not found HOT 4
- marginAlign error, jobTree HOT 3
- make test fail HOT 2
- marginCaller: very long Cigar causes "Argument list too long" error HOT 2
- Error: sees to be looking for enriron.pickle at the wrong location HOT 3
- No space left on device HOT 2
- RuntimeError: Got failed jobs HOT 5
- installation error HOT 3
- Job: ~/marginAlign/jobTree/jobs/t0/t1/t3/t3/t2/job is completely failed HOT 8
- raise RuntimeError("Got failed jobs") RuntimeError: Got failed jobs` HOT 2
- RuntimeError - "expected three arguments" HOT 5
- Please tag releases in git
- PySam error with MarginAlign+Minimap2 (last realease) HOT 2
- marginCall issues: AttributeError: 'NoneType' object has no attribute 'operationList'
- marginsStats HOT 3
- ImportError: No module named pysam HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from marginalign.