Comments (13)
sounds interesting, like the idea.
from porecov.
cleanup might be difficult due to the "root" files generated via docker. the easier solution would be for people to point via flag or "config file" to a designated trash area?
from porecov.
also nextflow has i believe a build-in "remove workdir" function
from porecov.
also nextflow has i believe a build-in "remove workdir" function
oh, this would be best then I think. An yes, of course, users can also point to -w /scratch/
or -w /tmp
or some other place where the data is deleted automatically eventually
from porecov.
Actually there's a(n undocumented) feature to cleanup automatically the work dir adding cleanup = true in the config.
from porecov.
oh nice find :)
from porecov.
Cleaning up after complete finish might be too late:
Just ran into this problem when running a 'heavy' test case of 54 fastqs.
With local executor, every kraken2 process writes an unpacked version of the krakenDB into the work folder.
This just filled up the disk until a kraken2 process died due to lack of space.
As the DB is not part of the process output, but only the kraken result ist - it should be fine to delete the unpacked copy after kraken2 is done?
from porecov.
@RaverJay true, that unpacked krakenDB can be just directly deleted in the kraken process to free-up storage while execution
from porecov.
Just hacked that in on github
#67
from porecov.
- i add
scratch true
to "heavy disk" workdirs - downside its not working properly on wildcards (eg. path("*.fasta")) but in case of kraken classification it works fine
from porecov.
* i add `scratch true` to "heavy disk" workdirs * downside its not working properly on wildcards (eg. path("*.fasta")) but in case of kraken classification it works fine
What does scratch true
actually do? Just asking bc/ on HPCs depending on their configuration issues can come up if stuff is written to e.g. /scratch
or /tmp
per default. Or dies this automatically clean up work dirs where it is set to true
(this would be awesome)
from porecov.
it is something internal for the nextflow command.run :
on_exit() {
exit_status=${ret:=$?}
printf $exit_status > /home/pditommaso/projects/rnatoy/work/ff/836e306cde436de39a0e2e5e4c8afc/.exitcode
set +u
[[ "$COUT" ]] && rm -f "$COUT" || true
[[ "$CERR" ]] && rm -f "$CERR" || true
(sudo -n true && sudo rm -rf "$NXF_SCRATCH" || rm -rf "$NXF_SCRATCH")&>/dev/null || true
exit $exit_status
}
- however i think its only working correctly if the output is not using wild cards.
- I tried that on WtP as a global flag and was not working for some process - so this needs to be individually tested ( I added it to the kraken process as it was working in this case)
from porecov.
nextflow writes its own "scratch dirs"
from porecov.
Related Issues (20)
- Pangolin v4.0 HOT 11
- Include artic-tools "validate_scheme" HOT 1
- Runs fail at barcode22 HOT 8
- Private mutation from nextclade HOT 1
- Frameshift correction
- "split-fasta"-process fails due to leading empty line
- Workflow failed at artic_medaka, no '2.Genomes' output, with test_fastq and actual sample HOT 1
- export variant file (vcf) HOT 1
- add skip scorpio parameter to pangolin HOT 1
- Only calculate NanoPlot after read filtering step HOT 5
- Add new V5 ARTIC primer BED HOT 5
- Medaka step fails in the -profile fastq_test HOT 3
- summary_report.py fails HOT 7
- publish primersitereport from medaka output
- VarSkipV2b primer does not work as expected HOT 7
- Update Medaka to support R10.4.1 models HOT 14
- Update --help to list up-to-date primer schemes that are supported
- MinKNOW/Guppy update needs new model for R10.4.1 5 kHz HOT 6
- Warning when execution report and timeline already exists HOT 1
- The pipeline fails in artic_ncov_wf_artic_medaka HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from porecov.