nf-core / mcmicro Goto Github PK
View Code? Open in Web Editor NEWAn end-to-end processing pipeline that transforms multi-channel whole-slide images into single-cell data.
Home Page: https://nf-co.re/mcmicro
License: MIT License
An end-to-end processing pipeline that transforms multi-channel whole-slide images into single-cell data.
Home Page: https://nf-co.re/mcmicro
License: MIT License
create a local module to stage the macsima .tif files for usage with ashlar. Macsima produces .tif files per channel, per tile, per exposure time, etc. The staging module should stack all this .tif files into an ome.tif per cycle as ashlar requires.
Even though the outdir parameter is defined as required in the schema file the workflow runs when it is missing.
'''
nextflow run ~/github/mcmicro-nf-core/main.nf --input_cycle ~/github/mcmicro-nf-core/assets/samplesheet_1_row_sample_cycle_multi_2.csv --marker_sheet ~/github/mcmicro-nf-core/assets/markers_multi_8.csv -profile test_work -params-file ~/github/mcmicro-nf-core/conf/workflow.json
'''
It runs to completion, but output is stored in a directory named 'null'.
No response
No response
Description of feature
Adding a phenotyping module for automated phenotyping using MAPS. In the first version, this module takes a user-pretrained model and the quantification table as input. The output will be the modified quantification table with an additional column for cell phenotypes.
An associated issue already exists in nf-core/modules to create it not only as a local module
The test
profile should run our small tonsil test image data through as much of the pipeline as we have working.
For a simple way to specify module parameters, for example change ashlar's max_shift to 20.. or change sigma?
I believe they should be added to nextflow.config.
Create a module to stage the PhenoImager .tif files for usage with Ashlar. PhenoImager produces float32 .tif files per tile per cycle. The staging module would transform float32 to uint16, extract metadata information and stack all .tif files into an ome.tif per cycle as Ashlar requires.
sample sheet format:
sample, cycle_number, channel_count, image_tiles
the marker sheet (markesrs.csv) will be input via a pipeline parameter
To Do:
Running nfcore/mcmicro test locally on a macOS (m2) runs into an issue with Mesmer.
Probably due to Docker container incompatibility with amd64 processing.
nextflow run main.nf --input ./assets/samplesheet-test.csv --outdir ./output/ -profile docker
N E X T F L O W ~ version 23.10.1
Launching `main.nf` [nauseous_cantor] DSL2 - revision: 68b73cf768
WARN: Access to undefined parameter `monochromeLogs` -- Initialise it to a default value eg. `params.monochromeLogs = some_value`
------------------------------------------------------
,--./,-.
___ __ __ __ ___ /,-._.--~'
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'
nf-core/mcmicro v2.0.0
------------------------------------------------------
Core Nextflow options
runName : nauseous_cantor
containerEngine: docker
launchDir : /Users/jnimoca/Jose_BI/MCMICRO/mcmicro
workDir : /Users/jnimoca/Jose_BI/MCMICRO/mcmicro/work
projectDir : /Users/jnimoca/Jose_BI/MCMICRO/mcmicro
userName : jnimoca
profile : docker
configFiles :
Input/output options
input : ./assets/samplesheet-test.csv
outdir : ./output/
!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use nf-core/mcmicro for your analysis please cite:
* The pipeline
* The nf-core framework
https://doi.org/10.1038/s41587-020-0439-x
* Software dependencies
https://github.com/nf-core/mcmicro/blob/master/CITATIONS.md
------------------------------------------------------
executor > local (2)
executor > local (2)
executor > local (2)
[56/3e9311] process > NFCORE_MCMICRO:MCMICRO:ASHLAR (TEST1) [100%] 1 of 1 ✔
[23/f99a61] process > NFCORE_MCMICRO:MCMICRO:DEEPCELL_MESMER (TEST1) [100%] 1 of 1, failed: 1 ✘
[- ] process > NFCORE_MCMICRO:MCMICRO:MCQUANT -
[- ] process > NFCORE_MCMICRO:MCMICRO:MULTIQC -
Execution cancelled -- Finishing pending tasks before exit
-[nf-core/mcmicro] Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_MCMICRO:MCMICRO:DEEPCELL_MESMER (TEST1)'
Caused by:
Process `NFCORE_MCMICRO:MCMICRO:DEEPCELL_MESMER (TEST1)` terminated with an error exit status (125)
Command executed:
python /usr/src/app/run_app.py mesmer --squeeze --nuclear-image TEST1.ome.tif --output-directory . --output-name mask.tif --image-mpp=0.215 --nuclear-channel 0 --compartment nuclear
cat <<-END_VERSIONS > versions.yml
"NFCORE_MCMICRO:MCMICRO:DEEPCELL_MESMER":
deepcell_mesmer:: 0.4.0
END_VERSIONS
Command exit status:
125
Command output:
(empty)
Command error:
Unable to find image 'vanvalenlab/deepcell-applications:0.4.1' locally
0.4.1: Pulling from vanvalenlab/deepcell-applications
docker: no matching manifest for linux/arm64/v8 in the manifest list entries.
See 'docker run --help'.
Work dir:
/Users/jnimoca/Jose_BI/MCMICRO/mcmicro/work/23/f99a614e8a4663dda3090b8e713ba2
Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
-- Check '.nextflow.log' file for details
Nextflow version 23.10.1
Hardware: MacBook Pro Apple M2 Pro 16Gb Memory
Executor: Local
Container Engine: Docker
OS: Sonoma 14.0 (23A344)
nfcore/mcmicro version: https://github.com/jmuhlich/mcmicro/tree/sample_sheet_mod
When tma_dearray is enabled, should outdir include the registration output (whole-slide image) or should we only publish the individually cropped TMA cores images? The registration output is helpful for debugging stitching problems, for now. When we get some good multiqc output for registration and tma_dearray, we should reconsider this.
Hey all,
I will create a local module to aggregate all single csv files from mcquant, to a single csv file.
An extra column will be created with "sample_id" as the labelling column.
Why? In the hackathon we want to create some simple modules that produce simple spatial analysis outputs.
Phenotyping with gates, and spatialLDA outputs.
@migueLib @chiarasch <-- working on SpatialLDA
I am aware that there is a step somewhere in vanilla mcmicro that does this, but we wanted a single module as a intermediate step.
Subdirectories should be named after the corresponding module name, with possible parallel outputs from different tools under different names or sub-sub-directories therein. E.g. instead of ashlar/
we should have registration/
.
No response
No response
No response
Let's add Stardist for mcmicro, I created the nf-core module for stardist https://nf-co.re/modules/stardist/
Specifying a param value like --input_cycle as a bare filename does not work -- nextflow doesn't seem to "see" the file.
Prefixing the filename with its full absolute path works around this. This is probably something about nf-validation handling of the params.
# samplesheet.csv DOES exist here!
$ nextflow run nf-core/mcmicro --input_cycle samplesheet.csv ...
...
ERROR ~ Error executing process > 'NFCORE_MCMICRO:MCMICRO:INPUT_CHECK:SAMPLE_AND_MARKER_SHEET_CHECK'
Caused by:
Not a valid path value: 'samplesheet.csv'
...
No response
No response
nf-validaton is not catching If a file given in the samplesheets does not exist
No response
No response
No response
We should decide whether markers.csv should have a per-cycle channel number column in addition to the running global channel number.
basicpy output file paths/names are based only on the input filenames. If a run has multiple samples with the same filenames then their basicpy outputs will clash and overwrite each other.
No response
No response
No response
move bin/check_marker_sheet.py and bin/check_sample_and_marker_sheet.py to groovy
Add Palom as a registration option for users that have prestitched whole-slide data with a disclaimer that for ensuring proper result quality, tiles with overlaps would be preferred.
TODO:
I realized that --illumination manual
may be semi-redundant with including the ffp
and/or dfp
columns in the sample sheet. We should either add a check in PIPELINE_INITIALISATION that these two things are aligned, or rethink the interface to this feature (perhaps the presence of user-provided illumination profiles would imply illumination="manual" and the illumination
param would only serve to invoke basicpy or another future tool).
Merge this into modules.config
:
process {
withName: BASICPY {
ext.args = "--darkfield"
}
}
Currently MCQuant has the option to pass additional skimage regionprops using the CLI, which is very nice. We would like to extend that by including specific glcm features in the MCQuant script.
Further, we would like McQuant to accept all available bioformats (e.g. also qptiff)
Other adjustments might have to be implemented if SpatialData is included (again depending on where - from start, after ASHLAR or after quant?)
Allow any non-empty string for marker_name and filter in schema_marker
I'll add a test for this too
No response
No response
No response
I am running Exemplar001.
I created a samplesheet (1 row per cycle)
Running defaults
It gets stuck at ASHLAR creating a too large image.. is it just an ashlar bug? or something else going wrong?
Will try to change some parameters and report back
nextflow run nf-core/mcmicro -r dev
--outdir /Users/jnimoca/Jose_BI/P12E19_nfcore_mcmicro/output/
--marker_sheet
/Users/jnimoca/Jose_BI/P12E19_nfcore_mcmicro/markers.csv
-profile
docker
--input_cycle
/Users/jnimoca/Jose_BI/P12E19_nfcore_mcmicro/samplesheet.csv
Output:
N E X T F L O W ~ version 23.10.1
Launching `https://github.com/nf-core/mcmicro` [silly_newton] DSL2 - revision: b8d46de48e [dev]
WARN: Access to undefined parameter `monochromeLogs` -- Initialise it to a default value eg. `params.monochromeLogs = some_value`
------------------------------------------------------
,--./,-.
___ __ __ __ ___ /,-._.--~'
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'
nf-core/mcmicro v2.0.0-gb8d46de
------------------------------------------------------
Core Nextflow options
revision : dev
runName : silly_newton
containerEngine: docker
launchDir : /Users/jnimoca/Jose_BI/P12E19_nfcore_mcmicro
workDir : /Users/jnimoca/Jose_BI/P12E19_nfcore_mcmicro/work
projectDir : /Users/jnimoca/.nextflow/assets/nf-core/mcmicro
userName : jnimoca
profile : docker
configFiles :
Input/output options
input_cycle : /Users/jnimoca/Jose_BI/P12E19_nfcore_mcmicro/samplesheet.csv
outdir : /Users/jnimoca/Jose_BI/P12E19_nfcore_mcmicro/output/
marker_sheet : /Users/jnimoca/Jose_BI/P12E19_nfcore_mcmicro/markers.csv
!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use nf-core/mcmicro for your analysis please cite:
* The pipeline
* The nf-core framework
https://doi.org/10.1038/s41587-020-0439-x
* Software dependencies
https://github.com/nf-core/mcmicro/blob/master/CITATIONS.md
------------------------------------------------------
WARN: Samplesheet warnings:
The samplesheet contains following unchecked field(s): [Filter]
executor > local (3)
[ca/23253c] process > NFCORE_MCMICRO:MCMICRO:INPUT_CHECK:SAMPLE_AND_MARKER_SHEET_CHECK (markers.csv) [100%] 1 of 1 ✔
[a3/b43fba] process > NFCORE_MCMICRO:MCMICRO:INPUT_CHECK:MARKER_SHEET_CHECK (markers.csv) [100%] 1 of 1 ✔
executor > local (3)
[ca/23253c] process > NFCORE_MCMICRO:MCMICRO:INPUT_CHECK:SAMPLE_AND_MARKER_SHEET_CHECK (markers.csv) [100%] 1 of 1 ✔
[a3/b43fba] process > NFCORE_MCMICRO:MCMICRO:INPUT_CHECK:MARKER_SHEET_CHECK (markers.csv) [100%] 1 of 1 ✔
[62/73bd10] process > NFCORE_MCMICRO:MCMICRO:ASHLAR (Exemplar001) [ 0%] 0 of 1
[- ] process > NFCORE_MCMICRO:MCMICRO:DEEPCELL_MESMER -
[- ] process > NFCORE_MCMICRO:MCMICRO:MCQUANT -
[- ] process > NFCORE_MCMICRO:MCMICRO:MULTIQC -
ASHLAR output from work folder:
Nextflow: version 23.10.1
M2 Macbook pro
executor: local
Container engine: Docker
OS: MacOS
nf-core/mcmicro v2.0.0-gb8d46de
When using input_sample, the image directory files are collected via Groovy's File.eachFileRecurse which does not yield the results in any sorted order. The results must be collected and then sorted explicitly by filename. In addition it's probably better not to recurse and require all files are provided at the top level, as user expectations will vary on how files in subdirectories should be sorted relative to each other. This is as simple as replacing eachFileRecurse with eachFile. Or eachFileMatch could be used to perform the .ome.tif filename match as well.
No response
No response
No response
conf/modules.config
in nf-core pipelines should always prevent versions.yml
files from ending up in the publishDir for individual modules. Instead the pipeline itself publishes a combined single versions file in pipeline_info/
.
No response
No response
No response
Sarek has some nice logic to restart a previous run from a specific step in the pipeline. At each restart point in the pipeline, they emit a samplesheet into outdir
that represents the state of the pipeline, potentially referencing other files in outdir
. When a later run specifies the step
param, the relevant samplesheet is automatically pulled out of outdir
. This seems like a good way to implement the old mcmicro start-at feature while still retaining the separation of inputs and outdir.
The modules.config entry for Mesmer includes --image-mpp=0.215
in ext.args which is not appropriate. We do need a smart way to pass the pixel size to Mesmer (and other tools that need it), but hard coding it here is definitely wrong. Should this be perhaps extracted from the Ashlar output image and included in the metadata map or in its own separate channel? While we're at it we could extract other useful image metadata too. We could also extract metadata from the raw image data -- this would be particularly useful to cross-check against the samplesheet and markersheet content in the input validation step.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.