cmkobel / assemblycomparator2

🦠📇 Genomes to report pipeline - for Bacteria and Archaea

                                      🦠  🧫  🔬  👾  🧪  💉  ☣ 

Hello. I'm a bioinformatician working in microbial ecology. I have a focus on software engineering and holo-omics. I research the interaction between gut microbes and their hosts. I try to keep my pipelines lean and share my workflows here on github mostly to train myself to make my work portable and well documented. I largely follow people on github who are into microbiology and omic-integrative methods.

My opinion on holo-omics is that we need to do more fundamental research and basic bioinformatic tool sharpening before we can truly dive in and understand the holistic interactions in the holobiont super-organisms. That is why I am currently also focusing on developing basic stuff like annotation and pipeline tools.

🚀 My current flagship project is the Assemblycomparator2 genomes-to-report pipeline which has just been published on bioconda.

I'm looking for a postdoctoral fellowship position in Copenhagen or online from spring 2025. Let me know if you have a lead.

                                   🦾   🔬   💻   🔣   💾   🚲   🧬

assemblycomparator2's Issues

Press ctrl-c to skip validation

Refer to fasttree paper in apescript

https://github.com/cmkobel/assemblycomparator/blob/98713e7c490f6ce7e0a084c07a772fe90bafe49c/workflow_templates.py#L246

Silly issue with long sample names in phylogeny

Resume analysis on error

If a .gwf directory already exists in the run folder, assemblycomparator can skip everything before running gwf.

Use python script instead of bash alias

Make a real install script that configures that python script.

The python script should use argparse, and calls the subsequent snakemake pipeline.

Default inputs/outputs means that starting the pipeline can be called as always (as with the alias)

Use a hash table to check if an assembly has already been annotated (prokka).

And then prefill the annotation.

GTDBtk

https://ecogenomics.github.io/GTDBTk/installing/index.html

Confusing dirnames

Rename docker_imgs to docker_files.

Makes more sense.

header should write the version number

Should only pass if the dir exists already.

assemblycomparator2/snakefile

Line 92 in 59b3eef

except:

pseudotargets

Make a pseudotarget named "cheap" or "fast" that just runs the quick stuff like:

• sequence_lengths
• assembly_stats
• mashtree
• ?something more?

add option to skip any2fasta

Often the user knows that the genomes are good stuff. And any2fasta can be skipped. Think about ~1000 genome datasets.

The report will fail if the metadata has not been created

Running assemblycomparator2 --until <rule> for any rule which is not metadata, on an uninitialized directory, the report will fail because the metadata file does not exist.

On solution is to have the metadata output as input in every single rule in the pipeline such that metadata will be forced to be created. Update, this solution might be suboptimal as it means that all jobs will have to run again if you update the metadata?

Weird keyerror when --untilling

roary dir overwrites on change of thresholds

The blastp-threshold should be part of the roary directory name.
Thus running more roary analyses with different thresholds, would be comparable.

kraken2 access on genomedk

export ASSCOM2_KRAKEN2_DB='/project/ClinicalMicrobio/faststorage/database/kraken2/k2_pluspf_20210127'

rule sequence_lengths (collector) does not void the report

Maybe each of the sequence_lengths_individual rules should touch it? Or what is the best option?

Add some form of clustering

.. based on the minhash distances.

Update: or on gene absence/presence

CheckM

assemblycomparator (shell) and workflow (py) don't agree on the files included

See "Warning: the file: .Rhistory doesn't look like a fasta file. Consider its inclusion." which is not part of the list made på the shell-script.

┌─┐┌─┐┌─┐┌─┐┌┬┐┌┐ ┬ ┬ ┬  ┌─┐┌─┐┌┬┐┌─┐┌─┐┬─┐┌─┐┌┬┐┌─┐┬─┐KMA
├─┤└─┐└─┐├┤ │││├┴┐│ └┬┘  │  │ ││││├─┘├─┤├┬┘├─┤ │ │ │├┬┘
┴ ┴└─┘└─┘└─┘┴ ┴└─┘┴─┘┴   └─┘└─┘┴ ┴┴  ┴ ┴┴└─┴ ┴ ┴ └─┘┴└─

Report issues at
    https://github.com/cmkobel/assemblycomparator/issues

Info: The blastp-identity threshold is set to 95 (default).
        (can be changed with the --blastp argument)

These are the 10 assemblies considered for project 2020_07_Ecoli_test_iqtree:
 B18_236667.fa
 B18_241039.fa
 B18_309150.fa
 B18_312563.fa
 B18_343222.fa
 B18_390375.fa
 B18_412827.fa
 B18_558476.fa
 B18_576661.fa
 B18_630107.fa

Do you wish to proceed? [y/n] y
 proceeding...
 activating environment...
 validating assembly files...
 backing up old content...
 archiving content...

These are the jobs:
BLASTP: 95
Warning: the file: .Rhistory doesn't look like a fasta file. Consider its inclusion.
cmp_copy_2020_07_Ecoli_test_iqtree_             shouldrun       0.00% [1/0/0/0]
cmp_kraken2_                                    shouldrun       0.00% [2/0/0/0]
cmp_abricate_                                   shouldrun       0.00% [2/0/0/0]
cmp_prokka_2020_07_Ecoli_test_iqtree_           shouldrun       0.00% [2/0/0/0]
cmp_summary_tables_2020_07_Ecoli_test_iqtree    shouldrun      88.24% [4/0/0/30]
cmp_mlst_2020_07_Ecoli_test_iqtree              shouldrun      83.33% [2/0/0/10]
cmp_roary_95_2020_07_Ecoli_test_iqtree          shouldrun      86.96% [3/0/0/20]
cmp_fasttree_2020_07_Ecoli_test_iqtree          shouldrun      83.33% [4/0/0/20]
cmp_iqtree_2020_07_Ecoli_test_iqtree            shouldrun      83.33% [4/0/0/20]
cmp_roary_plots_2020_07_Ecoli_test_iqtree       shouldrun      80.00% [5/0/0/20]
cmp_panito_2020_07_Ecoli_test_iqtree            shouldrun      80.00% [5/0/0/20]
cmp_mail_2020_07_Ecoli_test_iqtree              shouldrun      78.43% [11/0/0/40]

Do you wish to submit this job list? [y/n]

spaces

It seems that asscom2 cannot handle spaces in the parent path of the working directory.
Neither in the filenames of the assemblies.

Why is there a || ?

https://github.com/cmkobel/assemblycomparator/blob/98713e7c490f6ce7e0a084c07a772fe90bafe49c/workflow_templates.py#L246

use conda run instead of conda activate ...

https://docs.conda.io/projects/conda/en/latest/commands/run.html

Might be a cleaner solution.

checkm2_download fails when using apptainer

Because it can't write the path setting.

use mem_mib instead of mem_mb

`read` in shell is confusing

Pressing a mouse button or arrow keys emulates a "no"-answer.

It would be better if the user could confirm the entered value with an enter stroke.

Neat conda package

Instead of setting up aliases and system varibles, and installing snakemake+mamba, maybe there could just be a conda package that would do all that for you.
Setting up the slurm/pbs stuff should still be manual but that is a different topic.

Report v3 progress

Almost all is done, but some things are missing

• mlst
• abricate
• fasttree/quicktree on core (consider skipping)
• bakta (just copy-paste prokka)
• eggnog (not so urgent)
• gapseq (Should be easy, just make a heatmap of pathways)
• dbcan results
• panaroo

Ide til løsning af rapport-caveat

1. Lige nu kører rapporten some en rule. Men hvad med at installere R og alle pakkerne direkte i assemblycomparator2-condamiljøet. I dette tilfælde vil det være muligt at lave rapporten som et sidste script kald. Jeg mener der er en mulighed for at definere et script når pipelinen er færdig med at køre men jeg kan simpelthen ikke finde det.

2. Alternativt kunne man definere det som en ekstra ting i aliaset, således at når snakemake afslutter (exit 0) kan et nyt kald blive lavet. Hvis rule report ikke er med i outputlisten kunne man bare skrive ... && snakemake <blabla> --until report og så vil rapporten komme ud.

Håber det giver mening når jeg læser det her om 2 år.

Some rules should not run when there is only one sample.

Many of the comparison tools. Mashtree, roary, ...

The tools will just fail, which isn't a big deal. But would be nice if this was handled in a more graceful manner.

Sliding k-mer frequency visualization.

Use tabseq to calculate frequencies over the contigs. Visualize with some beautiful colors.

Considering to remove singularity support and just go with conda.

If the pipeline is to work with conda for local setups anyway, I might as well ditch singularity, and focus all the debugging effort on the conda envs instead.

Remove Card and Plasmidfinder from report

Results are not so relevant I think.

Support for different sub analyses

--cpo runs the CPO-relevant analyses (plasmids, resistance, etc)

--agg runs something relevant to Aggregatibacter

etc..

Pseudo rules like interproscan and prokka could be removed

since --until would work just as good.

I wonder why I never thought about that, as if all repeated jobs must end in a single??

Functional enrichment for plasmids

Bootstrap the tree

pseudo reads-ize the genomes for kraken2

Idea inspired by Tseemanns shred-reads command for snippy.

The idea is, that if you cut the genomes into reads, kraken will get much higher resolution. If I cut into the size that the kraken database is made up of, the species recognition will become more granulated.

Continuous integration with some testing?

Running a few tests.

• Installing, both with conda and apptainer
• Downloading databases.
• Running a few tests and checking that everything is produced correctly:
  • Single sample
  • Handful of samples
• Set up on thylakoid with cron
  • Enable error reporting through email or something..

Parametrize databases path

Consider if database setup can be made more parametric. Caveat with docker images is that it may be harder to acquire access to the databases. But maybe there is a way of doing that.

perl root and perl moo fails when the snakemake conda prefix variable is set

When I set the snakemake conda prefix variable to something, prokka (and possibly also mlst) fail because of problems with perl root and perl moo.

In the installation, the variable is set as such.
echo "export SNAKEMAKE_CONDA_PREFIX=${ASSCOM2_BASE}/conda_base" >> ~/.bashrc

A workaround is to delete the variable from the ~/.bashrc andor ~/.zshrc

Note: this issue only pertains to using conda (--use-conda)

Roary hack

assemblycomparator2/conda_envs/roary.yaml

Line 3 in 62d1518

- conda-forge

.. only works if you do:

export PERL5LIB=/home/cmkobel/assemblycomparator2/conda_base/b51707c89e77d1771344e5a65ab516a5_/lib/perl5/site_perl/5.22.0

But that is never gonna be a good solution.

Visualization of GC content throughout contigs

Use tabseq for development.

Check that kraken path exists before starting the workflow

Gives the user a chance to fix the problem ahead of time.

gtdbtk running time should be scaled to number of samples

assemblycomparator2/snakefile

Line 537 in 2d1a2b0

resources:

Resistance tables are much too wide

Solution: Pivot, so the samples are columns, and res. gene calls are rows.

Timed report

After 24 hours, or when everything is done:
write a report, with the results that have been created so far.

Pan/core legend is way too big

Solution: Make it into a continuous color scale legend bar.

versioning of packages

Fix (fixate) versions for everything in the conda environments.

Roary doesn't create a unique file on the blastp_identity

It should, because otherwise it might not rerun if the user changes the setting between runs.