dib-lab / 2022-branchwater-benchmarking

Benchmarking etc code for https://dib-lab.github.io/2022-paper-branchwater-software/

command to get file sizes

find . -type f -printf "%k %p\n" | tee ~/scratch/magsearch/ls-wort.txt

2022-branchwater-benchmarking's Issues

benchmarks with snakemake

from code added in 2b78b96, this:

srun -p high2 --time=5:00:00 --nodes=1 --cpus-per-task 32 --mem 50GB --pty /bin/bash
snakemake -j 32 --snakefile benchmarking.snakefile 
csvtk concat benchmarks/*.txt | csvtk cut -t -f "h:m:s",cpu_time,max_rss

yields:

h:m:s   cpu_time        max_rss
0:23:14 40626.79        16106.77
0:23:14 38643.81        17553.49
0:22:37 37936.37        18257.87
0:25:05 39209.39        16107.48
0:26:36 37791.57        14046.44

is pyo3_branchwater much slower than sra_search??

branchwater benchmarking for 5 searches of 1,000 genomes x 10,000 metagenomes

old code, sra_search

time (minutes) 24.2, std 1.7
RSS/mem: mean 16.4 GB, std 1.6 GB
I/O: mean 93.4 GB, std 1.9

new code, pyo3_branchwater v ~0.5.1

time (minutes) 130.2, std 32.7
RSS/mem: mean 10.4 GB, std 146.9 MB
I/O: mean 66GB, std 13.1

first cut observations

time is ...a LOT slower. Not sure why that would be!? maybe need to look at other stats too, around efficiency and so on.

why would memory decrease?!
and esp why would I/O decrease?

maybe threading isn't working properly in pyo3_Branchwater?

example output for benchmarking

        Command being timed: "/home/ctbrown/scratch/magsearch/bin/searcher -k 31 --scaled=1000 -o output.csv siglist-a.txt data/wort-list-a.txt"
        User time (seconds): 46185.31
        System time (seconds): 1329.98
        Percent of CPU this job got: 2593%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 30:31.75
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 24259940
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 12
        Minor (reclaiming a frame) page faults: 428254673
        Voluntary context switches: 166157
        Involuntary context switches: 7161352
        Swaps: 0
        File system inputs: 196555504
        File system outputs: 2304
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

average and median file size

in ~ctbrown/scratch/2022-magsearch-benchmarking,

%  scripts/intersect-ls-and-catalog.py data/ls-wort.txt data/metagenomes-catalog.txt
loaded 767277 accessions from 'data/metagenomes-catalog.txt'
example acc: ERR1083538
Loaded 767277 file sizes from 'data/ls-wort.txt'
average file size: 9716.0
median file size: 570

output of full run (1000 x entire catalog)

rule a_vs_catalog:

        Command being timed: "snakemake -j 32 --snakefile benchmarking.snakefile"
        User time (seconds): 3609042.48
        System time (seconds): 98105.28
        Percent of CPU this job got: 2963%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 34:44:51
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 52518880
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 24
        Minor (reclaiming a frame) page faults: 30125462817
        Voluntary context switches: 9804070
        Involuntary context switches: 564183263
        Swaps: 0
        File system inputs: 14879893680
        File system outputs: 178544
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0