Parallelization of predictor computations (DP recursions) about intarna

[Siddhant085]

I have experience with parallelization and the pthread and openmp libraries. Can I help solving this issue?

[bgruening]

@Siddhant085 always! :)

[martin-raden]

Hi @Siddhant085,
thanks for your interest in helping! If you want to have a look what we are needing, please fork the "intarna-2" branch, which is the current dev branch to be merged into master soon.

I have left a few "TODO"s in comments within the code to mark possible spots where we might be able to easily parallelize certain loops. Partially these are couples of nested loops, where a direct parallelization of both wont work, but at least the inner loop should be easily changed to parallelized processing.

An important constraint on parallelization: we need to be able to fix the maximal number of threads (via an additional command line argument that I can setup) to get the parallelization compatible with restricted multi-thread environments, eg. on HPC systems using an SGE queuing. At the beginning, a preprocessor variable will be sufficient like

#define INTARNA_MAX_THREADS 4

to be added to general.h

[Siddhant085]

@martin-mann Do you have some irc or slack channel where we can talk. I figured out that we can use openmp to parallelize the loops that you marked but to measure correctness and improve in performance I will need to run the code. I am having troubles in installation. I hope you can help.

[martin-raden]

@Siddhant085 sorry, dont have a channel at hand (do you?)

concerning installation:

(1) download and install the Vienna RNA package using e.g.

./configure  --without-doc --without-python --without-perl --without-swig --without-kinwalker --without-cluster --without-forester --without-kinfold --without-gsl --without-json --without-svm;
make install;

to strip away all unnecessary stuff. you might also want to set --prefix=YOURVRNAPATH

(2) install boost version > 1.50

(3) download the intarna-2 branch and configure like this

bash autotools-init.sh;
./configure --prefix=YOURINTARNAPATH;
make install;

while you might have to add --with-RNA=YOURVRNAPATH and --with-boost=YOURBOOTHPATH if you installed to a dedicated directory (which I suggest at least for the VRNA package to enable clean updates of the dependency if necessary). please check ./configure --help for other options e.g. for enabling debug information via --enable-debug etc.

for a simple test after compilation, you might run it like this

./YOURINTARNAPATH/bin/IntaRNA -t AAAAAACCCCCCGGGGGGGGGUUUUU -q GGGGGGCGCCCCCCC

concerning benchmarking

to benchmark performance changes, you can use the following data each within a separate text file and passed to -q and -t respectively:

>query
gugcggccugaaaaacagugcugugcccuuguaacucaucauaauaauuuacggcgcagccaagauuucccugguguuggcgcaguauucgcgcaccccggucuagccggggucauuuuuu

use more or less of the following content for the target file to increase/decrease workload for testing. you can even duplicate the input but would have to make the identifiers (after the > symbol) unique again. Note, a sequence entry in this FASTA format consists of

• a header line with name starting with > and
• one or more successive lines containing the sequence data (which is concatenated to a single space free sequence)

>b0001
AAGTATTTTTCAGCTTTTCATTCTGACTGCAACGGGCAATATGTCTCTGTGTGGATTAAAAAAAGAGTGTCTGATAGCAGCTTCTGAACTGGTTACCTGCCGTGAGTAAATTAAAATTTTATTGACTTAGGTCACTAAATACTTTAACCAATATAGGCATAGCGCACAGACAGATAAAAATTACAGAGTACACAACATCCATGAAACGCATTAGCACCACCATTACCACCACCATCACCATTACCACAGGTAACGGTGCGGGCTGACGCGTACAGGAAACACAGAAAAAAGCCCGCACCT
>b0002
CCAATATAGGCATAGCGCACAGACAGATAAAAATTACAGAGTACACAACATCCATGAAACGCATTAGCACCACCATTACCACCACCATCACCATTACCACAGGTAACGGTGCGGGCTGACGCGTACAGGAAACACAGAAAAAAGCCCGCACCTGACAGTGCGGGCTTTTTTTTTCGACCAAAGGTAACGAGGTAACAACCATGCGAGTGTTGAAGTTCGGCGGTACATCAGTGGCAAATGCAGAACGTTTTCTGCGTGTTGCCGATATTCTGGAAAGCAATGCCAGGCAGGGGCAGGTGG
>b0003
CCGCGTGAAGATTGCCGAAGTGGATGGTAATGATCCGCTGTTCAAAGTGAAAAATGGCGAAAACGCCCTGGCCTTCTATAGCCACTATTATCAGCCGCTGCCGTTGGTACTGCGCGGATATGGTGCGGGCAATGACGTTACAGCTGCCGGTGTCTTTGCTGATCTGCTACGTACCCTCTCATGGAAGTTAGGAGTCTGACATGGTTAAAGTTTATGCCCCGGCTTCCAGTGCCAATATGAGCGTCGGGTTTGATGTGCTCGGGGCGGCGGTGACACCTGTTGATGGTGCATTGCTCGGAG
>b0004
CGCGGCAGGCGGTCGCGGAAATCGGCGCGGTAGCGAGCGGTATCTCCGGCTCCGGCCCGACCTTGTTCGCTCTGTGTGACAAGCCGGAAACCGCCCAGCGCGTTGCCGACTGGTTGGGTAAGAACTACCTGCAAAATCAGGAAGGTTTTGTTCATATTTGCCGGCTGGATACGGCGGGCGCACGAGTACTGGAAAACTAAATGAAACTCTACAATCTGAAAGATCACAACGAGCAGGTCAGCTTTGCGCAAGCCGTAACCCAGGGGTTGGGCAAAAATCAGGGGCTGTTTTTTCCGCACG
>b0005
TCTCAATCAGGCCGGGTTTGCTTTTATGCAGCCCGGCTTTTTTATGAAGAAATTATGGAGAAAAATGACAGGGAAAAAGGAGAAATTCTCAATAAATGCGGTAACTTAGAGATTAGGATTGCGGAGAATAACAACCGCCGTTCTCATCGAGTAATCTCCGGATATCGACCCATAACGGGCAATGATAAAAGGAGTAACCTGTGAAAAAGATGCAATCTATCGTACTCGCACTTTCCCTGGTTCTGGTCGCTCCCATGGCAGCACAGGCTGCGGAAATTACGTTAGTCCCGTCAGTAAAAT
>b0006
TTCATACCATTGCCAGTGATTATCTACGCCAGCGTAAACTCGGCGTGCGCCCGGTGTTTGATCCATTGCGTTATCCGGATATCGGTCGCCAGCTTTCTCCGGACGCGTGGGATGATGTTTCGCAGGAGTAATCACAACTATCGATCAACTCATTCTCATTTTTTGCTAAAGTCGGCATAAATTTCCTGCAAGGACTGGATATGCTGATTCTTATTTCACCTGCGAAAACGCTTGATTACCAAAGCCCGTTGACCACCACGCGCTATACGCTGCCGGAGCTGTTAGACAATTCCCAGCAGT
>b0007
CAACCGCCGACAAGACGGTTATCGCCAGGCGAGACTGTTTCGGATTTCTGACACTGCGTTGATATCGCCCGCCATTTTTATACAAAACCTCATGTATGCTACGCAGAAGTTATCAAGTACCTCGTAGCGTATATACTTCTTAAACAATTGGTAACGTTTACACAGGAAAGTCATCGCGACCGGCAATAAGAGGGATATGCATGCCAGATTTTTTCTCCTTCATTAACAGCGTCCTTTGGGGATCGGTAATGATTTACCTGCTCTTCGGCGCAGGTTGTTGGTTCACTTTTCGCACCGGAT
>b0008
AGGTACTTGATAACTTCTGCGTAGCATACATGAGGTTTTGTATAAAAATGGCGGGCGATATCAACGCAGTGTCAGAAATCCGAAACAGTCTCGCCTGGCGATAACCGTCTTGTCGGCGGTTGCGCTGACGTTGCGTCGTGATATCATCAGGGCAGACCGGTTACATCCCCCTAACAAGCTGTTTAAAGAGAAATACTATCATGACGGACAAATTGACCTCCCTTCGTCAGTACACCACCGTAGTGGCCGACACTGGGGACATCGCGGCAATGAAGCTGTATCAACCGCAGGATGCCACAA
>b0009
TGGCAGTAGATAAACTGGCGGAAGGTATCCGTAAGTTTGCTATTGACCAGGAAAAACTGGAAAAAATGATCGGCGATCTGCTGTAATCATTCTTAGCGTGACCGGGAAGTCGGTCACGCTACCTCTTCTGAAGCCTGTCTGTCACTCCCTTCGCAGTGTATCATTCTGTTTAACGAGACTGTTTAAACGGAAAAATCTTGATGAATACTTTACGTATTGGCTTAGTTTCCATCTCTGATCGCGCATCCAGCGGCGTTTATCAGGATAAAGGCATCCCTGCGCTGGAAGAATGGCTGACAT
>b0010
CGTACTGCAAATCGCCTGCCTGCGCCGGATGGTTAGCGCCACTCAGGTCTGATACCGGAAACCGGGGAAAATTTTCATTTTCCCCTCAAAAGATCGTAGACACTGCCCCACTGGCTGATTATTATGCCGCGCCCTGAAAACACTACAGTTATTCAGGGAAATTATTTCACCATTCATTCGATGATGATTTTTGAGGAATTATGGGCAACACTAAGTTGGCTAATCCGGCACCGCTGGGCCTGATGGGCTTCGGCATGACCACCATTCTGCTTAACCTGCACAACGTGGGTTATTTCGCTC
>b0011
TAGCGCCAGTCAGAGCCTGAATATTCCTTCAGAAATAAAAGAAGGGCAAACCACTGACTGGATCAACATTAACAGCGATAACGACAATAAACGCTGCGTCTCAAAAATCACCTTTTCGGGTCATACGGTGAACTCATCGGATATGGCCACGCTGAAAATTATCGGCGACGACTAACGCCAGAATATAGAAGCCACAAAAAATGAATGTTAATTACCTGAATGATTCAGATCTGGATTTTCTTCAGCATTGTAGTGAGGAACAGTTGGCAAATTTCGCCCGATTGCTCACCCATAATGAAA
>b0013
TTTGCGCAGAAATTTCGCGATATGGCAGGACAAGACGTAAAATTGACAGCAATTTTATTCATATAAAGAATGAATCGCAGCCAAGGCATTCATCAAAAAATTGTAATAAAAAGAAAAGATTACGTGCCTGAATCTTCTCTTTATCAGCAGTAAACTAGTGGGTATTCATCCCCCTACCTCTTCCCACTAAGAGAATCCTTATGAAATCCGTTTTTACGATTTCCGCCAGCCTGGCGATTAGCCTGATGCTGTGCTGCACGGCGCAGGCAAACGACCATAAACTCCTCGGCGCCATTGCAA
>b0014
CATATCGCGAAATTTCTGCGCAAAAGCACAAAAAATTTTTGCATCTCCCCCTTGATGACGTGGTTTACGACCCCATTTAGTAGTCAACCGCAGTGAGTGAGTCTGCAAAAAAATGAAATTGGGCAGTTGAAACCAGACGTTTCGCCCCTATTACAGACTCACAACCACATGATGACCGAATATATAGTGGAGACGTTTAGATGGGTAAAATAATTGGTATCGACCTGGGTACTACCAACTCTTGTGTAGCGATTATGGATGGCACCACTCCTCGCGTGCTGGAGAACGCCGAAGGCGATC
>b0015
CCAGCAGCAACATGCCCAGCAGCAGACTGCCGGTGCTGATGCTTCTGCAAACAACGCGAAAGATGACGATGTTGTCGACGCTGAATTTGAAGAAGTCAAAGACAAAAAATAATCGCCCTATAAACGGGTAATTATACTGACACGGGCGAAGGGGAATTTCCTCTCCGCCCGTGCATTCATCTAGGGGCAATTTAAAAAAGATGGCTAAGCAAGATTATTACGAGATTTTAGGCGTTTCCAAAACAGCGGAAGAGCGTGAAATCAGAAAGGCCTACAAACGCCTGGCCATGAAATACCACC
>b0016
TCAAAGAGCTTCTTTGATGGTGTGAAGAAGTTTTTTGACGACCTGACCCGCTAACCTCCCCAAAAGCCTGCCCGTGGGCAGGCCTGGGTAAAAATAGGGTGCGTTGAAGATATGCGAGCACCTGTAAAGTGGCGGGGATCACTCCCATAAGCGCTAACTTAAGGGTTGTGGTATTACGCCTGATATGATTTAACGTGCCGATGAATTACTCTCACGATAACTGGTCAGCAATTCTGGCCCATATTGGTAAGCCCGAAGAACTGGATACTTCGGCACGTAATGCCGGGGCTCTAACCCGCC
>b0018
AGTTAAATAATTAAATCATTAAAAAATTACGCCACAAAAATGCGAGGCGTCTTCAGGATTAGAATCGATAAAACAAAATGGGTTAAAGAGAGCGATATAATAGCGGCGGGTGCTTGAGGCTGTCTGTCTCAGGCATTAGCTGAACGGCAGATAGAGAAAAGCCCCGAGTGATATTTTACCATCAACCCGAGGCCTCCTATATGCTGAACACATGTAGAGTGCCTCTTACTGACCGTAAGGTCAAGGAGAAGAGAGCAATGAAGCAGCATAAGGCGATGATTGTCGCCCTGATCGTCATCT
>b4412
ATTAGAATCGATAAAACAAAATGGGTTAAAGAGAGCGATATAATAGCGGCGGGTGCTTGAGGCTGTCTGTCTCAGGCATTAGCTGAACGGCAGATAGAGAAAAGCCCCGAGTGATATTTTACCATCAACCCGAGGCCTCCTATATGCTGAACACATGTAGAGTGCCTCTTACTGACCGTAAGGTCAAGGAGAAGAGAGCAATGAAGCAGCATAAGGCGATGATTGTCGCCCTGATCGTCATCTGTATCACCGCCGTAGTGGCGGCGCTGGTAACGAGAAAAGACCTCTGTGAGGTTCACA
>b0019
ACCTTAAGCTATGATTATCTAGGCTTAGGGTCACTCGTGAGCGCTTACAGCCGTCAAAAACGCATCTCACCGCTGATGGCGCAAATTCTTCAATAGCTCGTAAAAAACGAATTATTCCTACACTATAATCTGATTTTAACGATGATTCGTGCGGGGTAAAATAGTAAAAACGATCTATTCACCTGAAAGAGAAATAAAAAGTGAAACATCTGCATCGATTCTTTAGCAGTGATGCCTCGGGAGGCATTATTCTTATCATTGCCGCTATCCTGGCGATGATTATGGCCAACAGCGGCGCAA
>b0020
ATCTTTATTGCCAGCCTGGCCTTTGGTAGCGTAGATCCAGAACTGATTAACTGGGCGAAACTCGGTATCCTGGTCGGTTCTATCTCTTCGGCGGTAATTGGATACAGCTGGTTACGCGTTCGTTTGCGTCCATCAGTTTGACAGGACGGTTTACCGGGGAGCCATAAACGGCTCCCTTTTCATTGTTATCAGGGAGAGAAATGAGCATGTCTCATATCAATTACAACCACTTGTATTACTTCTGGCATGTCTATAAAGAAGGTTCCGTGGTTGGCGCAGCGGAGGCGCTTTATTTAACTC
>b0021
GCTCCAGTGGCTTCTGTTTCTATCAGCTGTCCCTCCTGTTCAGCTACTGACGGGGTGGTGCGTAACGGCAAAAGCACCGCCGGACATCAGCGCTATCTCTGCTCTCACTGCCGTAAAACATGGCAACTGCAGTTCACTTACACCGCTTCTCAACCCGGTACGCACCAGAAAATCATTGATATGGCCATGAATGGCGTTGGATGCCGGGCAACAGCCCGCATTATGGGCGTTGGCCTCAACACGATTTTACGTCACTTAAAAAACTCAGGCCGCAGTCGGTAACCTCGCGCATACAGCCGG
>b0022
AAAAGAGACATGCCGTTGTACTATCTGAACTCTTCGTGGCTTAAGAATCGTAGAACCATCGGCTGATACATTCTGAACAAGAATATCTACTCAATCTCCAGACTCCCCCACAAAGAATATGGATATTGTGATACACATTGAGGTAGGTGATGCTGCCAACTTACTGATTTAGTGTATGATGGTGTTTTTGAGGTGCTCCAGTGGCTTCTGTTTCTATCAGCTGTCCCTCCTGTTCAGCTACTGACGGGGTGGTGCGTAACGGCAAAAGCACCGCCGGACATCAGCGCTATCTCTGCTCTC
>b0023
TGACCGCCTGCTGCAATTTTTATCGCGGAAAAGCTGTATTCACACCCCGCAAGCTGGTAGAATCCTGCGCCATCACTACGTAACGAGTGCCGGCACATTAACGGCGCTTATTTGCACAAATCCATTGACAAAAGAAGGCTAAAAGGGCATATTCCTCGGCCTTTGAATTGTCCATATAGAACACATTTGGGAGTTGGACCTTGGCTAATATCAAATCAGCTAAGAAGCGCGCCATTCAGTCTGAAAAGGCTCGTAAGCACAACGCAAGCCGTCGCTCTATGATGCGTACTTTCATCAAGA
>b0024
TTGATGAAAGTACGCATCATAGAGCGACGGCTTGCGTTGTGCTTACGAGCCTTTTCAGACTGAATGGCGCGCTTCTTAGCTGATTTGATATTAGCCAAGGTCCAACTCCCAAATGTGTTCTATATGGACAATTCAAAGGCCGAGGAATATGCCCTTTTAGCCTTCTTTTGTCAATGGATTTGTGCAAATAAGCGCCGTTAATGTGCCGGCACTCGTTACGTAGTGATGGCGCAGGATTCTACCAGCTTGCGGGGTGTGAATACAGCTTTTCCGCGATAAAAATTGCAGCAGGCGGTCAGT
>b0025
TGGCGCAGGATTCTACCAGCTTGCGGGGTGTGAATACAGCTTTTCCGCGATAAAAATTGCAGCAGGCGGTCAGTTTCTTCCCGTGATTTGCGCCATGGCAATGAAAAGCCACTTCTTTCTGATTTCGGTACTCAATCGCCGGTTAACCTTGACCGCTGTACAAGGTATACTCGGACGATTTTCACTGTTTTGAGCCAGACATGAAGCTGATACGCGGCATACATAATCTCAGCCAGGCCCCGCAAGAAGGGTGTGTGCTGACTATTGGTAATTTCGACGGCGTGCATCGCGGTCATCGCG
>b0026
ATGTTGCAATGGACCTTTACGGTCGCCATATACAAGTAGTGCTGCGTAAAAAAATACGCAATGAGCAGCGATTTGCGTCGCTGGACGAACTGAAAGCGCAGATTGCGCGTGATGAATTAACCGCCCGCGAATTTTTTGGGCTAACAAAACCGGCTTAAGCCTGTTATGTAATCAAACCGAAATACGGAACCGAGAATCTGATGAGTGACTATAAATCAACCCTGAATTTGCCGGAAACAGGGTTCCCGATGCGTGGCGATCTCGCCAAGCGCGAACCCGGAATGCTGGCGCGTTGGACTG
>b0027
TATAACGACGCACCTGCTGATGCTCAGCAGAGCGAAGTACTCAAAGGGCTGAAAGTCGCGTTGAGTAAAGCCGAAGGTGAGAAGTGCCCACGCTGCTGGCACTACACCCAGGATGTCGGCAAGGTGGCGGAACACGCAGAAATCTGCGGCCGCTGTGTCAGCAACGTCGCCGGTGACGGTGAAAAACGTAAGTTTGCCTGATGAGTCAATCGATCTGTTCAACAGGGCTACGCTGGCTGTGGCTGGTGGTAGTCGTGCTGATTATCGATCTGGGCAGCAAATACCTGATCCTCCAGAACT
>b0028
CGATACTGCCATCTGTGTCGGTGCGGCACTGATTGTGCTGGAAGGTTTTTTGCCTTCTAGAGCGAAAAAACAATAATAAACCCTGCCGGATGCGATGCTGACGCATCTTATCCGGCCTACAGATTGCTGCGAAATCGTAGGCCGGATAAGGCGTTTACGCCGCATCCGGCAAAAATCCTTAAATATAAGAGCAAACCTGCATGTCTGAATCTGTACAGAGCAATAGCGCCGTCCTGGTGCACTTCACGCTAAAACTCGACGATGGCACCACCGCCGAGTCTACCCGCAACAACGGTAAAC
>b0029
CCGTGAATTTATGGATGCAGGCGAGCCAGAAATTGGCGCAATCATGCTTTTTACCGCAATGGATGGCAGTGAGATGCCTGGCGTGATCCGCGAAATTAACGGCGACTCCATTACCGTTGATTTCAACCATCCGCTGGCCGGGCAGACCGTTCATTTTGATATTGAAGTGCTGGAAATCGATCCGGCACTGGAGGCGTAACATGCAGATCCTGTTGGCCAACCCGCGTGGTTTTTGTGCCGGGGTAGACCGCGCTATCAGCATTGTTGAAAACGCGCTGGCCATTTACGGCGCACCGATAT
>b0030
ATTCTGGTGCAGAATGTGGTGGCACGTTTGCAGCAGCTGGGCGGTGGTGAAGCCATTCCGCTGGAAGGCCGTGAAGAAAACATTGTTTTCGAAGTGCCGAAAGAGCTGCGTGTCGATATTCGTGAAGTCGATTAAGTCATTAGCAGCCTAAGTTATGCGAAAATGCCGGTCTTGTTACCGGCATTTTTTATGGAGAAAACATGCGTTTACCTATCTTCCTCGATACTGACCCCGGCATTGACGATGCCGTCGCCATTGCCGCCGCGATTTTTGCACCCGAACTCGACCTGCAACTGATGA
>b0031
GTGGGTGGCTGAGGTGCTGGCTCTGGCGTCGTAACCTGTCACATGTTATTGGCATGCAGTCATTCATCGACTCATGCCTTTCACTGATATCCCTCCCTGTTTATCATTAATTTCTAATTATCAGCGTTTTTGGCTGGCGGCGTAGCGATGCGCTGGTTACTCTGAAAACGGTCTATGCAAATTAACAAAAGAGAATAGCTATGCATGATGCAAACATCCGCGTTGCCATCGCGGGAGCCGGGGGGCGTATGGGCCGCCAGTTGATTCAGGCGGCGCTGGCATTAGAGGGCGTGCAGTTGG
>b0032
TTGTCGCTTAATGCCTGTAAAACATGCATGAGCCACAAAATAATATAAAAAATCCCGCCATTAAGTTGACTTTTAGCGCCCATATCTCCAGAATGCCGCCGTTTGCCAGAAATTCGTCGGTAAGCAGATTTGCATTGATTTACGTCATCATTGTGAATTAATATGCAAATAAAGTGAGTGAATATTCTCTGGAGGGTGTTTTGATTAAGTCAGCGCTATTGGTTCTGGAAGACGGAACCCAGTTTCACGGTCGGGCCATAGGGGCAACAGGTTCGGCGGTTGGGGAAGTCGTTTTCAATA
>b0033
GCAAACCTGCGTGTCACGCATAAATCCCTGTTCGACGGTACGTTACAGGGCATTCATCGCACCGATAAACCGGCATTCAGCTTCCAGGGGCACCCTGAAGCCAGCCCTGGTCCACACGACGCCGCGCCGTTGTTCGACCACTTTATCGAGTTAATTGAGCAGTACCGTAAAACCGCTAAGTAATCAGGAGTAAAAGAGCCATGCCAAAACGTACAGATATAAAAAGTATCCTGATTCTGGGTGCGGGCCCGATTGTTATCGGTCAGGCGTGTGAGTTTGACTACTCTGGCGCGCAAGCGT
>b0034
TGATGAGGACCGTTTTTTTTTGCCCATTAAGTAAATCTTTTGGGGAATCGATATTTTTGATGACATAAGCAGGATTTAGCTCACACTTATCGACGGTGAAGTTGCATACTATCGATATATCCACAATTTTAATATGGCCTTGTTTAATTGCTTCAAAACGAGTCATAGCCAGACTTTTAATTTGTGAAACTGGAGTTCGTATGTGTGAAGGATATGTTGAAAAACCACTCTACTTGTTAATCGCCGAATGGATGATGGCTGAAAATCGGTGGGTGATAGCAAGAGAGATCTCTATTCATT
>b0035
AACAGCGCCCCGCTGGCGATTGCGGCGCTGAAAGAGATCTACCGCACCACCAGCGAAATGCCGGTAGAAGAAGCGTATCGCTATATTCGCAGCGGCGTGTTGAAACACTATCCATCGGTTCTGCATTCGGAAGATGCCATTGAAGGGCCGCTGGCGTTTGCCGAGAAGCGCGATCCGGTGTGGAAAGGACGTTAACAACCGTGAGCTATTACGCCTTTGAGGGGTTAATTCCGGTGGTTCACCCGACGGCGTTTGTCCATCCCAGTGCCGTCTTGATTGGCGATGTGATTGTGGGAGCCG
>b0036
TGGCGAAATTTAAAGTGCCCTCTTATCTGGAGATCAGAAAAGATCTGCCACGTAATTGCTCGGGGAAAATAATTAGAAAGAATCTGAAATAAATGAAACAGCAGGGAACTACCCTGCCAGCTAATAATCACACTCTAAAACAATATGCATTTTTTGCAGGGATGTTGTCGTCCCTGAAAAAGCAAAAATGGAGAAAAGGAATGAGTGAATCATTACATCTGACCCGCAATGGATCAATTCTGGAAATTACCCTTGATCGTCCAAAAGCGAATGCTATTGATGCAAAAACCAGCTTTGAAA
>b0037
CGGACAAATCTGGCGCGGAATGCCCTCACATGGCATGGACACGGCTGCCATTTTGAAAAATATCGGCTACAGCGAAAACGACATTCAGGAGTTGGTCAGCAAAGGTCTGGCCAAAGTTGAGGACTAAACGCGTGGCGTCAGTCGGCTTGCCTGACTAGCGCCCGCCCAATAACAAACGAACAAAAATGGATAGAGGTGCAATGGATATCATTGGCGGACAACATCTACGTCAAATGTGGGACGATCTTGCGGACGTTTACGGTCATAAAACGGCGCTGATTTGTGAATCCAGCGGCGGAG
>b0038
CGCGTCTCCGGGGGATCTGACGAAATGCAGATCCTGACGCTGGGTCGTGCGGTGCTGAAGCAATACCGCTAAGTCCGTGCGCTGCCTGATGCGACGCTGATGCGTCTTATCAGGCCTACGGGAGCACGAATGTAGGCCGGATAAAGCGTTTACGCCGCATCCGGCAGTCATGCGCTACATAAATTTTTCAGGAGCAAATAATGGATCATCTACCCATGCCGAAATTCGGGCCGTTGGCCGGATTGCGCGTTGTCTTCTCCGGTATCGAAATCGCCGGACCGTTTGCCGGGCAAATGTTCG
>b0039
GTATCGGTTGGTCAATTCTGGTTGGCATTATCGGTATTGTTCTGCTGGCGCTCGGCGGCCTGAAACCGATTCAAACCGCCATTATCGCCGGAGGATGCCCGCTGTTCTTCGTCAACATTATGGTGACGCTCTCCTTTATTAAAGACGCGAAACAGAACTGGAAAGATTAATTAACCCCCAAAATATCAAGAGGTTGAAAGATGGATTTTAATTTAAATGATGAGCAGGAACTGTTTGTCGCCGGTATCCGCGAACTGATGGCCAGCGAAAACTGGGAGGCCTATTTTGCCGAGTGCGACC
>b0040
ATCAAAATTAACGAAAAAACGCCCTCTATTCACCCGAAACAAAAATGTGATACCAATCACAGAATACAGCTTATTGAATACCCATTATGAGTTAGCCATTAACGCGTCCACGAGGTTAATAATAATTATATTAAATGTTAACAAAAATAAAACAAACGGGAACGCAAAATAAATATTCGTTTTCACAGTGGAATTAACTCATGAAGAATGAAAAGAGAAAAACGGGAATAGAACCGAAGGTTTTCTTTCCGCCGTTAATAATCGTCGGCATACTTTGTTGGCTTACAGTCAGAGATCTGG
>b0041
AACAATATTGAAAATTTCTTTTTGCTACGCCGTGTTTTCAATATTGGTGAGGAACTTAACAATATTGAAAGTTGGATTTATCTGCGTGTGACATTTTCAATATTGGTGATTAAAGTTTTATTTCAAAATTAAAGGGCGTGATATCTGTAATTAACACCACCGATATGAACGACGTTTCCTTCATGATTTCTGGAGATGCAATGAAGATTATTACTTGCTATAAGTGCGTGCCTGATGAACAGGATATTGCGGTCAATAATGCTGATGGTTCATTAGACTTCAGCAAAGCCGATGCCAAAA
>b0042
CTCGGCGCGGCGAAAAAGCCCGTCCAGGTATGGTCGGCGGCGGATATTGGTTTTAACGCAGAGGCAGCCTGGTCAGAACAACAGGTTGCCGCGCCGAAACAGCGCGAACGTCAGCGCATCGTGATTGAAGGCGACGGCGAAGAACAGATCGCCGCATTTGCTGAAAATCTTCGCAAAGTCATTTAATTACAGGGGATGCTATGAACACGTTTTCTCAAGTCTGGGTATTCAGCGATACCCCTTCTCGTCTGCCGGAACTGATGAACGGTGCGCAGGCTTTAGCTAATCAAATCAACACCT
>b0043
ATCCAGCACATGGTTGGCGCTAACGCGTCGCAAACCATTTTCGCCATCAATAAAGATAAAAATGCGCCGATCTTCCAGTACGCGGATTACGGCATTGTTGGCGACGCCGTGAAGATCCTTCCGGCGCTGACCGCAGCTTTAGCGCGTTGATCCACTCTGGCAGGGCTGCATTTTGGCCCTGCCGCTGACAGGGAGCTCTTATGTCCGAAGATATCTTTGACGCCATCATCGTCGGTGCAGGGCTTGCCGGTTCGGTTGCCGCACTGGTGCTCGCCCGCGAAGGTGCGCAAGTGTTAGTTA
>b0044
ATGTACCAGAAACTACCGGCGTTCCTTGATAACCCACGCATGTTTAGCGGCTACCCGGAGCTGGCGGTGGGTGTGGCGCGTGACCTGTTCACCATTGATGGCAGCGCGCCGGAACTGATGCGCAAGAAAATCCTCCGCCACGGCAAGAAAGTGGGCTTCATCAATCTAATCAAGGATGGCATGAAAGGAGTGACCGTTTTATGACTTCTCCCGTCAATGTGGACGTCAAACTGGGCGTCAATAAATTCAATGTCGATGAAGAGCATCCGCACATTGTTGTGAAGGCCGATGCTGATAAAC
>b0045
AAGAAGCAGGATGACGGCAGTGTGCGCTTCGATTACGCCGGATGTCTGGAGTGCGGCACCTGTCGCATTCTGGGGCTGGGGTCGGCGCTGGAACAGTGGGAATACCCGCGCGGCACCTTTGGTGTGGAGTTCCGTTACGGCTGATGTTGGTTTGATACGTAACGCCGCACTGACTCTCATTGCAAAAAACAGGAATAACCATGCAACCGTCCAGAAACTTTGACGATCTCAAATTCTCCTCTATTCACCGCCGCATTTTGCTGTGGGGAAGCGGTGGTCCGTTTCTGGATGGTTATGTAC
>b0046
TTGTTGATTTCCGTAGCGTTTGCCCCGGAGACTCGAGGGATGTCACTGGCGCAGACCAGCAATATGACGATCCGCGGGCAGAGAATGGGGTAAATTGTTCAGATTTCTCTCTTTTCTGAATCAATATTATTGACTATAAGCCGCGTGAATATATGACTACACTTTGTGGGAAAACAAAGGCGTAATCACGCGGGCTACCTATGATTCTTATAATTTATGCGCATCCGTATCCGCATCATTCCCATGCGAATAAACGGATGCTTGAACAGGCAAGGACGCTGGAAGGCGTCGAAATTCGCT

thanks for your interest and help!

[Siddhant085]

I am getting the following
error configure: error: cannot find install-sh, install.sh, or shtool in "." "./.." "./../.."

I am able to successfully install VRNA and boost. This error occurs when I run ./configure in the IntraRNA folder with branch as intarna-2.

[martin-raden]

hah.. sorry.. forgot a central pre-processing step when you checkout the IntaRNA sources:

you have to run or source the autotools-init.sh script! (I have updated the list above now)

this script does all the required GNU autotools calls, since the repository content is no distribution (with autotool-version dependent files).

[Siddhant085]

I am able to compile and run the code. I performed some profiling just to get the data. Most of the time is consumed by the boost matrix access methods after which we get predictorMfe2d:fillHybridE . I went through the code and the variable curMinE might have race condition problems. I say might because I am not from a biology background and I am not able to understand what the algorithm is supposed to do completely. If curMinE is significant for each i1,i2 pair it is free from race conditions otherwise we will have to find a workaround. Some light on this will be helpful. I will be putting up the results of the profiling in two days.

[martin-raden]

thanks for investigation. I checked the semantics of the underlying dynamic programming recursion and found that for a fixed i1 the computation for all i2 settings are independent. thus, a parallization of the inner loop is possible, the outer processing has to be serial.

thus, one can move curMinE into the inner loop (only outside to avoid redundant allocation) to parallelize it. all other variables shouldnt be changed beside the according hybridE*(i1,i2) entry.

the finally called updateOptima() function might be the bottleneck for multithreading. maybe one can do a serial update using a second iteration on i2... but that is to be checked, if this is really needed.

thanks for investigation and I am curious about the results..

[Siddhant085]

@martin-mann refrer to the blog https://siddhant085.wordpress.com/2017/01/28/the-project/.
I moved out the updateOptima and parallelized the inner loop, however the performance improvement is almost neglegible.

[martin-raden]

Hi @Siddhant085,
it would be good to see the performance change when you swap the input files, i.e. take the big file as query and the small as target. the query sequence is handled by the inner loop, thus, the longer the sequence the higher the yield from parallelization. The good thing: you dont have to change anything to see the difference.. ;)

Thanks for looking into this,
Martin

[martin-raden]

@Siddhant085 : could you please upload your changes to your github fork? Otherwise your efforts are not available... thanks!

[Siddhant085]

@martin-mann I pushed my change and created a pull request so that you can see the changes. I tried measuring the performance by using the big file as query but it gave almost the same result.

[martin-raden]

@Siddhant085 thanks for the pull request. Concerning your evaluation, I have a few questions and suggestions.

• How many threads/cores did you use? Sure that many were used?
• Did you run it with profiler information switched off? (no -pg compiler flag)
• Did you reenable compilation optimization with at least -O2 compiler flag?
• What machine (CPU,cores) are you testing with?

Most of the IntaRNA runtime is consumed by preprocessing, which can be switched off by setting the --qAcc=N --tAcc=N IntaRNA runtime arguments. Given these two additional arguments, one might get a better picture of possible runtime improvements. Also the use of a longer query sequence can help, e.g.

>query_long
UUUAAAUUAAAAAAUCAUAGAAAAAGUAUCGUUUGAUACUUGUGAUUAUACUCAGUUAUA
CAGUAUCUUAAGGUGUUAUUAAUAGUGGUGAGGAGAAUUUAUGAAGCUUUUCAAAAGCUU
GCUUGUGGCACCUGCAACUCUUGGUCUUUUAGCACCAAUGACCGCUACUGCUAAU
UGCUUCUUAGUAAUGGCCAGCUCGGGGAUUUGGAACUGGUGGUUGUUAAUUCUAGGCAAA
UAGAUUAAAAUUAAAUUACUCAAAUUGUGUGACGAGUUUUAUGAAGCUUUUUAAAAGCUU
GUUGGUAGCUCCAGCAACGAUUGGGCUACUAGCUCCAUUUUCAACGUUUGCUGGC
CAGUAUCUUAAGGUGUUAUUAAUAGUGGUGAGGAGAAUUUAUGAAGCUUUUCAAAAGCUU

Thanks for your efforts!

[Siddhant085]

@martin-mann Sorry for the late response. I thought about the questions you asked and that gave me a much better understanding of the working. I timed only the fillHybrid_E function and found that it takes up the majority of the time and also that it increases with longer query strings. I am working on parallelizing the part. Openmp is throwing some errors. Will solve it and report to you the performance improvement and also the answers to the questions you asked in the last post.

[Siddhant085]

@martin-mann I changed the inner loop like this

#pragma omp parallel for

	for (i2=hybridE.size2(); i2 > 0;i2--) {

		// direct cell access

		curCell = &(hybridE(i1,i2-1));

		// check if left side can pair

		if (E_isINF(curCell->E)) {

			continue;

		}

However on running in parallel

for (w1=1; w1-1 <= energy.getMaxInternalLoopSize1() && i1+w1<hybridE.size1(); w1++) {
		for (w2=1; w2-1 <= energy.getMaxInternalLoopSize2() && (i2-1+w2)<hybridE.size2(); w2++) {

The test conditions for these loops get violated for some reason. I am working on it. Will keep you informed on updates.

[martin-raden]

You might have to enable shared use of all class member and outer temp variables (see recent source in intarna.cpp), i.e. energy, hybridE, i1, ... omp has a lot of additional options for the loop parallization.

Furthermore, move the definition of the w1,w2 vars into the parallized loop, otherwise they might get concurrently written by the threads, causing the problems. That was only a single thread optimization..

So far my ideas..

[Siddhant085]

I was able to find what the problem was. The variables curCell and curCellEtotal are shared. I am able to get the correct output if I put in the changes to these variables in a critical section. However this approach takes longer than usual. I will try to declare them as shared variables and reduce them at the end of the loop. This post is to keep you updated on the progress. I have my college internal assessment next week because of which I might be a little delayed in the updates, but I will try to keep up.

[Siddhant085]

I am able to remove all the race conditions. Also instead of parallelizing inner loop I am parallelizing the outer loop in order to get better performance. updateOptima is put into a critical section. However I am performance of the parallel code is almost double than that of the serial code. This is not due to the critical section around the updateOptima as I tried it without it also. I think its because of False shairing. I will work on it and update. I have updated the code in the pull request.

[Siddhant085]

The increase in time is because of calls to getE and getE_interLeft functions in the inner most loop. This might also be because this functions might be using VRNA library functions which might not be able to run in parallel.

[martin-raden]

thanks for the update and investigations. mhh.. the VRNA functions should not be the bottleneck and OMP conform.. did you compile the linked VRNA package with OMP suppor? there is a configure option, as far as i remember right now..

strange that you dont get any speedup. when parallelizing the outer loop, ensure one of the double loop is fixed, otherwise there should be race conditions.

[Siddhant085]

I was experimenting with the code and as I mentioned earlier that the call to getE function was causing the bottle neck. I tried critical section around it and the code started running 4 seconds faster on one test case, which was contradictory to what I was expecting.

[Siddhant085]

The delay was caused by a piece of code introduced by me. I am able to get a speed up on parallelizing. On using the 45 cases given as target and a single query I am getting 38.191s as the real time in serial and I am getting a time of 29.153s as real time on running in parallel. I feel the code can perform better. I will post the performance comparison in a more formal way soon.

[martin-raden]

Thanks, that looks very promising! I reviewed your changes and found a problem in the parallelization that breaks the underlying recursion logic. Please have a look. I guess the necessary fix might reduce your performance benefit.. :/

BUT, as I also commented in the review, I guess the bottleneck is the updateOptima() call, which cannot be taken out of the omp_critical when placed where it is right now. So not avoidable without restructuring the code.
If you want to get rid of that problem, you have to first fill all cells (in parallel) with a successive second i1-i2 loop reporting serially all non-E_INF cells via updateOptima(). That way you would circumvent the omp_critical and get the maximum out of the parallelization.

[Siddhant085]

I fixed the code. Moved the parallelization to the inner loop. The performance is hit for small cases but for large cases I am getting the same performance improvement, which is good news. This is maybe because all the work is being done inside the inner loop which should be enough to overweight the overhead generated by the assigning of tasks to threads. I will look into reducing the bottleneck that is the call to the updateOptima to see if anymore increase in performance is possible.

[martin-raden]

thanks for the update and looking forward to the new results!

[Siddhant085]

@martin-mann I am sorry I was not able to work for a few days. I looked into the problem. We need the values of curCell and curCellEtotal to call updateOptima(). So are you suggesting that I create a new data structure to store these values for all values of i2 and then call updateOptima?

[Siddhant085]

I declared an array and stored all the curCell and curCellEtotal and called updateOptima outside the parallelized region however I didn't get any major performance improvement. It came down from 30.64s to 29.95s.

[martin-raden]

Hi @Siddhant085 was a offline for holidays and try to catch up all that stacked inbetween..

Concerning your changes, could you please update your git repo, such that I can have a look?
https://github.com/Siddhant085/IntaRNA/tree/Issue%2318 is from 28th Feb..

You shouldn't need an extra data structure (all optimal results are stored within the table already) but "only" repeat the for-loops to travers the computed cells (serially after parallel computation) calling updateOptima() whenever a cell is not equal to E_INF...

But it is still possible that there is not much to gain with parallelization here.. :/

[Siddhant085]

@martin-mann Latest commit

Have a look at the latest commit,in which I have implemented whatever I mentioned in the last comment. I am not able to understand where do we get the values of curCell and curCellEtotal outside the parallelized region.

[martin-raden]

Hi @Siddhant085
I have commented your code in the pull request with according statements how to get the values.
Thanks,
Martin

[Siddhant085]

@martin-mann I changed the code to remove the redundant data structure. I failed to observe the fact that curCell is a pointer and the actual data is getting updated in hybridE. I thought curCell and curCellEtotal assume local values. Have a look

[martin-raden]

thanks for the changes, and yes, that is what I had in mind..

to get an idea on your test setup, please tell me:

• how long are the target sequences used?
• how long are the query sequences used?

the current parallelization is for the indices i2 of the query sequences ... i1 are the target sequence indices.

thus, if you want to see the benefit of parallelization, you need to put a long sequence as query. e.g. generate one by repeated copy of a single sequence.

[Siddhant085]

I used 45 cases for target and one for the query. I will post a detailed analysis on the increase in performance by the end of the week.

[martin-raden]

as I said, if you use a short query sequence, there wont be much parallelization and performance gain...

[Siddhant085]

I will do the analysis with multiple query sequences.

[Siddhant085]

@martin-mann Have a look at the graph. We can see that the difference between the serial and parallel execution time grows as number of targets and the length of query string increases.

[martin-raden]

Thanks, that gives a good feeling about the expected performance change via the parallelization. I am still astound that it changes only so little..

the very good point: one doesnt get slower by parallelization (which is possible)!

but since you are tweaking the already fast heuristic predictor, it seems there is not much to gain. With a similar parallelization of the exact computation, I would guess one would see a more serious impact.

anyway, thanks a lot for the investigation and work!

[Siddhant085]

The difference in the performance increase as the problem size increases. Will this solution get merged with you original code?