"buffer overflow detected" when using psipred. about rosettafold

[li-ziang]

I found that the bug occurs at https://github.com/psipred/psipred/blob/master/src/sspred_avpred.c#L241, buffer size is 256 when I change it to 512, it can run, but I got the following output which seems not correct:

>ss_pred
CCCCCCCCCCCCCCCCCCCCCEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEECCEECCCCCCCCCCCCCCCC
>ss_conf
572999999899030000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000999998999999

Could you give me an example of the t000_.msa0.tmp.mtx file in step 2?

Here is mine:t000_.msa0.tmp.mtx.txt

Line 15 and the following lines of the file seems to be wrong because there are many -2147483648? Is that true?

[qiyubio]

The working around works for me is by adding
export BLASTMAT=your_path_to/RoseTTAFold/your_conda_env/share/blast-2.2.26/data/ to input_prep/make_ss.sh

Without it, makemat will generate the wrong size of mtx file.
makemat -P t000_.msa0.tmp
44K Jul 21 13:50 t000_.msa0.tmp.mtx
19 Jul 21 13:50 t000_.msa0.tmp.mn
74 Jul 21 13:50 t000_.msa0.tmp.aux

With BLASTMAT:
24K Jul 21 13:44 t000_.msa0.tmp.mtx
19 Jul 21 13:44 t000_.msa0.tmp.mn
74 Jul 21 13:44 t000_.msa0.tmp.aux

[xubeisi]

The working around works for me is by adding
export BLASTMAT=your_path_to/RoseTTAFold/your_conda_env/share/blast-2.2.26/data/ to input_prep/make_ss.sh

Without it, makemat will generate the wrong size of mtx file.
makemat -P t000_.msa0.tmp
44K Jul 21 13:50 t000_.msa0.tmp.mtx
19 Jul 21 13:50 t000_.msa0.tmp.mn
74 Jul 21 13:50 t000_.msa0.tmp.aux

With BLASTMAT:
24K Jul 21 13:44 t000_.msa0.tmp.mtx
19 Jul 21 13:44 t000_.msa0.tmp.mn
74 Jul 21 13:44 t000_.msa0.tmp.aux

this works for me too,
you could either
conda install --force-reinstall biocore::blast-legacy=2.2.26

or used "data" in

https://ftp.ncbi.nlm.nih.gov/blast/executables/legacy.NOTSUPPORTED/2.2.26/blast-2.2.26-x64-linux.tar.gz

thanks!

[minkbaek]

Could you provide more details for your machine environment? We used gcc 9.3.0 (see below)

gcc --version
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[moonblue333]

This version is OK for me, @ubuntu 18.0.4.
wget https://github.com/soedinglab/hh-suite/releases/download/v3.2.0/hhsuite-3.2.0-AVX2-Linux.tar.gz --no-check-certificate
tar xvfz hhsuite-3.2.0-AVX2-Linux.tar.gz

[moonblue333]

hhsuite-3.3.0 is NOT OK.

[li-ziang]

Hi there, I had the same problem as @panganqi ,
2920146 Aborted (core dumped) psipred $ID.mtx $DATADIR/weights.dat $DATADIR/weights.dat2 $DATADIR/weights.dat3 > $ID.ss
I installed hhsuite by using:

$ wget https://github.com/soedinglab/hh-suite/releases/download/v3.2.0/hhsuite-3.2.0-AVX2-Linux.tar.gz --no-check-certificate
$ tar xvfz hhsuite-3.2.0-AVX2-Linux.tar.gz
$ export PATH="$(pwd)/bin:$(pwd)/scripts:$PATH"

Here is my machine environment:

$ hhblits
HHblits 3.1.0:
HMM-HMM-based lightning-fast iterative sequence search
HHblits is a sensitive, general-purpose, iterative sequence search tool that represents
both query and database sequences by HMMs. You can search HHblits databases starting
with a single query sequence, a multiple sequence alignment (MSA), or an HMM. HHblits
prints out a ranked list of database HMMs/MSAs and can also generate an MSA by merging
the significant database HMMs/MSAs onto the query MSA.

$ gcc --version
gcc (Ubuntu 9.4.0-1ubuntu1~18.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

I have already tried to use the file in this repo (https://github.com/RosettaCommons/RoseTTAFold/blob/main/example/pyrosetta/t000_.msa0.a3m) as the input of step 2 in https://github.com/RosettaCommons/RoseTTAFold/blob/main/run_pyrosetta_ver.sh#L44, but still got the same result.

It seems that it is the psipred package which goes wrong, according to the things front.

Can you offer some advice or help @minkbaek?
Thank you!

[minkbaek]

Not sure what exactly causes the problem here as it works fine for my Linux machine. One thing you can try is to install psipred from the source code (https://github.com/psipred/psipred). For easy-install, I included conda installation for psipred, but it might be not compatible with your Linux environment.

[andrewcchang]

I am facing the same 'buffer overflow' error while running PSIPRED. I did try installing from the source code with no success.

Just curious, is this secondary structure info crucial to the prediction, can we ignore it and just use the MSA from hhblits? If we still have no luck getting psipred to work, maybe we can ignore this step?

Thank you!

[li-ziang]

I have already tried to install psipred from the source code (https://github.com/psipred/psipred), but it did not succeed either. I am using ubuntu1~18.04, so this means that maybe I can try ubuntu1~20.04?

[pp3ndwich]

In my case, updating Ubuntu from 18.04 to 20.04 perfectly solved the buffer overflow problem. Guess the latest version of Ubuntu and gcc somehow would not encounter this problem. Hope this helps.

[cuiyoutian]

I have the same issue. I am using Ubuntu 16.04.7 and gcc 5.4.0. I am using HHblits 3.3.0.

[li-ziang]

In my case, updating Ubuntu from 18.04 to 20.04 perfectly solved the buffer overflow problem. Guess the latest version of Ubuntu and gcc somehow would not encounter this problem. Hope this helps.
@pp3ndwich

That would be perfect!
Could you please give your gcc version too? I will try to find a docker to solve the problem.
By the way, is HHblits of version 3.3.0 installed by conda ok?
THANKS IN ADVANCE!!!

[pp3ndwich]

In my case, updating Ubuntu from 18.04 to 20.04 perfectly solved the buffer overflow problem. Guess the latest version of Ubuntu and gcc somehow would not encounter this problem. Hope this helps.
@pp3ndwich

That would be perfect!
Could you please give your gcc version too? I will try to find a docker to solve the problem.
By the way, is HHblits of version 3.3.0 installed by conda ok?
THANKS IN ADVANCE!!!

Here is the gcc version on our computer:

gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0

This version of gcc is bundled with the latest version of Ubuntu.
I checked my version of hhblits, the same as yours, 3.3.0. The package is installed in the RoseTTAFold environment in conda, so if you do "conda install hhsuite" under the base environment, I guess the script would fail to detect the package. Anyway, I doubt hhblits has anything related to the problem.
Good luck!

[li-ziang]

Good news!
Thanks to the nvidia docker (nvidia/cuda:11.0.3-devel-ubuntu20.04), I perfectly solve the problem.
This docker has the right environment.
Many thanks again!!!

[pfreezer]

The working around works for me is by adding
export BLASTMAT=your_path_to/RoseTTAFold/your_conda_env/share/blast-2.2.26/data/ to input_prep/make_ss.sh

Without it, makemat will generate the wrong size of mtx file.
makemat -P t000_.msa0.tmp
44K Jul 21 13:50 t000_.msa0.tmp.mtx
19 Jul 21 13:50 t000_.msa0.tmp.mn
74 Jul 21 13:50 t000_.msa0.tmp.aux

With BLASTMAT:
24K Jul 21 13:44 t000_.msa0.tmp.mtx
19 Jul 21 13:44 t000_.msa0.tmp.mn
74 Jul 21 13:44 t000_.msa0.tmp.aux

Hi, could you specify what is "your_path_to" and "your_conda_env" here,
you_path_to is the path where I pull the code from github?
and "your_conda_env" is the name the conda environment, like RoseTTAFold or folding?
I tried this but it cannot solve the problem

[qiyubio]

The working around works for me is by adding
export BLASTMAT=your_path_to/RoseTTAFold/your_conda_env/share/blast-2.2.26/data/ to input_prep/make_ss.sh
Without it, makemat will generate the wrong size of mtx file.
makemat -P t000_.msa0.tmp
44K Jul 21 13:50 t000_.msa0.tmp.mtx
19 Jul 21 13:50 t000_.msa0.tmp.mn
74 Jul 21 13:50 t000_.msa0.tmp.aux
With BLASTMAT:
24K Jul 21 13:44 t000_.msa0.tmp.mtx
19 Jul 21 13:44 t000_.msa0.tmp.mn
74 Jul 21 13:44 t000_.msa0.tmp.aux

Hi, could you specify what is "your_path_to" and "your_conda_env" here,
you_path_to is the path where I pull the code from github?
and "your_conda_env" is the name the conda environment, like RoseTTAFold or folding?
I tried this but it cannot solve the problem

The path to your conda env, if you didn't change anything in the yaml file, then "RoseTTAFold" env. blast should be installed already in the environment:

RoseTTAFold/RoseTTAFold-linux.yml

Line 12 in 5510443

- biocore::blast-legacy=2.2.26

.

[pfreezer]

The working around works for me is by adding
export BLASTMAT=your_path_to/RoseTTAFold/your_conda_env/share/blast-2.2.26/data/ to input_prep/make_ss.sh
Without it, makemat will generate the wrong size of mtx file.
makemat -P t000_.msa0.tmp
44K Jul 21 13:50 t000_.msa0.tmp.mtx
19 Jul 21 13:50 t000_.msa0.tmp.mn
74 Jul 21 13:50 t000_.msa0.tmp.aux
With BLASTMAT:
24K Jul 21 13:44 t000_.msa0.tmp.mtx
19 Jul 21 13:44 t000_.msa0.tmp.mn
74 Jul 21 13:44 t000_.msa0.tmp.aux

Hi, could you specify what is "your_path_to" and "your_conda_env" here,
you_path_to is the path where I pull the code from github?
and "your_conda_env" is the name the conda environment, like RoseTTAFold or folding?
I tried this but it cannot solve the problem

The path to your conda env, if you didn't change anything in the yaml file, then "RoseTTAFold" env. blast should be installed already in the environment:

RoseTTAFold/RoseTTAFold-linux.yml

Line 12 in 5510443

- biocore::blast-legacy=2.2.26

.

Thanks, it turns out if you use RoseTTAFold/RoseTTAFold-linux-cu101.yml instead of RoseTTAFold/RoseTTAFold-linux.yml, you can not find $CONDA_PREFIX/env/RoseTTAFold/share/blast-2.2.26/data/, but if I download the package mention by xubeisi,
it works well, thanks @xubeisi

Overall, the problem is not related with psipred but makemat, with BLASTMAT defined, correct mtx file can be generated and so psipred can work well.

[thuxugang]

The working around works for me is by adding
export BLASTMAT=your_path_to/RoseTTAFold/your_conda_env/share/blast-2.2.26/data/ to input_prep/make_ss.sh
Without it, makemat will generate the wrong size of mtx file.
makemat -P t000_.msa0.tmp
44K Jul 21 13:50 t000_.msa0.tmp.mtx
19 Jul 21 13:50 t000_.msa0.tmp.mn
74 Jul 21 13:50 t000_.msa0.tmp.aux
With BLASTMAT:
24K Jul 21 13:44 t000_.msa0.tmp.mtx
19 Jul 21 13:44 t000_.msa0.tmp.mn
74 Jul 21 13:44 t000_.msa0.tmp.aux

this works for me too,
you could either
conda install --force-reinstall biocore::blast-legacy=2.2.26

or used "data" in

https://ftp.ncbi.nlm.nih.gov/blast/executables/legacy.NOTSUPPORTED/2.2.26/blast-2.2.26-x64-linux.tar.gz

thanks!

works for me, thanks!

[wmustafaAwad]

Running conda install --force-reinstall biocore::blast-legacy=2.2.26 as suggested by xubeisi did it for me.

[BJWiley233]

No. All wrong :0. This is because we all followed the install.md from this post. It indicates to switch NCBI's default environmental variable $BLASTMAT to point to the csblast database under step 16. This is incorrect. Just look at the ouput of $ID.mtx when you do this. It looks like this below and the bits overload the program. makemat is a C program made by NCBI and it uses the BLASTMAT environmental variable to get the BLOSUM/PAM matrices. See section 10.2.4 Environment Variables

-32768  -2147483648  -32768  -2147483648  -2147483648  -2147483648  -2147483648  -2147483648  -2147483648  -2147483648  -2147483648  -2147483648

To test this, deactivate with conda deactivate and then conda activate RoseTTAFold

$ echo $BLASTMAT
/opt/conda/envs/RoseTTAFold/share/blast-2.2.26/data/

TADA!!!!