Failed to open -: unknown file type about gembs

[heathsc]

Hi,

Could you send me a few details about the issue. Are you running the test dataset? How did you install gemBS on your system?

Thanks,
Simon

[christacaggiano]

Hello,
Yes, this is running on the test dataset. I installed gemBS within a clean conda environment (python=3.6.6) on a Centos environment. The following packages and versions are in my conda environment.

# packages in environment at /ye/zaitlenlabstore/christacaggiano/miniconda3/envs/gembs:
#
# Name                    Version                   Build  Channel
blas                      1.0                         mkl
ca-certificates           2018.11.29           ha4d7672_0    conda-forge
certifi                   2018.11.29            py36_1000    conda-forge
cycler                    0.10.0                   py36_0
cycler                    0.10.0                    <pip>
dbus                      1.13.2               h714fa37_1
dill                      0.2.8.2               py36_1000    conda-forge
expat                     2.2.6                he6710b0_0
fontconfig                2.13.0               h9420a91_0
freetype                  2.9.1                h8a8886c_1
gemBS                     3.2.1                     <pip>
glib                      2.56.2               hd408876_0
gsl                       2.4                  h14c3975_4
gst-plugins-base          1.14.0               hbbd80ab_1
gstreamer                 1.14.0               hb453b48_1
icu                       58.2                 h9c2bf20_1
intel-openmp              2019.1                      144
jpeg                      9b                   h024ee3a_2
kiwisolver                1.0.1            py36hf484d3e_0
kiwisolver                1.0.1                     <pip>
libedit                   3.1.20170329         h6b74fdf_2
libffi                    3.2.1                hd88cf55_4
libgcc-ng                 8.2.0                hdf63c60_1
libgfortran               3.0.0                         1    conda-forge
libgfortran-ng            7.3.0                hdf63c60_0
libpng                    1.6.35               hbc83047_0
libstdcxx-ng              8.2.0                hdf63c60_1
libuuid                   1.0.3                h1bed415_2
libxcb                    1.13                 h1bed415_1
libxml2                   2.9.8                h26e45fe_1
matplotlib                3.0.2            py36h5429711_0
matplotlib                3.0.2                     <pip>
mkl                       2019.1                      144
mkl_fft                   1.0.6            py36hd81dba3_0
mkl_random                1.0.2            py36hd81dba3_0
multiprocess              0.70.6.1                  <pip>
multiprocess              0.70.6.1         py36h470a237_2    conda-forge
ncurses                   6.1                  he6710b0_1
numpy                     1.15.4           py36h7e9f1db_0
numpy                     1.15.4                    <pip>
numpy-base                1.15.4           py36hde5b4d6_0
openblas                  0.3.3                ha44fe06_1    conda-forge
openssl                   1.0.2p               h470a237_1    conda-forge
pcre                      8.42                 h439df22_0
pip                       18.1                     py36_0
pyparsing                 2.3.0                    py36_0
pyqt                      5.9.2            py36h05f1152_2
python                    3.6.6                h5001a0f_3    conda-forge
python-dateutil           2.7.5                    py36_0
pytz                      2018.7                   py36_0
qt                        5.9.6                h8703b6f_2
readline                  7.0                  h7b6447c_5
setuptools                40.6.3                   py36_0
sip                       4.19.8           py36hf484d3e_0
six                       1.12.0                    <pip>
six                       1.12.0                   py36_0
sqlite                    3.26.0               h7b6447c_0
tk                        8.6.8                hbc83047_0
tornado                   5.1.1            py36h7b6447c_0
wheel                     0.32.3                   py36_0
xz                        5.2.4                h14c3975_4
zlib                      1.2.11               h7b6447c_3

I installed gemBS specifically by running
python3 setup.py install --user
I did not specify a gsl path since gsl is installed with my conda environment. Bscall seems to exist and be properly compiled in the right path /ye/zaitlenlabstore/christacaggiano/miniconda3/envs/gembs/lib/python3.6/site-packages/gemBS/bin

Do you notice anything else that could be causing the problem? I do not have sudo access on this server.

[karl616]

I get the same error running the singularity container. For me it seems tied to the machine I'm using. When it fails it generates this output:

stderr.txt
stdout.txt

I should say that this is not the test data, but I'm perplexed as it works on one system.

[heathsc]

Can you send me the output you get from running: gemBS --loglevel debug call Thanks

…

On 19 Mar 2019, at 12:49, Karl Nordström ***@***.***> wrote: stdout.txt <https://github.com/heathsc/gemBS/files/2983088/stdout.txt>

[karl616]

Of course:

debug.txt

I can see the command that fails. I will try it step by step...

[heathsc]

You seem to be triggering a bug in bs_call. If you could find a BAM (or a short fragment of a BAM) that triggers the problem and share it with me then I will be able to track this down. Simon

…

On 19 Mar 2019, at 13:32, Karl Nordström ***@***.***> wrote: Of course: debug.txt <https://github.com/heathsc/gemBS/files/2983281/debug.txt> I can see the command that fails. I will try it step by step... — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#42 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADHPd1iuQGhBNqpfdZ-y1FyHIFp4hNt4ks5vYNjDgaJpZM4ZXfS_>.

[karl616]

I will do that, but it will take a little longer... we deleted the alignments here by mistake... I will regenerate them...

Another thought, can something happen when loading the reference? When just running the job up until bs_call, I had the feeling it didn't get past this step.

[heathsc]

Aside from running out of memory while reading in the reference I can’t see too many places where it could fail. Simon

…

On 19 Mar 2019, at 13:56, Karl Nordström ***@***.***> wrote: I will do that, but it will take a little longer... we deleted the alignments here by mistake... I will regenerate them... Another thought, can something happen when loading the reference? When just running the job up until bs_call, I had the feeling it didn't get past this step. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#42 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADHPdwydhD9F3dodXTDp_sRoYlHjahILks5vYN5wgaJpZM4ZXfS_>.

[karl616]

OK, here are the first 1000 rows from a file that fails:

test.sam.txt

Inside the container this command fails:

cat test.sam.txt | /usr/local/lib/python3.5/dist-packages/gemBS/gemBSbinaries/bs_call -r /vol/data/reference/mm10/wgbs/mm10_no_alt_analysis_set_ENCODE.fasta.gz -n liver_late_2 --contig-bed /vol/data/test_WGBS/output/calls/liver_late_2/contigs_liver_late_2_chr18.bed --report-file /vol/data/test_WGBS/output/calls/liver_late_2/liver_late_2_chr18.json --right-trim 0 --left-trim 5 --conversion 0.0100,0.0500 --reference-bias 2 --mapq-threshold 10 --bq-threshold 13

but not on all machines.

[Smart-zhi]

Hello, I had the same problem. Has this problem been solved? If it possible please tell me how to solve it.

[heathsc]

Good morning, Can you confirm the version of gemBS that you are using, and at what stage of the process you see the problem? Thanks, Simon

…

On 25 Jun 2019, at 10:23, Smart-zhi ***@***.***> wrote: Hello, I had the same problem. Has this problem been solved? If it possible please tell me how to solve it. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#42?email_source=notifications&email_token=AAY465ZSA5VT7K2AELWJN3TP4HIY3A5CNFSM4GK56S72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYPOMBI#issuecomment-505341445>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAY4654XLUG4NBTPBVTYYKTP4HIY3ANCNFSM4GK56S7Q>.

[Smart-zhi]

Hello Simon,
My apologies for the delay in responding. I'm using gemBS 3.3.0.
When I ran my data, I got AttributeError: 'list' object has no attribute 'lower' by using gemBS call. Then, I used gemBS call -C 0.01,0.05 ,and I got this issue：

: 
: Command call started at 2019-07-02 22:01:30.985422
: 
: ----------- Methylation Calling --------
: Reference       : reference/Arabidopsis_thaliana.fa
: Species         : Arabidopsis
: Right Trim      : 0
: Left Trim       : 5
: Chromosomes     : ['NC_003070.9', 'NC_003076.8', '@pool_1', '@pool_2', '@pool_3']
: Threads         : 12
: Sample: A1    Bam: ./mapping/A1.bam
: 
: Methylation Calling...
2019-07-02 22:01:31,144 ERROR: Process '/home/usr1/.local/lib/python3.7/site-packages/gemBS/bin/bcftools' finished with 255
2019-07-02 22:01:31,144 ERROR: Failed to open -: unknown file type
2019-07-02 22:01:31,144 ERROR: guments)
2019-07-02 22:01:31,144 ERROR: c (conversion) option requires two comma separated arguments
Exception in thread Thread-1:
Traceback (most recent call last):
ValueError: Error while executing the bscall process.

2019-07-02 22:01:31,151 ERROR: Process '/home/usr1/.local/lib/python3.7/site-packages/gemBS/bin/bcftools' finished with 255
2019-07-02 22:01:31,151 ERROR: Failed to open -: unknown file type
2019-07-02 22:01:31,151 ERROR: guments)
2019-07-02 22:01:31,151 ERROR: c (conversion) option requires two comma separated arguments
Exception in thread Thread-2:
Traceback (most recent call last):
ValueError: Error while executing the bscall process.

2019-07-02 22:01:31,411 ERROR: Process '/home/usr1/.local/lib/python3.7/site-packages/gemBS/bin/bcftools' finished with 255
2019-07-02 22:01:31,412 ERROR: Failed to open -: unknown file type
Exception in thread Thread-3:
Traceback (most recent call last):
ValueError: Error while executing the bscall process.

2019-07-02 22:01:31,419 ERROR: Process '/home/usr1/.local/lib/python3.7/site-packages/gemBS/bin/bcftools' finished with 255
2019-07-02 22:01:31,419 ERROR: Failed to open -: unknown file type
2019-07-02 22:01:31,419 ERROR: guments)
2019-07-02 22:01:31,420 ERROR: c (conversion) option requires two comma separated arguments
Exception in thread Thread-4:
Traceback (most recent call last):
ValueError: Error while executing the bscall process.

: Methylation call done, samples performed: A1

Thanks,
Zhi

[Smart-zhi]

Hello Simon, I have solved my problem. I modified gemBS.json in ./.gemBS/ .

{
  "config": {

    "calling": {
      "conversion": "0.01,0.05",
    },

  }
}

[heathsc]

Thank you - this is a work around (because if you specify the conversion in the config file, you don’t need to set it in on the command line), but I will fix the original problem as well. Thanks for the report. Simon

…

On 3 Jul 2019, at 08:49, Smart-zhi ***@***.***> wrote: Hello Simon, I have solved my problem. I modified gemBS.json in ./.gemBS/ . { "config": { "calling": { "conversion": "0.01,0.05", }, } } — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#42?email_source=notifications&email_token=AAY4657W365ROF7SYH7QYZTP5RDYNA5CNFSM4GK56S72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZDOWII#issuecomment-507964193>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAY46536TA4S44LFW4B5WKTP5RDYNANCNFSM4GK56S7Q>.

[heathsc]

Fixed in 3.3