[Repeat] Running deepconsensus results in "free(): invalid pointer" error about deepconsensus

[pichuan]

Got it. Thanks for the context @mjg0 .
Given that there's a workaround, let's close this issue for now.
But, if you do find out a solution, please give an update here because I'm sure some future users can benefit from it.

[pichuan]

Hi @mjg0
Sorry that it seems like we have missed this issue. We'll try to take a look next week.

[akolesnikov]

Hi @mjg0

We are trying to reproduce the issue. So far we were not able to reproduce it with the latest deepvariant 1.1.0 pip installtion.

Here are my steps:

# Create a VM
gcloud compute instances create "${USER}-cpu" --scopes "compute-rw,storage-full,cloud-platform" --image-family "ubuntu-2004-lts" --image-project "ubuntu-os-cloud" --machine-type "custom-64-131072" --boot-disk-size "300" --zone "us-west2-b" --min-cpu-platform "Intel Skylake"

# SSH to the new VM:
gcloud compute ssh "${USER}-cpu" --zone us-west2-b

#  Install virtualenv and pip
sudo apt update -y && sudo apt install -y python3-virtualenv && sudo apt install -y python3-pip

# Staring a virtualenv
virtualenv github-test
source github-test/bin/activate

# Versions inside virtual env:
uname -a
Linux koles-dc-pip-issue 5.15.0-1027-gcp #34~20.04.1-Ubuntu SMP Mon Jan 9 18:40:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

python3 --version
Python 3.8.10

pip --version
pip 23.0 from /home/koles/github-test/lib/python3.8/site-packages/pip (python 3.8)

# Install DeepConsensus
pip install deepconsensus[cpu]==1.1.0

pip list | grep tensorflow
intel_tensorflow              2.11.0
tensorflow                    2.11.0
tensorflow-addons             0.19.0
tensorflow-datasets           4.8.2
tensorflow-estimator          2.11.0
tensorflow-hub                0.12.0
tensorflow-io-gcs-filesystem  0.30.0
tensorflow-metadata           1.12.0
tensorflow-model-optimization 0.7.3
tensorflow-text               2.11.0

# Run DeepConsensus
deepconsensus -h

In order to reproduce the issue we will need more information. We can try to reproduce the exact environment you have. The only difference I see now is that you use Python 3.8.2 vs ours' 3.8.10.

[mjg0]

@akolesnikov sorry, I guess I neglected to even specify my OS--I'm on RHEL 7.9. I used Charliecloud and CentOS 7 to replicate the issue, I'm guessing the same can be done on gcloud.

The Dockerfile:

FROM centos:7

# Install Python 3.8
RUN yum install -y gcc openssl-devel libffi-devel bzip2-devel wget make
RUN wget https://www.python.org/ftp/python/3.8.10/Python-3.8.10.tgz
RUN tar xf Python-3.8.10.tgz
WORKDIR Python-3.8.10
RUN ./configure
RUN make -j altinstall

# Create venv
RUN python3.8 -m venv /dcvenv
RUN source /dcvenv/bin/activate && \
    pip install deepconsensus[cpu]==1.1.0

Create the container and trigger the problem:

$ ch-image build -t dctest . &>/dev/null # Dockerfile is in this directory
$ ch-convert dctest ./dctest &>/dev/null # image to directory
$ ch-run dctest -- bash -c "source /dcvenv/bin/activate && deepconsensus -h"
2023-02-17 23:12:46.298229: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
*** Error in `/dcvenv/bin/python3.8': free(): invalid pointer: 0x00007f117eb5bc80 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81329)[0x7f11a90f0329]
/lib64/libstdc++.so.6(_ZNSt6locale5_Impl16_M_install_facetEPKNS_2idEPKNS_5facetE+0x142)[0x7f117e8c5192]
/lib64/libstdc++.so.6(_ZNSt6locale5_ImplC1Em+0x1e3)[0x7f117e8c55e3]
/lib64/libstdc++.so.6(+0x71555)[0x7f117e8c6555]
/lib64/libpthread.so.0(+0x620b)[0x7f11a9b4c20b]
/lib64/libstdc++.so.6(+0x715a1)[0x7f117e8c65a1]
/lib64/libstdc++.so.6(_ZNSt6localeC2Ev+0x13)[0x7f117e8c65e3]
/lib64/libstdc++.so.6(_ZNSt8ios_base4InitC2Ev+0xbc)[0x7f117e8c343c]
/dcvenv/lib/python3.8/site-packages/google/protobuf/pyext/_message.cpython-38-x86_64-linux-gnu.so(+0xb1150)[0x7f117e695150]
/lib64/ld-linux-x86-64.so.2(+0xf9c3)[0x7f11a9fa89c3]
/lib64/ld-linux-x86-64.so.2(+0x1459e)[0x7f11a9fad59e]
/lib64/ld-linux-x86-64.so.2(+0xf7d4)[0x7f11a9fa87d4]
/lib64/ld-linux-x86-64.so.2(+0x13b8b)[0x7f11a9facb8b]
/lib64/libdl.so.2(+0xfab)[0x7f11a9942fab]
/lib64/ld-linux-x86-64.so.2(+0xf7d4)[0x7f11a9fa87d4]
/lib64/libdl.so.2(+0x15ad)[0x7f11a99435ad]
/lib64/libdl.so.2(dlopen+0x31)[0x7f11a9943041]
/dcvenv/bin/python3.8(_PyImport_FindSharedFuncptr+0x17b)[0x55205b]
/dcvenv/bin/python3.8(_PyImport_LoadDynamicModuleWithSpec+0x170)[0x510ce0]
/dcvenv/bin/python3.8[0x50db63]
/dcvenv/bin/python3.8[0x472260]
/dcvenv/bin/python3.8(PyVectorcall_Call+0x5c)[0x43b9bc]
/dcvenv/bin/python3.8(_PyEval_EvalFrameDefault+0x95df)[0x42bd8f]
/dcvenv/bin/python3.8(_PyEval_EvalCodeWithName+0x85d)[0x4ec1bd]
/dcvenv/bin/python3.8(_PyFunction_Vectorcall+0xb2)[0x4397d2]
/dcvenv/bin/python3.8(_PyEval_EvalFrameDefault+0x6f76)[0x429726]
/dcvenv/bin/python3.8[0x4215b8]
/dcvenv/bin/python3.8(_PyEval_EvalFrameDefault+0x1b83)[0x424333]
/dcvenv/bin/python3.8[0x4215b8]
/dcvenv/bin/python3.8(_PyEval_EvalFrameDefault+0x30ff)[0x4258af]
/dcvenv/bin/python3.8[0x4215b8]
/dcvenv/bin/python3.8(_PyEval_EvalFrameDefault+0x30ff)[0x4258af]
/dcvenv/bin/python3.8[0x4215b8]
/dcvenv/bin/python3.8(_PyEval_EvalFrameDefault+0x30ff)[0x4258af]
/dcvenv/bin/python3.8[0x4215b8]
/dcvenv/bin/python3.8[0x439088]
/dcvenv/bin/python3.8(_PyObject_CallMethodIdObjArgs+0xe5)[0x43b3d5]
/dcvenv/bin/python3.8(PyImport_ImportModuleLevelObject+0x4cd)[0x50f4ed]
/dcvenv/bin/python3.8[0x61beeb]
/dcvenv/bin/python3.8(PyCFunction_Call+0x13a)[0x43bdca]
/dcvenv/bin/python3.8(_PyEval_EvalFrameDefault+0x95df)[0x42bd8f]
/dcvenv/bin/python3.8(_PyEval_EvalCodeWithName+0x85d)[0x4ec1bd]
/dcvenv/bin/python3.8(_PyFunction_Vectorcall+0xb2)[0x4397d2]
/dcvenv/bin/python3.8(_PyEval_EvalFrameDefault+0x30ff)[0x4258af]
/dcvenv/bin/python3.8(_PyEval_EvalCodeWithName+0x85d)[0x4ec1bd]
/dcvenv/bin/python3.8(_PyFunction_Vectorcall+0xb2)[0x4397d2]
/dcvenv/bin/python3.8[0x439088]
/dcvenv/bin/python3.8(_PyObject_CallMethodIdObjArgs+0xe5)[0x43b3d5]
/dcvenv/bin/python3.8(PyImport_ImportModuleLevelObject+0x2a5)[0x50f2c5]
/dcvenv/bin/python3.8(_PyEval_EvalFrameDefault+0x76af)[0x429e5f]
/dcvenv/bin/python3.8(PyEval_EvalCode+0x458)[0x4eb8f8]
/dcvenv/bin/python3.8[0x61cdbd]
/dcvenv/bin/python3.8[0x472260]
/dcvenv/bin/python3.8(PyVectorcall_Call+0x5c)[0x43b9bc]
/dcvenv/bin/python3.8(_PyEval_EvalFrameDefault+0x95df)[0x42bd8f]
/dcvenv/bin/python3.8(_PyEval_EvalCodeWithName+0x85d)[0x4ec1bd]
/dcvenv/bin/python3.8(_PyFunction_Vectorcall+0xb2)[0x4397d2]
/dcvenv/bin/python3.8(_PyEval_EvalFrameDefault+0x6f76)[0x429726]
/dcvenv/bin/python3.8[0x4215b8]
/dcvenv/bin/python3.8(_PyEval_EvalFrameDefault+0x1b83)[0x424333]
/dcvenv/bin/python3.8[0x4215b8]
/dcvenv/bin/python3.8(_PyEval_EvalFrameDefault+0x30ff)[0x4258af]
/dcvenv/bin/python3.8[0x4215b8]
======= Memory map: ========
00400000-00710000 r-xp 00000000 fd:03 4328802775                         /usr/local/bin/python3.8
0090f000-00910000 r--p 0030f000 fd:03 4328802775                         /usr/local/bin/python3.8
00910000-00948000 rw-p 00310000 fd:03 4328802775                         /usr/local/bin/python3.8
00948000-0096a000 rw-p 00000000 00:00 0 
01eb1000-032cd000 rw-p 00000000 00:00 0                                  [heap]
7f1178000000-7f1178021000 rw-p 00000000 00:00 0 
7f1178021000-7f117c000000 ---p 00000000 00:00 0 
7f117e5e4000-7f117e681000 r--p 00000000 fd:03 6477225674                 /dcvenv/lib/python3.8/site-packages/google/protobuf/pyext/_message.cpython-38-x86_64-linux-gnu.so
7f117e681000-7f117e7d0000 r-xp 0009d000 fd:03 6477225674                 /dcvenv/lib/python3.8/site-packages/google/protobuf/pyext/_message.cpython-38-x86_64-linux-gnu.so
7f117e7d0000-7f117e844000 r--p 001ec000 fd:03 6477225674                 /dcvenv/lib/python3.8/site-packages/google/protobuf/pyext/_message.cpython-38-x86_64-linux-gnu.so
7f117e844000-7f117e845000 ---p 00260000 fd:03 6477225674                 /dcvenv/lib/python3.8/site-packages/google/protobuf/pyext/_message.cpython-38-x86_64-linux-gnu.so
7f117e845000-7f117e84a000 r--p 00260000 fd:03 6477225674                 /dcvenv/lib/python3.8/site-packages/google/protobuf/pyext/_message.cpython-38-x86_64-linux-gnu.so
7f117e84a000-7f117e854000 rw-p 00265000 fd:03 6477225674                 /dcvenv/lib/python3.8/site-packages/google/protobuf/pyext/_message.cpython-38-x86_64-linux-gnu.so
7f117e854000-7f117e855000 rw-p 00000000 00:00 0 
7f117e855000-7f117e93e000 r-xp 00000000 fd:03 53405456                   /usr/lib64/libstdc++.so.6.0.19
7f117e93e000-7f117eb3e000 ---p 000e9000 fd:03 53405456                   /usr/lib64/libstdc++.so.6.0.19
7f117eb3e000-7f117eb46000 r--p 000e9000 fd:03 53405456                   /usr/lib64/libstdc++.so.6.0.19
7f117eb46000-7f117eb48000 rw-p 000f1000 fd:03 53405456                   /usr/lib64/libstdc++.so.6.0.19
7f117eb48000-7f117ec18000 rw-p 00000000 00:00 0 
7f117ec18000-7f117ec50000 r--p 00000000 fd:03 4330264302                 /dcvenv/lib/python3.8/site-packages/_solib_k8/_U_S_Sthird_Uparty_Smkl_Cmkl_Ulibs_Ulinux___Uexternal_Sllvm_Uopenmp/libiomp5.so
7f117ec50000-7f117eccf000 r-xp 00037000 fd:03 4330264302                 /dcvenv/lib/python3.8/site-packages/_solib_k8/_U_S_Sthird_Uparty_Smkl_Cmkl_Ulibs_Ulinux___Uexternal_Sllvm_Uopenmp/libiomp5.so
7f117eccf000-7f117ecd2000 r--p 000b5000 fd:03 4330264302                 /dcvenv/lib/python3.8/site-packages/_solib_k8/_U_S_Sthird_Uparty_Smkl_Cmkl_Ulibs_Ulinux___Uexternal_Sllvm_Uopenmp/libiomp5.so
7f117ecd2000-7f117ecd6000 rw-p 000b7000 fd:03 4330264302                 /dcvenv/lib/python3.8/site-packages/_solib_k8/_U_S_Sthird_Uparty_Smkl_Cmkl_Ulibs_Ulinux___Uexternal_Sllvm_Uopenmp/libiomp5.so
7f117ecd6000-7f117ecda000 rw-p 00000000 00:00 0 
7f117ecda000-7f117f7b5000 r--p 00000000 fd:03 4329090168                 /dcvenv/lib/python3.8/site-packages/tensorflow/libtensorflow_framework.so.2
7f117f7b5000-7f1180e7d000 r-xp 00ada000 fd:03 4329090168                 /dcvenv/lib/python3.8/site-packages/tensorflow/libtensorflow_framework.so.2
7f1180e7d000-7f1180f18000 r--p 021a1000 fd:03 4329090168                 /dcvenv/lib/python3.8/site-packages/tensorflow/libtensorflow_framework.so.2
7f1180f18000-7f1180f63000 rw-p 0223b000 fd:03 4329090168                 /dcvenv/lib/python3.8/site-packages/tensorflow/libtensorflow_framework.so.2
7f1180f63000-7f1180f7e000 rw-p 00000000 00:00 0 
7f1180f7e000-7f11879af000 r--p 00000000 fd:03 4329086998                 /dcvenv/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so
7f11879af000-7f1199988000 r-xp 06a30000 fd:03 4329086998                 /dcvenv/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so
7f1199988000-7f1199f5c000 r--p 18a08000 fd:03 4329086998                 /dcvenv/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so
7f1199f5c000-7f1199f8a000 rw-p 18fdb000 fd:03 4329086998                 /dcvenv/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so
7f1199f8a000-7f119a046000 rw-p 00000000 00:00 0 
7f119a046000-7f119a08a000 r--p 00000000 fd:03 2185801691                 /dcvenv/lib/python3.8/site-packages/tensorflow/python/platform/_pywrap_cpu_feature_guard.so
7f119a08a000-7f119a134000 r-xp 00043000 fd:03 2185801691                 /dcvenv/lib/python3.8/site-packages/tensorflow/python/platform/_pywrap_cpu_feature_guard.so
7f119a134000-7f119a13d000 r--p 000ec000 fd:03 2185801691                 /dcvenv/lib/python3.8/site-packages/tensorflow/python/platform/_pywrap_cpu_feature_guard.so
7f119a13d000-7f119a13e000 rw-p 000f4000 fd:03 2185801691                 /dcvenv/lib/python3.8/site-packages/tensorflow/python/platform/_pywrap_cpu_feature_guard.so
7f119a13e000-7f119a183000 rw-p 00000000 00:00 0 
7f119a183000-7f119a194000 r-xp 00000000 fd:03 6477885742                 /dcvenv/lib/python3.8/site-packages/pysam/libcbgzf.cpython-38-x86_64-linux-gnu.sobash: line 1: 191833 Aborted                 deepconsensus -h

[pichuan]

@mjg0 Now that 1.2.0 is out, can you give it a try and let us know you're still having an issue with 1.2?

[mjg0]

The same issue occurs with 1.2.0; when I run the same steps as above, replacing only one line in the Dockerfile:

$ diff Dockerfile Dockerfile.orig
14c14
<     pip install deepconsensus[cpu]==1.2.0
---
>     pip install deepconsensus[cpu]==1.1.0

...a similar error message is printed:

*** Error in `/dcvenv/bin/python3.8': free(): invalid pointer: 0x00007f7d984c6c80 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81329)[0x7f7dc1704329]
/lib64/libstdc++.so.6(_ZNSt6locale5_Impl16_M_install_facetEPKNS_2idEPKNS_5facetE+0x142)[0x7f7d98230192]
/lib64/libstdc++.so.6(_ZNSt6locale5_ImplC1Em+0x1e3)[0x7f7d982305e3]
/lib64/libstdc++.so.6(+0x71555)[0x7f7d98231555]
/lib64/libpthread.so.0(+0x620b)[0x7f7dc216020b]
/lib64/libstdc++.so.6(+0x715a1)[0x7f7d982315a1]
/lib64/libstdc++.so.6(_ZNSt6localeC2Ev+0x13)[0x7f7d982315e3]
/lib64/libstdc++.so.6(_ZNSt8ios_base4InitC2Ev+0xbc)[0x7f7d9822e43c]
/dcvenv/lib/python3.8/site-packages/google/protobuf/pyext/_message.cpython-38-x86_64-linux-gnu.so(+0xb1150)[0x7f7d98000150]
/lib64/ld-linux-x86-64.so.2(+0xf9c3)[0x7f7dc25bc9c3]
/lib64/ld-linux-x86-64.so.2(+0x1459e)[0x7f7dc25c159e]
/lib64/ld-linux-x86-64.so.2(+0xf7d4)[0x7f7dc25bc7d4]
/lib64/ld-linux-x86-64.so.2(+0x13b8b)[0x7f7dc25c0b8b]
/lib64/libdl.so.2(+0xfab)[0x7f7dc1f56fab]
/lib64/ld-linux-x86-64.so.2(+0xf7d4)[0x7f7dc25bc7d4]
/lib64/libdl.so.2(+0x15ad)[0x7f7dc1f575ad]
/lib64/libdl.so.2(dlopen+0x31)[0x7f7dc1f57041]
/dcvenv/bin/python3.8(_PyImport_FindSharedFuncptr+0x17b)[0x55205b]
/dcvenv/bin/python3.8(_PyImport_LoadDynamicModuleWithSpec+0x170)[0x510ce0]
/dcvenv/bin/python3.8[0x50db63]
/dcvenv/bin/python3.8[0x472260]
/dcvenv/bin/python3.8(PyVectorcall_Call+0x5c)[0x43b9bc]
/dcvenv/bin/python3.8(_PyEval_EvalFrameDefault+0x95df)[0x42bd8f]
/dcvenv/bin/python3.8(_PyEval_EvalCodeWithName+0x85d)[0x4ec1bd]
/dcvenv/bin/python3.8(_PyFunction_Vectorcall+0xb2)[0x4397d2]
/dcvenv/bin/python3.8(_PyEval_EvalFrameDefault+0x6f76)[0x429726]
/dcvenv/bin/python3.8[0x4215b8]
/dcvenv/bin/python3.8(_PyEval_EvalFrameDefault+0x1b83)[0x424333]
/dcvenv/bin/python3.8[0x4215b8]
/dcvenv/bin/python3.8(_PyEval_EvalFrameDefault+0x30ff)[0x4258af]
/dcvenv/bin/python3.8[0x4215b8]
/dcvenv/bin/python3.8(_PyEval_EvalFrameDefault+0x30ff)[0x4258af]
/dcvenv/bin/python3.8[0x4215b8]
/dcvenv/bin/python3.8(_PyEval_EvalFrameDefault+0x30ff)[0x4258af]
/dcvenv/bin/python3.8[0x4215b8]
/dcvenv/bin/python3.8[0x439088]
/dcvenv/bin/python3.8(_PyObject_CallMethodIdObjArgs+0xe5)[0x43b3d5]
/dcvenv/bin/python3.8(PyImport_ImportModuleLevelObject+0x4cd)[0x50f4ed]
/dcvenv/bin/python3.8[0x61beeb]
/dcvenv/bin/python3.8(PyCFunction_Call+0x13a)[0x43bdca]
/dcvenv/bin/python3.8(_PyEval_EvalFrameDefault+0x95df)[0x42bd8f]
/dcvenv/bin/python3.8(_PyEval_EvalCodeWithName+0x85d)[0x4ec1bd]
/dcvenv/bin/python3.8(_PyFunction_Vectorcall+0xb2)[0x4397d2]
/dcvenv/bin/python3.8(_PyEval_EvalFrameDefault+0x30ff)[0x4258af]
/dcvenv/bin/python3.8(_PyEval_EvalCodeWithName+0x85d)[0x4ec1bd]
/dcvenv/bin/python3.8(_PyFunction_Vectorcall+0xb2)[0x4397d2]
/dcvenv/bin/python3.8[0x439088]
/dcvenv/bin/python3.8(_PyObject_CallMethodIdObjArgs+0xe5)[0x43b3d5]
/dcvenv/bin/python3.8(PyImport_ImportModuleLevelObject+0x2a5)[0x50f2c5]
/dcvenv/bin/python3.8(_PyEval_EvalFrameDefault+0x76af)[0x429e5f]
/dcvenv/bin/python3.8(PyEval_EvalCode+0x458)[0x4eb8f8]
/dcvenv/bin/python3.8[0x61cdbd]
/dcvenv/bin/python3.8[0x472260]
/dcvenv/bin/python3.8(PyVectorcall_Call+0x5c)[0x43b9bc]
/dcvenv/bin/python3.8(_PyEval_EvalFrameDefault+0x95df)[0x42bd8f]
/dcvenv/bin/python3.8(_PyEval_EvalCodeWithName+0x85d)[0x4ec1bd]
/dcvenv/bin/python3.8(_PyFunction_Vectorcall+0xb2)[0x4397d2]
/dcvenv/bin/python3.8(_PyEval_EvalFrameDefault+0x6f76)[0x429726]
/dcvenv/bin/python3.8[0x4215b8]
/dcvenv/bin/python3.8(_PyEval_EvalFrameDefault+0x1b83)[0x424333]
/dcvenv/bin/python3.8[0x4215b8]
/dcvenv/bin/python3.8(_PyEval_EvalFrameDefault+0x30ff)[0x4258af]
/dcvenv/bin/python3.8[0x4215b8]
======= Memory map: ========
00400000-00710000 r-xp 00000000 fd:03 4358346078                         /usr/local/bin/python3.8
0090f000-00910000 r--p 0030f000 fd:03 4358346078                         /usr/local/bin/python3.8
00910000-00948000 rw-p 00310000 fd:03 4358346078                         /usr/local/bin/python3.8
00948000-0096a000 rw-p 00000000 00:00 0 
015b8000-029c4000 rw-p 00000000 00:00 0                                  [heap]
7f7d90000000-7f7d90021000 rw-p 00000000 00:00 0 
7f7d90021000-7f7d94000000 ---p 00000000 00:00 0 
7f7d97f4f000-7f7d97fec000 r--p 00000000 fd:03 4358553907                 /dcvenv/lib/python3.8/site-packages/google/protobuf/pyext/_message.cpython-38-x86_64-linux-gnu.so
7f7d97fec000-7f7d9813b000 r-xp 0009d000 fd:03 4358553907                 /dcvenv/lib/python3.8/site-packages/google/protobuf/pyext/_message.cpython-38-x86_64-linux-gnu.so
7f7d9813b000-7f7d981af000 r--p 001ec000 fd:03 4358553907                 /dcvenv/lib/python3.8/site-packages/google/protobuf/pyext/_message.cpython-38-x86_64-linux-gnu.so
7f7d981af000-7f7d981b0000 ---p 00260000 fd:03 4358553907                 /dcvenv/lib/python3.8/site-packages/google/protobuf/pyext/_message.cpython-38-x86_64-linux-gnu.so
7f7d981b0000-7f7d981b5000 r--p 00260000 fd:03 4358553907                 /dcvenv/lib/python3.8/site-packages/google/protobuf/pyext/_message.cpython-38-x86_64-linux-gnu.so
7f7d981b5000-7f7d981bf000 rw-p 00265000 fd:03 4358553907                 /dcvenv/lib/python3.8/site-packages/google/protobuf/pyext/_message.cpython-38-x86_64-linux-gnu.so
7f7d981bf000-7f7d981c0000 rw-p 00000000 00:00 0 
7f7d981c0000-7f7d982a9000 r-xp 00000000 fd:03 55622958                   /usr/lib64/libstdc++.so.6.0.19
7f7d982a9000-7f7d984a9000 ---p 000e9000 fd:03 55622958                   /usr/lib64/libstdc++.so.6.0.19
7f7d984a9000-7f7d984b1000 r--p 000e9000 fd:03 55622958                   /usr/lib64/libstdc++.so.6.0.19
7f7d984b1000-7f7d984b3000 rw-p 000f1000 fd:03 55622958                   /usr/lib64/libstdc++.so.6.0.19
7f7d984b3000-7f7d98583000 rw-p 00000000 00:00 0 
7f7d98583000-7f7d99029000 r--p 00000000 fd:03 4359366356                 /dcvenv/lib/python3.8/site-packages/tensorflow/libtensorflow_framework.so.2
7f7d99029000-7f7d9a68d000 r-xp 00aa5000 fd:03 4359366356                 /dcvenv/lib/python3.8/site-packages/tensorflow/libtensorflow_framework.so.2
7f7d9a68d000-7f7d9a727000 r--p 02108000 fd:03 4359366356                 /dcvenv/lib/python3.8/site-packages/tensorflow/libtensorflow_framework.so.2
7f7d9a727000-7f7d9a731000 rw-p 021a1000 fd:03 4359366356                 /dcvenv/lib/python3.8/site-packages/tensorflow/libtensorflow_framework.so.2
7f7d9a731000-7f7d9a74e000 rw-p 00000000 00:00 0 
7f7d9a74e000-7f7d9a786000 r--p 00000000 fd:03 56568882                   /dcvenv/lib/python3.8/site-packages/_solib_k8/_U_S_Sthird_Uparty_Smkl_Cmkl_Ulibs_Ulinux___Uexternal_Sllvm_Uopenmp/libiomp5.so
7f7d9a786000-7f7d9a805000 r-xp 00037000 fd:03 56568882                   /dcvenv/lib/python3.8/site-packages/_solib_k8/_U_S_Sthird_Uparty_Smkl_Cmkl_Ulibs_Ulinux___Uexternal_Sllvm_Uopenmp/libiomp5.so
7f7d9a805000-7f7d9a808000 r--p 000b5000 fd:03 56568882                   /dcvenv/lib/python3.8/site-packages/_solib_k8/_U_S_Sthird_Uparty_Smkl_Cmkl_Ulibs_Ulinux___Uexternal_Sllvm_Uopenmp/libiomp5.so
7f7d9a808000-7f7d9a80c000 rw-p 000b7000 fd:03 56568882                   /dcvenv/lib/python3.8/site-packages/_solib_k8/_U_S_Sthird_Uparty_Smkl_Cmkl_Ulibs_Ulinux___Uexternal_Sllvm_Uopenmp/libiomp5.so
7f7d9a80c000-7f7d9a810000 rw-p 00000000 00:00 0 
7f7d9a810000-7f7da0e4d000 r--p 00000000 fd:03 56561157                   /dcvenv/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so
7f7da0e4d000-7f7db1ec7000 r-xp 0663c000 fd:03 56561157                   /dcvenv/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so
7f7db1ec7000-7f7db2460000 r--p 176b5000 fd:03 56561157                   /dcvenv/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so
7f7db2460000-7f7db2496000 rw-p 17c4d000 fd:03 56561157                   /dcvenv/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so
7f7db2496000-7f7db2529000 rw-p 00000000 00:00 0 
7f7db2529000-7f7db2619000 r--p 00000000 fd:03 2218606238                 /dcvenv/lib/python3.8/site-packages/tensorflow/core/platform/_cpu_feature_guard.so
7f7db2619000-7f7db2743000 r-xp 000ef000 fd:03 2218606238                 /dcvenv/lib/python3.8/site-packages/tensorflow/core/platform/_cpu_feature_guard.so
7f7db2743000-7f7db2750000 r--p 00218000 fd:03 2218606238                 /dcvenv/lib/python3.8/site-packages/tensorflow/core/platform/_cpu_feature_guard.so
7f7db2750000-7f7db2751000 rw-p 00224000 fd:03 2218606238                 /dcvenv/lib/python3.8/site-packages/tensorflow/core/platform/_cpu_feature_guard.sobash: line 1: 237730 Aborted                 deepconsensus -h

[akolesnikov]

Thank you @mjg0 ,

At this point we are going to try to reproduce the issue.
I will update the issue once we have some results.

[pichuan]

Hi @mjg0 ,
Sorry that this dropped off my queue at some point. Were you able to resolve this issue?
I'll leave this open. If you haven't resolved it, I'll give this another try to reproduce the issue as well.

[pgrosu]

@mjg0 When you installed 3.8.10 as an alternate version without overwriting the symbolic links, is there another version of python 3.8 on the system? The reason I ask is that you launch it via "RUN python3.8 -m venv /dcvenv" and that could point to the old one. I suspect one of the modules is triggering the non-allocated address.

[mjg0]

@pichuan I haven't been able to figure out a workaround.

@pgrosu CentOS 7.9 doesn't come with Python 3--the only one available in that container is the one I installed:

$ ch-run dctest -- bash -c "python3 --version; python3.8 --version"
bash: python3: command not found
Python 3.8.10

[pichuan]

@mjg0 I'm testing it on a CentOS7 machine. I'll report back soon.

Before that, I have a question:

I'm reading through your comment in Feb : #59 (comment) . I see that you're building through Docker. If you can do that, is there a reason why you have to use CentOS7?

Do you think you can try our Dockerfile instead: https://github.com/google/deepconsensus/blob/r1.2/Dockerfile ?

[pichuan]

@mjg0 On my CentOS7 machine, following the same steps you did (installing Python 3.8 on my own), I was able to reproduce the error you have.

I think @pgrosu 's point could be relevant here. Being on a CentOS7 machine (where the default is still Python2), installing Python3.8 on our own could have been an issue.

If you can use Docker, you can consider using our Docker version directory.

On my CentOS7 machine, I tried:

[pichuan@pichuan-centos ~]$ sudo service docker start
Redirecting to /bin/systemctl start docker.service
[pichuan@pichuan-centos ~]$ sudo docker pull google/deepconsensus:1.2.0

[pichuan@pichuan-centos ~]$ sudo docker run google/deepconsensus:1.2.0 deepconsensus run -h

worked for me.

@mjg0 Would this be a possible solution for you?

[mjg0]

@pichuan using the Docker image via Charliecloud or Singularity works. The main reason for installing in a virtualenv is to make life easier for users, many of whom aren't technically inclined--really it's just to remove a step in their pipelines.

Since there is a reasonable workaround in using the Docker container, I'm okay with closing this issue; conversely, if you'd like to keep debugging I'm willing to help if I can.

[pgrosu]

@mjg0 If you want, I can try to figure out how to make it work on your machine. Unfortunately I don't have resources on my end, but if you might be willing to provide a secure, isolated environment for me to experiment that matches your setup, I can give it a try in my spare time as it becomes available. If we can make it work, it would be nice if you can share our solution here so others can benefit from it. Would that be something that would work for you?

[mjg0]

@pgrosu I've created a Docker container, mjg0/deepconsensus-centos-7-issue-59, that replicates the issue. It's slightly different than the one above:

FROM centos:7

# Install Python 3.8
RUN yum install -y gcc openssl-devel libffi-devel bzip2-devel make
RUN curl -O https://www.python.org/ftp/python/3.8.15/Python-3.8.15.tgz
RUN tar xf Python-3.8.15.tgz
WORKDIR Python-3.8.15
RUN ./configure
RUN make -j altinstall

# Create venv
RUN python3.8 -m venv /dcvenv
RUN source /dcvenv/bin/activate && \
    pip install deepconsensus[cpu]==1.2.0

# Trigger bug
CMD source /dcvenv/bin/activate && \
    deepconsensus --help

Unfortunately our system is closed, so the container is the best I can do. If that isn't sufficient for debugging don't worry about it, the workaround isn't painful.

[pgrosu]

@mjg0 Thank you for providing the Dockerfile, which should be enough -- totally understandable. I was able to replicate the memory deallocation issue, and will analyze what might be happening.

Thanks,
~p

[pgrosu]

Hi @mjg0,

So get it to work under CentOS 7, all you have to run in your virtual environment are the these two additional commands:

pip uninstall intel-tensorflow==2.9.1
pip install "tensorflow==2.9.1" --force-reinstall

To run it just type the following:

python3.8 -m deepconsensus.cli

Below is an example of a test run:

(dcvenv) [root@cdb6010787d0 Python-3.8.15]# python3.8 -m deepconsensus.cli --help
2023-06-09 04:34:15.205104: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2023-06-09 04:34:15.206623: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
usage: DeepConsensus

Usage:
  deepconsensus <command> [optional arguments]

Commands:
  preprocess: Convert aligned subreads to tf.Example format.
  run: Run DeepConsenseus beginning with aligned subreads.
  calibrate: Calculate base-quality calibration.
  filter_reads: Filters reads from fastq or bam at a given q-value.
cli.py: error: the following arguments are required: command

(dcvenv) [root@cdb6010787d0 Python-3.8.15]# python3.8 -m deepconsensus.utils.utils_test
2023-06-09 04:34:29.369728: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2023-06-09 04:34:29.371172: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Running tests under Python 3.8.15: /dcvenv/bin/python3.8
2023-06-09 04:34:32.462167: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2023-06-09 04:34:32.463344: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
2023-06-09 04:34:32.464317: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (cdb6010787d0): /proc/driver/nvidia/version does not exist
[ RUN      ] EncodedSequenceToString.test_score_to_string0 ([0], ' ')
[       OK ] EncodedSequenceToString.test_score_to_string0 ([0], ' ')
[ RUN      ] EncodedSequenceToString.test_score_to_string1 ([1], 'A')
[       OK ] EncodedSequenceToString.test_score_to_string1 ([1], 'A')
[ RUN      ] EncodedSequenceToString.test_score_to_string2 ([2], 'T')
[       OK ] EncodedSequenceToString.test_score_to_string2 ([2], 'T')
[ RUN      ] EncodedSequenceToString.test_score_to_string3 ([3], 'C')
[       OK ] EncodedSequenceToString.test_score_to_string3 ([3], 'C')
[ RUN      ] EncodedSequenceToString.test_score_to_string4 ([4], 'G')
[       OK ] EncodedSequenceToString.test_score_to_string4 ([4], 'G')
[ RUN      ] EncodedSequenceToString.test_score_to_string5 ([2, 0, 1], 'T A')
[       OK ] EncodedSequenceToString.test_score_to_string5 ([2, 0, 1], 'T A')
[ RUN      ] QualityScoreToStringTest.test_score_list_to_string0 (array([], dtype=float64), '')
[       OK ] QualityScoreToStringTest.test_score_list_to_string0 (array([], dtype=float64), '')
[ RUN      ] QualityScoreToStringTest.test_score_list_to_string1 (array([ 0, 10, 20, 30, 40]), '!+5?I')
[       OK ] QualityScoreToStringTest.test_score_list_to_string1 (array([ 0, 10, 20, 30, 40]), '!+5?I')
[ RUN      ] QualityScoreToStringTest.test_score_to_string0 (0, '!')
[       OK ] QualityScoreToStringTest.test_score_to_string0 (0, '!')
[ RUN      ] QualityScoreToStringTest.test_score_to_string1 (40, 'I')
[       OK ] QualityScoreToStringTest.test_score_to_string1 (40, 'I')
[ RUN      ] QualityScoreToStringTest.test_score_to_string2 (20, '5')
[       OK ] QualityScoreToStringTest.test_score_to_string2 (20, '5')
[ RUN      ] QualityStringToArrayTest.test_string_to_int0 ('', [])
[       OK ] QualityStringToArrayTest.test_string_to_int0 ('', [])
[ RUN      ] QualityStringToArrayTest.test_string_to_int1 ('!', [0])
[       OK ] QualityStringToArrayTest.test_string_to_int1 ('!', [0])
[ RUN      ] QualityStringToArrayTest.test_string_to_int2 ('I', [40])
[       OK ] QualityStringToArrayTest.test_string_to_int2 ('I', [40])
[ RUN      ] QualityStringToArrayTest.test_string_to_int3 ('5', [20])
[       OK ] QualityStringToArrayTest.test_string_to_int3 ('5', [20])
[ RUN      ] QualityStringToArrayTest.test_string_to_int4 ('!+5?I', [0, 10, 20, 30, 40])
[       OK ] QualityStringToArrayTest.test_string_to_int4 ('!+5?I', [0, 10, 20, 30, 40])
[ RUN      ] TestAvgPhred.test_avg_ccs_quality_integer list
[       OK ] TestAvgPhred.test_avg_ccs_quality_integer list
[ RUN      ] TestAvgPhred.test_avg_ccs_quality_multiple_values
[       OK ] TestAvgPhred.test_avg_ccs_quality_multiple_values
[ RUN      ] TestAvgPhred.test_avg_ccs_quality_no values
[       OK ] TestAvgPhred.test_avg_ccs_quality_no values
[ RUN      ] TestAvgPhred.test_avg_ccs_quality_single_value
[       OK ] TestAvgPhred.test_avg_ccs_quality_single_value
[ RUN      ] TestAvgPhred.test_avg_ccs_quality_spacer values
[       OK ] TestAvgPhred.test_avg_ccs_quality_spacer values
[ RUN      ] TestBatchLeftSeq.test_avg_ccs_quality_no_gap
[       OK ] TestBatchLeftSeq.test_avg_ccs_quality_no_gap
[ RUN      ] TestBatchLeftSeq.test_avg_ccs_quality_staggered_gaps
[       OK ] TestBatchLeftSeq.test_avg_ccs_quality_staggered_gaps
[ RUN      ] TestLeftSeq.test_avg_ccs_quality_multiple_gaps
[       OK ] TestLeftSeq.test_avg_ccs_quality_multiple_gaps
[ RUN      ] TestLeftSeq.test_avg_ccs_quality_no_gap
[       OK ] TestLeftSeq.test_avg_ccs_quality_no_gap
[ RUN      ] TestLeftSeq.test_avg_ccs_quality_single_gap
[       OK ] TestLeftSeq.test_avg_ccs_quality_single_gap
----------------------------------------------------------------------
Ran 26 tests in 0.147s

OK
(dcvenv) [root@cdb6010787d0 Python-3.8.15]#

I have not tested it heavily, which might nice to do to ensure the installation is working properly.

Thanks,
~p