Error message
root@a19b333cfe3b:/home/keras/notebook# pip install warp_rnnt
Collecting warp_rnnt
Using cached warp_rnnt-0.5.0.tar.gz (10 kB)
Requirement already satisfied: pybind11 in /opt/conda/lib/python3.7/site-packages (from warp_rnnt) (2.9.0)
Requirement already satisfied: numpy in /opt/conda/lib/python3.7/site-packages (from warp_rnnt) (1.19.5)
Requirement already satisfied: torch>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from warp_rnnt) (1.9.0)
Requirement already satisfied: typing_extensions in /opt/conda/lib/python3.7/site-packages (from torch>=1.0.0->warp_rnnt) (3.7.4.3)
Building wheels for collected packages: warp-rnnt
Building wheel for warp-rnnt (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /opt/conda/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/setup.py'"'"'; __file__='"'"'/tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c8
50ea4a028137251/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compil
e(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-73ng5boj
cwd: /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/
Complete output (106 lines):
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/warp_rnnt
copying warp_rnnt/__init__.py -> build/lib.linux-x86_64-3.7/warp_rnnt
copying warp_rnnt/test.py -> build/lib.linux-x86_64-3.7/warp_rnnt
running build_ext
building 'warp_rnnt._C' extension
creating /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7
Emitting ninja build file /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] c++ -MMD -MF /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/binding.o.d -pthread -B /opt/conda/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fP
IC -I/opt/conda/lib/python3.7/site-packages/torch/include -I/opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.7/site-packages/torch/include/TH -I/opt/conda/lib/python3.7/site-packages/torch/include/THC
-I/usr/local/cuda/include -I/opt/conda/include/python3.7m -c -c /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/binding.cpp -o /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/binding.
o -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
FAILED: /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/binding.o
c++ -MMD -MF /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/binding.o.d -pthread -B /opt/conda/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/
opt/conda/lib/python3.7/site-packages/torch/include -I/opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.7/site-packages/torch/include/TH -I/opt/conda/lib/python3.7/site-packages/torch/include/THC -I/usr
/local/cuda/include -I/opt/conda/include/python3.7m -c -c /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/binding.cpp -o /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/binding.o -DTO
RCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from /usr/include/c++/7/ext/string_conversions.h:41:0,
from /usr/include/c++/7/bits/basic_string.h:6361,
from /usr/include/c++/7/string:52,
from /usr/include/c++/7/stdexcept:39,
from /usr/include/c++/7/array:39,
from /usr/include/c++/7/tuple:39,
from /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/binding.cpp:1:
/usr/include/c++/7/cstdlib:75:15: fatal error: stdlib.h: No such file or directory
#include_next
^~~~~~~~~~ compilation terminated.
compilation terminated. [2/3] /usr/local/cuda/bin/nvcc -I/opt/conda/lib/python3.7/site-packages/torch/include -I/opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.7/site-packages/torch/include/TH -I/opt/conda/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/opt/conda/include/python3.7m -c -c /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/core.cu -o /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/tem
p.linux-x86_64-3.7/core.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIN
D11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,co
de=sm_86 -std=c++14
FAILED: /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/core.o
/usr/local/cuda/bin/nvcc -I/opt/conda/lib/python3.7/site-packages/torch/include -I/opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.7/site-packages/torch/include/TH -I/opt/conda/lib/python3.7/site-pa
ckages/torch/include/THC -I/usr/local/cuda/include -I/opt/conda/include/python3.7m -c -c /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/core.cu -o /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linu
x-x86_64-3.7/core.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_CO
MPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_
86 -std=c++14
In file included from /usr/local/cuda/include/crt/math_functions.h:8958:0,
from /usr/local/cuda/include/crt/common_functions.h:295,
from /usr/local/cuda/include/cuda_runtime.h:115,
from :0:
/usr/include/c++/7/cmath:45:15: fatal error: math.h: No such file or directory
#include_next
^~~~~~~~
compilation terminated.
[3/3] /usr/local/cuda/bin/nvcc -I/opt/conda/lib/python3.7/site-packages/torch/include -I/opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.7/site-packages/torch/include/TH -I/opt/conda/lib/python3.7/s
ite-packages/torch/include/THC -I/usr/local/cuda/include -I/opt/conda/include/python3.7m -c -c /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/core_gather.cu -o /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/bu
ild/temp.linux-x86_64-3.7/core_gather.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENS
ION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_86,code=compute_86 -gencode=arch
=compute_86,code=sm_86 -std=c++14
FAILED: /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/core_gather.o
/usr/local/cuda/bin/nvcc -I/opt/conda/lib/python3.7/site-packages/torch/include -I/opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.7/site-packages/torch/include/TH -I/opt/conda/lib/python3.7/site-pa
ckages/torch/include/THC -I/usr/local/cuda/include -I/opt/conda/include/python3.7m -c -c /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/core_gather.cu -o /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/te
mp.linux-x86_64-3.7/core_gather.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H
'-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compu
te_86,code=sm_86 -std=c++14
In file included from /usr/local/cuda/include/crt/math_functions.h:8958:0,
from /usr/local/cuda/include/crt/common_functions.h:295,
from /usr/local/cuda/include/cuda_runtime.h:115,
from :0:
/usr/include/c++/7/cmath:45:15: fatal error: math.h: No such file or directory
#include_next
^~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1672, in _run_ninja_build
env=env)
File "/opt/conda/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/setup.py", line 64, in
"Topic :: Software Development :: Libraries :: Python Modules",
File "/opt/conda/lib/python3.7/site-packages/setuptools/init.py", line 153, in setup
return distutils.core.setup(**attrs)
File "/opt/conda/lib/python3.7/distutils/core.py", line 148, in setup
dist.run_commands()
File "/opt/conda/lib/python3.7/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/opt/conda/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/opt/conda/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 290, in run
self.run_command('build')
File "/opt/conda/lib/python3.7/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/opt/conda/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/opt/conda/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 79, in run
_build_ext.run(self)
File "/opt/conda/lib/python3.7/distutils/command/build_ext.py", line 340, in run
self.build_extensions()
File "/opt/conda/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 709, in build_extensions
build_ext.build_extensions(self)
File "/opt/conda/lib/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
self._build_extensions_serial()
File "/opt/conda/lib/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
self.build_extension(ext)
File "/opt/conda/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 196, in build_extension
_build_ext.build_extension(self, ext)
File "/opt/conda/lib/python3.7/distutils/command/build_ext.py", line 534, in build_extension
depends=ext.depends)
File "/opt/conda/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 539, in unix_wrap_ninja_compile
with_cuda=with_cuda)
File "/opt/conda/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1360, in _write_ninja_file_and_compile_objects
error_prefix='Error compiling objects for extension')
File "/opt/conda/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1682, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error compiling objects for extension
ERROR: Failed building wheel for warp-rnnt
Running setup.py clean for warp-rnnt
Failed to build warp-rnnt
Installing collected packages: warp-rnnt
Running setup.py install for warp-rnnt ... error
ERROR: Command errored out with exit status 1:
command: /opt/conda/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/setup.py'"'"'; file='"'"'/tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427
c850ea4a028137251/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(comp
ile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-vvd1smyc/install-record.txt --single-version-externally-managed --compile --install-headers /opt/conda/include/python3.7m/warp-rnnt
cwd: /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/
Complete output (108 lines):
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/warp_rnnt
copying warp_rnnt/init.py -> build/lib.linux-x86_64-3.7/warp_rnnt
copying warp_rnnt/test.py -> build/lib.linux-x86_64-3.7/warp_rnnt
running build_ext
building 'warp_rnnt.C' extension
creating /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7
Emitting ninja build file /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] c++ -MMD -MF /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/binding.o.d -pthread -B /opt/conda/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -
fPIC -I/opt/conda/lib/python3.7/site-packages/torch/include -I/opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.7/site-packages/torch/include/TH -I/opt/conda/lib/python3.7/site-packages/torch/include/TH
C -I/usr/local/cuda/include -I/opt/conda/include/python3.7m -c -c /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/binding.cpp -o /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/bindin
g.o -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
FAILED: /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/binding.o
c++ -MMD -MF /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/binding.o.d -pthread -B /opt/conda/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -
I/opt/conda/lib/python3.7/site-packages/torch/include -I/opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.7/site-packages/torch/include/TH -I/opt/conda/lib/python3.7/site-packages/torch/include/THC -I/u
sr/local/cuda/include -I/opt/conda/include/python3.7m -c -c /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/binding.cpp -o /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/binding.o -D
TORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from /usr/include/c++/7/ext/string_conversions.h:41:0,
from /usr/include/c++/7/bits/basic_string.h:6361,
from /usr/include/c++/7/string:52,
from /usr/include/c++/7/stdexcept:39,
from /usr/include/c++/7/array:39,
from /usr/include/c++/7/tuple:39,
from /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/binding.cpp:1:
/usr/include/c++/7/cstdlib:75:15: fatal error: stdlib.h: No such file or directory
#include_next <stdlib.h>
^~~~~~~~~~
compilation terminated.
[2/3] /usr/local/cuda/bin/nvcc -I/opt/conda/lib/python3.7/site-packages/torch/include -I/opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.7/site-packages/torch/include/TH -I/opt/conda/lib/python3.7
/site-packages/torch/include/THC -I/usr/local/cuda/include -I/opt/conda/include/python3.7m -c -c /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/core_gather.cu -o /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/
build/temp.linux-x86_64-3.7/core_gather.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTE
NSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_86,code=compute_86 -gencode=ar
ch=compute_86,code=sm_86 -std=c++14
FAILED: /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/core_gather.o
/usr/local/cuda/bin/nvcc -I/opt/conda/lib/python3.7/site-packages/torch/include -I/opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.7/site-packages/torch/include/TH -I/opt/conda/lib/python3.7/site-
packages/torch/include/THC -I/usr/local/cuda/include -I/opt/conda/include/python3.7m -c -c /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/core_gather.cu -o /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/
temp.linux-x86_64-3.7/core_gather.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS_ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_
H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_86,code=compute_86 -gencode=arch=com
pute_86,code=sm_86 -std=c++14
In file included from /usr/local/cuda/include/crt/math_functions.h:8958:0,
from /usr/local/cuda/include/crt/common_functions.h:295,
from /usr/local/cuda/include/cuda_runtime.h:115,
from :0:
/usr/include/c++/7/cmath:45:15: fatal error: math.h: No such file or directory
#include_next <math.h>
^~~~~~~~
compilation terminated.
[3/3] /usr/local/cuda/bin/nvcc -I/opt/conda/lib/python3.7/site-packages/torch/include -I/opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.7/site-packages/torch/include/TH -I/opt/conda/lib/python3.7
/site-packages/torch/include/THC -I/usr/local/cuda/include -I/opt/conda/include/python3.7m -c -c /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/core.cu -o /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/t
emp.linux-x86_64-3.7/core.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H '-DPYB
IND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,
code=sm_86 -std=c++14
FAILED: /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.linux-x86_64-3.7/core.o
/usr/local/cuda/bin/nvcc -I/opt/conda/lib/python3.7/site-packages/torch/include -I/opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.7/site-packages/torch/include/TH -I/opt/conda/lib/python3.7/site-
packages/torch/include/THC -I/usr/local/cuda/include -I/opt/conda/include/python3.7m -c -c /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/core.cu -o /tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/build/temp.li
nux-x86_64-3.7/core.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_
COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=s
m_86 -std=c++14
In file included from /usr/local/cuda/include/crt/math_functions.h:8958:0,
from /usr/local/cuda/include/crt/common_functions.h:295,
from /usr/local/cuda/include/cuda_runtime.h:115,
from :0:
/usr/include/c++/7/cmath:45:15: fatal error: math.h: No such file or directory
#include_next <math.h>
^~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1672, in _run_ninja_build
env=env)
File "/opt/conda/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/setup.py", line 64, in <module>
"Topic :: Software Development :: Libraries :: Python Modules",
File "/opt/conda/lib/python3.7/site-packages/setuptools/__init__.py", line 153, in setup
return distutils.core.setup(**attrs)
File "/opt/conda/lib/python3.7/distutils/core.py", line 148, in setup
dist.run_commands()
File "/opt/conda/lib/python3.7/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/opt/conda/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/opt/conda/lib/python3.7/site-packages/setuptools/command/install.py", line 61, in run
return orig.install.run(self)
File "/opt/conda/lib/python3.7/distutils/command/install.py", line 545, in run
self.run_command('build')
File "/opt/conda/lib/python3.7/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/opt/conda/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/opt/conda/lib/python3.7/distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/opt/conda/lib/python3.7/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/opt/conda/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/opt/conda/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 79, in run
_build_ext.run(self)
File "/opt/conda/lib/python3.7/distutils/command/build_ext.py", line 340, in run
self.build_extensions()
File "/opt/conda/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 709, in build_extensions
build_ext.build_extensions(self)
File "/opt/conda/lib/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
self._build_extensions_serial()
File "/opt/conda/lib/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
self.build_extension(ext)
File "/opt/conda/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 196, in build_extension
_build_ext.build_extension(self, ext)
File "/opt/conda/lib/python3.7/distutils/command/build_ext.py", line 534, in build_extension
depends=ext.depends)
File "/opt/conda/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 539, in unix_wrap_ninja_compile
with_cuda=with_cuda)
File "/opt/conda/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1360, in _write_ninja_file_and_compile_objects
error_prefix='Error compiling objects for extension')
File "/opt/conda/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1682, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error compiling objects for extension
----------------------------------------
ERROR: Command errored out with exit status 1: /opt/conda/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-b1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/setup.py'"'"'; file='"'"'/tmp/pip-install-b
1mhbvnc/warp-rnnt_074f8d6e4c20427c850ea4a028137251/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"',
'"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-vvd1smyc/install-record.txt --single-version-externally-managed --compile --install-headers /opt/conda/include/python3.7m/warp-rnnt Check the logs fo
r full command output.
I used to have similar errors with warp-ctc
, and they were resolved with hacks like this:
RUN apt-get install gcc-5 g++-5 g++-5-multilib gfortran-5 -y && \
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 60 --slave /usr/bin/g++ g++ /usr/bin/g++-5 --slave /usr/bin/gfortran gfortran /usr/bin/gfortran-5 && \
update-alternatives --query gcc
RUN gcc --version
Is there anything I can do about this without downgrading or changing the current environment?
I want to have a stable loss which is rubust to labels_lengths when training.
What value should I pass to this two parmas?
What's more, what is the approximate relationship between loss and actual wer?
For example, if I want a wer aroud 0.5. How much should be the value of the loss?
log_probs (torch.FloatTensor): Input tensor with shape (N, T, U, V)
where N is the minibatch size, T is the maximum number of
input frames, U is the maximum number of output labels and V is
the vocabulary of labels (including the blank).
labels (torch.IntTensor): Tensor with shape (N, U-1) representing the
reference labels for all samples in the minibatch.
Hi, I am confused about the labels, why the shape should be U-1,
<eos>
should not be included in the labels ?
@1ytic
Hi @1ytic ,
Thank you for this work on optimizing warp rnn-t operation which is becoming increasingly useful for many speech recognition acoustic models. We have studied your implementation and have the following observations:
- When we run the benchmark script in your repo, the run time we got is as below, where new refers to this new repo, and the baseline is what we have now in RNN-T reference model. We used B=32, T=U=200, V= 29, which is a typical case in our dataset. From the output of the benchmark script, it does appear that the new loss function runs faster:
new: 1.76 ms
baseline: 6.10 ms
- However, in the benchmark script that the author provided, the run time was measured as:
t = timer()
costs = loss(xs, ys, xn, yn)
elapsed_time += timer() - t
This way of measuring has a problem that CPU could run ahead of GPU and stop the timer even before the kernel is completed. After adding synchronization as below,
torch.cuda.synchronize() # sync before start the timer
t = timer()
costs = loss(xs, ys, xn, yn)
torch.cuda.synchronize() # sync before stop the timer
elapsed_time += timer() - t
the run time we get is:
new: 15.82 ms
baseline: 6.12 ms
- This is similar to the run time we got from GPU profiler nsys:
new: 14.38 ms
baseline: 4.75 ms
In summary - It does not look like that the alternative loss function is running faster than what we have. The claimed speedup in the repo is likely caused by a flawed benchmark methodology.
Can you share your thought process on this?
Thanks,
Ashish
Thanks for your codes. The warp-rnnt is not supported in pytorch 1.7.
Could you upgrade the packages?
In current implementation, the warps along T axis are computed in fully serialized manner
|
if (t < actual_t && u < actual_u) { |
|
|
|
// Ready to compute alphas[t, u] |
|
|
|
unsigned int l = labels[idx2(n, u-1, U-1)]; |
|
|
|
float bias = log_probs[idx4(n, t-1, u, blank, T, U, V)]; |
|
float skip = alphas[idx3(n, p, u, T, U)] + bias; |
|
float emit = alphas[idx3(n, t, u-1, T, U)] + log_probs[idx4(n, t, u-1, l, T, U, V)]; |
|
|
|
float r = log_sum_exp(skip, emit); |
|
float output = r; |
|
|
|
for(unsigned int i = 1; i < W; i++) { |
|
r = __shfl_up_sync(0xffffffff, r, 1); |
|
if (i == d) { |
|
r = log_sum_exp(r + bias, emit); |
|
output = r; |
|
} |
|
} |
|
|
|
alphas[idx3(n, t, u, T, U)] = output; |
|
} |
The for loop of each warp is executed one-by-one, which means the ith warp at specific row u
, has to wait for all its leading warps to finish the loops, and that is i (num of warps) * W (for loop overhead, warpsize, 32 here)
time complexity.
However, we don't necessarily have to wait for previous warps to finish before we go into the loop in current warp.
Let's take forward computation of alphas as the example with warpsize=4
:
Here d
denotes the index inside a warp, so 0 <= d < W
. B
is the result from u-1
row and supposed to be ready.
The forward computation of alpha follows (indeed we do the computation in logarithm, here is just for discussion):
Note that alpha_0
relies on result from the last warp.
Here comes the trick, I rewrote alpha_3
formula to following
The underlined part is warp-independent. The first part (the product of emitting probability e_2 e_1 e_0
) can be computed by prefix sum (scan) algorithm in logarithm, and only introduce log2(W) complexity.
Finally, the new procedure is like:
- Compute local paths combination prob (the underlined part). O(W) complexity;
- Compute product of emitting probs (e2e1e0, ...) with prefix sum algorithm. O(log2(W)) complexity;
- Wait for previous warps to finish and compute final results. Constant complexity.
For all warps at row u, 1 & 2 can be done in parallel, ith warp has only to wait all previous warps to finish step 3. The new procedure should be considerably faster than current serialized execution, especially when T is large.
/***/warp-rnnt/pytorch_binding/warp_rnnt/_C.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZNSt19basic_ostringstreamIcSt11char_traitsIcESaIcEEC1Ev
Hi, I'm trying to use warp-rnnt
, but I get the error mentioned below.
My environment:
Python: Python 3.7.5
Torch: 1.8.0+cu111
CUDA Version: 11.2
Output of python -m warp_rnnt.test
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 183, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/usr/lib/python3.7/runpy.py", line 109, in _get_module_details
__import__(pkg_name)
File "/home/martynas/.../lib/python3.7/site-packages/warp_rnnt/__init__.py", line 2, in <module>
import warp_rnnt._C as core
ImportError: /home/martynas/.../lib/python3.7/site-packages/warp_rnnt/_C.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe28TypeMeta21_typeMetaDataInstanceIN3c107complexIfEEEEPKNS_6detail12TypeMetaDataEv
Is there anything I can do?
I search the error in the title,it happens when there are several losses.But in my code,there is only the RNN-t-loss,but it gives the error.
The full sentence is ''RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.".
I tried the "retain_graph=True" parameter in loss.backward(),it failed.
So do you have some ideas about the error?
Recommend Projects
-
-
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
An Open Source Machine Learning Framework for Everyone
-
The Web framework for perfectionists with deadlines.
-
A PHP framework for web artisans
-
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
Some thing interesting about web. New door for the world.
-
A server is a program made to process requests and deliver data to clients.
-
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Some thing interesting about visualization, use data art
-
Some thing interesting about game, make everyone happy.
-
Recommend Org
-
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Open source projects and samples from Microsoft.
-
Google ❤️ Open Source for everyone.
-
Alibaba Open Source for everyone
-
Data-Driven Documents codes.
-
China tencent open source team.
-
Jobs
Jooble