Loss is 0.000 When using --cuda parameter. about deepspeech.pytorch

[lintangsutawika]

It turns out to be wrong configurations. I followed the solution from issue #13, sudo cmake .. caused issue when I ran sudo make, then I tried sudo cmake ... -DCUDA_NVCC_FLAGS="-D_FORCE_INLINES" .. before sudo make again which worked but caused this problem.
I rebuilt warp-ctc with

cmake ..
make

now it's working fine.

[mohamad-hasan-sohan-ajini]

I build the warp_ctc with:

mkdir build; cd build
cmake ..
make

and install pytorch binding with (issue #112):

cd ../pytorch_binding
CUDA_HOME=/usr/local/cuda sudo -E python3 setup.py install

but face the same issue! Everything works fine with cpu, but set --cuda flag always gives you zero loss.
@SeanNaren Any Idea about what did I do wrong?

@lintangsutawika can you provide more detail about how you did solve the issue? building without sudo privilege seems is not the proper solution.

[SeanNaren]

Could we see the output once you build the pytorch warp_ctc bindings?

[mohamad-hasan-sohan-ajini]

cmake .. output:

aj@aj-pc:~/repo/warp-ctc/build$ cmake ..
-- The C compiler identification is GNU 4.8.5
-- The CXX compiler identification is GNU 4.8.5
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /usr/local/cuda (found suitable version "9.1", minimum required is "6.5")
-- cuda found TRUE
-- Found Torch7 in /home/aj/torch/install
-- Torch found /home/aj/torch/install/share/cmake/torch
-- Building shared library with GPU support
-- Building Torch Bindings with GPU support
-- Configuring done
-- Generating done
-- Build files have been written to: /home/aj/repo/warp-ctc/build

and make is as follows:

[ 10%] Building NVCC (Device) object CMakeFiles/warpctc.dir/src/warpctc_generated_reduce.cu.o
/home/aj/repo/warp-ctc/src/reduce.cu(44): warning: function "__shfl_down(float, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(278): here was declared deprecated ("__shfl_down() is not valid on compute_70 and above, and should be replaced with __shfl_down_sync().To continue using __shfl_down(), specify virtual architecture compute_60 when targeting sm_70 and above, for example, using the pair of compiler options: -arch=compute_60 -code=sm_70.")
detected during:
instantiation of "T CTAReduce<NT, T, Rop>::reduce(int, T, CTAReduce<NT, T, Rop>::Storage &, int, Rop) [with NT=128, T=float, Rop=ctc_helper::add<float, float>]"
(76): here
instantiation of "void reduce_rows<NT,Iop,Rop,T>(Iop, Rop, const T *, T *, int, int) [with NT=128, Iop=ctc_helper::negate<float, float>, Rop=ctc_helper::add<float, float>, T=float]"
(124): here
instantiation of "void ReduceHelper::impl(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::negate<float, float>, Rof=ctc_helper::add<float, float>]"
(139): here
instantiation of "ctcStatus_t reduce(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::negate<float, float>, Rof=ctc_helper::add<float, float>]"
(149): here
/home/aj/repo/warp-ctc/src/reduce.cu(44): warning: function "__shfl_down(float, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(278): here was declared deprecated ("__shfl_down() is not valid on compute_70 and above, and should be replaced with __shfl_down_sync().To continue using __shfl_down(), specify virtual architecture compute_60 when targeting sm_70 and above, for example, using the pair of compiler options: -arch=compute_60 -code=sm_70.")
detected during:
instantiation of "T CTAReduce<NT, T, Rop>::reduce(int, T, CTAReduce<NT, T, Rop>::Storage &, int, Rop) [with NT=128, T=float, Rop=ctc_helper::maximum<float, float>]"
(76): here
instantiation of "void reduce_rows<NT,Iop,Rop,T>(Iop, Rop, const T *, T *, int, int) [with NT=128, Iop=ctc_helper::identity<float, float>, Rop=ctc_helper::maximum<float, float>, T=float]"
(124): here
instantiation of "void ReduceHelper::impl(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::identity<float, float>, Rof=ctc_helper::maximum<float, float>]"
(139): here
instantiation of "ctcStatus_t reduce(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::identity<float, float>, Rof=ctc_helper::maximum<float, float>]"
(157): here
/home/aj/repo/warp-ctc/src/reduce.cu(44): warning: function "__shfl_down(float, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(278): here was declared deprecated ("__shfl_down() is deprecated in favor of __shfl_down_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
detected during:
instantiation of "T CTAReduce<NT, T, Rop>::reduce(int, T, CTAReduce<NT, T, Rop>::Storage &, int, Rop) [with NT=128, T=float, Rop=ctc_helper::add<float, float>]"
(76): here
instantiation of "void reduce_rows<NT,Iop,Rop,T>(Iop, Rop, const T *, T *, int, int) [with NT=128, Iop=ctc_helper::negate<float, float>, Rop=ctc_helper::add<float, float>, T=float]"
(124): here
instantiation of "void ReduceHelper::impl(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::negate<float, float>, Rof=ctc_helper::add<float, float>]"
(139): here
instantiation of "ctcStatus_t reduce(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::negate<float, float>, Rof=ctc_helper::add<float, float>]"
(149): here
/home/aj/repo/warp-ctc/src/reduce.cu(44): warning: function "__shfl_down(float, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(278): here was declared deprecated ("__shfl_down() is deprecated in favor of __shfl_down_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
detected during:
instantiation of "T CTAReduce<NT, T, Rop>::reduce(int, T, CTAReduce<NT, T, Rop>::Storage &, int, Rop) [with NT=128, T=float, Rop=ctc_helper::maximum<float, float>]"
(76): here
instantiation of "void reduce_rows<NT,Iop,Rop,T>(Iop, Rop, const T *, T *, int, int) [with NT=128, Iop=ctc_helper::identity<float, float>, Rop=ctc_helper::maximum<float, float>, T=float]"
(124): here
instantiation of "void ReduceHelper::impl(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::identity<float, float>, Rof=ctc_helper::maximum<float, float>]"
(139): here
instantiation of "ctcStatus_t reduce(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::identity<float, float>, Rof=ctc_helper::maximum<float, float>]"
(157): here
/home/aj/repo/warp-ctc/src/reduce.cu(44): warning: function "__shfl_down(float, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(278): here was declared deprecated ("__shfl_down() is deprecated in favor of __shfl_down_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
detected during:
instantiation of "T CTAReduce<NT, T, Rop>::reduce(int, T, CTAReduce<NT, T, Rop>::Storage &, int, Rop) [with NT=128, T=float, Rop=ctc_helper::add<float, float>]"
(76): here
instantiation of "void reduce_rows<NT,Iop,Rop,T>(Iop, Rop, const T *, T *, int, int) [with NT=128, Iop=ctc_helper::negate<float, float>, Rop=ctc_helper::add<float, float>, T=float]"
(124): here
instantiation of "void ReduceHelper::impl(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::negate<float, float>, Rof=ctc_helper::add<float, float>]"
(139): here
instantiation of "ctcStatus_t reduce(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::negate<float, float>, Rof=ctc_helper::add<float, float>]"
(149): here
/home/aj/repo/warp-ctc/src/reduce.cu(44): warning: function "__shfl_down(float, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(278): here was declared deprecated ("__shfl_down() is deprecated in favor of __shfl_down_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
detected during:
instantiation of "T CTAReduce<NT, T, Rop>::reduce(int, T, CTAReduce<NT, T, Rop>::Storage &, int, Rop) [with NT=128, T=float, Rop=ctc_helper::maximum<float, float>]"
(76): here
instantiation of "void reduce_rows<NT,Iop,Rop,T>(Iop, Rop, const T *, T *, int, int) [with NT=128, Iop=ctc_helper::identity<float, float>, Rop=ctc_helper::maximum<float, float>, T=float]"
(124): here
instantiation of "void ReduceHelper::impl(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::identity<float, float>, Rof=ctc_helper::maximum<float, float>]"
(139): here
instantiation of "ctcStatus_t reduce(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::identity<float, float>, Rof=ctc_helper::maximum<float, float>]"
(157): here
/home/aj/repo/warp-ctc/src/reduce.cu(44): warning: function "__shfl_down(float, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(278): here was declared deprecated ("__shfl_down() is deprecated in favor of __shfl_down_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
detected during:
instantiation of "T CTAReduce<NT, T, Rop>::reduce(int, T, CTAReduce<NT, T, Rop>::Storage &, int, Rop) [with NT=128, T=float, Rop=ctc_helper::add<float, float>]"
(76): here
instantiation of "void reduce_rows<NT,Iop,Rop,T>(Iop, Rop, const T *, T *, int, int) [with NT=128, Iop=ctc_helper::negate<float, float>, Rop=ctc_helper::add<float, float>, T=float]"
(124): here
instantiation of "void ReduceHelper::impl(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::negate<float, float>, Rof=ctc_helper::add<float, float>]"
(139): here
instantiation of "ctcStatus_t reduce(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::negate<float, float>, Rof=ctc_helper::add<float, float>]"
(149): here
/home/aj/repo/warp-ctc/src/reduce.cu(44): warning: function "__shfl_down(float, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(278): here was declared deprecated ("__shfl_down() is deprecated in favor of __shfl_down_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
detected during:
instantiation of "T CTAReduce<NT, T, Rop>::reduce(int, T, CTAReduce<NT, T, Rop>::Storage &, int, Rop) [with NT=128, T=float, Rop=ctc_helper::maximum<float, float>]"
(76): here
instantiation of "void reduce_rows<NT,Iop,Rop,T>(Iop, Rop, const T *, T *, int, int) [with NT=128, Iop=ctc_helper::identity<float, float>, Rop=ctc_helper::maximum<float, float>, T=float]"
(124): here
instantiation of "void ReduceHelper::impl(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::identity<float, float>, Rof=ctc_helper::maximum<float, float>]"
(139): here
instantiation of "ctcStatus_t reduce(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::identity<float, float>, Rof=ctc_helper::maximum<float, float>]"
(157): here
/home/aj/repo/warp-ctc/src/reduce.cu(44): warning: function "__shfl_down(float, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(278): here was declared deprecated ("__shfl_down() is deprecated in favor of __shfl_down_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
detected during:
instantiation of "T CTAReduce<NT, T, Rop>::reduce(int, T, CTAReduce<NT, T, Rop>::Storage &, int, Rop) [with NT=128, T=float, Rop=ctc_helper::add<float, float>]"
(76): here
instantiation of "void reduce_rows<NT,Iop,Rop,T>(Iop, Rop, const T *, T *, int, int) [with NT=128, Iop=ctc_helper::negate<float, float>, Rop=ctc_helper::add<float, float>, T=float]"
(124): here
instantiation of "void ReduceHelper::impl(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::negate<float, float>, Rof=ctc_helper::add<float, float>]"
(139): here
instantiation of "ctcStatus_t reduce(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::negate<float, float>, Rof=ctc_helper::add<float, float>]"
(149): here
/home/aj/repo/warp-ctc/src/reduce.cu(44): warning: function "__shfl_down(float, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(278): here was declared deprecated ("__shfl_down() is deprecated in favor of __shfl_down_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
detected during:
instantiation of "T CTAReduce<NT, T, Rop>::reduce(int, T, CTAReduce<NT, T, Rop>::Storage &, int, Rop) [with NT=128, T=float, Rop=ctc_helper::maximum<float, float>]"
(76): here
instantiation of "void reduce_rows<NT,Iop,Rop,T>(Iop, Rop, const T *, T *, int, int) [with NT=128, Iop=ctc_helper::identity<float, float>, Rop=ctc_helper::maximum<float, float>, T=float]"
(124): here
instantiation of "void ReduceHelper::impl(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::identity<float, float>, Rof=ctc_helper::maximum<float, float>]"
(139): here
instantiation of "ctcStatus_t reduce(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::identity<float, float>, Rof=ctc_helper::maximum<float, float>]"
(157): here
/home/aj/repo/warp-ctc/src/reduce.cu(44): warning: function "__shfl_down(float, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(278): here was declared deprecated ("__shfl_down() is deprecated in favor of __shfl_down_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
detected during:
instantiation of "T CTAReduce<NT, T, Rop>::reduce(int, T, CTAReduce<NT, T, Rop>::Storage &, int, Rop) [with NT=128, T=float, Rop=ctc_helper::add<float, float>]"
(76): here
instantiation of "void reduce_rows<NT,Iop,Rop,T>(Iop, Rop, const T *, T *, int, int) [with NT=128, Iop=ctc_helper::negate<float, float>, Rop=ctc_helper::add<float, float>, T=float]"
(124): here
instantiation of "void ReduceHelper::impl(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::negate<float, float>, Rof=ctc_helper::add<float, float>]"
(139): here
instantiation of "ctcStatus_t reduce(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::negate<float, float>, Rof=ctc_helper::add<float, float>]"
(149): here
/home/aj/repo/warp-ctc/src/reduce.cu(44): warning: function "__shfl_down(float, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(278): here was declared deprecated ("__shfl_down() is deprecated in favor of __shfl_down_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
detected during:
instantiation of "T CTAReduce<NT, T, Rop>::reduce(int, T, CTAReduce<NT, T, Rop>::Storage &, int, Rop) [with NT=128, T=float, Rop=ctc_helper::maximum<float, float>]"
(76): here
instantiation of "void reduce_rows<NT,Iop,Rop,T>(Iop, Rop, const T *, T *, int, int) [with NT=128, Iop=ctc_helper::identity<float, float>, Rop=ctc_helper::maximum<float, float>, T=float]"
(124): here
instantiation of "void ReduceHelper::impl(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::identity<float, float>, Rof=ctc_helper::maximum<float, float>]"
(139): here
instantiation of "ctcStatus_t reduce(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::identity<float, float>, Rof=ctc_helper::maximum<float, float>]"
(157): here
/home/aj/repo/warp-ctc/src/reduce.cu(44): warning: function "__shfl_down(float, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(278): here was declared deprecated ("__shfl_down() is deprecated in favor of __shfl_down_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
detected during:
instantiation of "T CTAReduce<NT, T, Rop>::reduce(int, T, CTAReduce<NT, T, Rop>::Storage &, int, Rop) [with NT=128, T=float, Rop=ctc_helper::add<float, float>]"
(76): here
instantiation of "void reduce_rows<NT,Iop,Rop,T>(Iop, Rop, const T *, T *, int, int) [with NT=128, Iop=ctc_helper::negate<float, float>, Rop=ctc_helper::add<float, float>, T=float]"
(124): here
instantiation of "void ReduceHelper::impl(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::negate<float, float>, Rof=ctc_helper::add<float, float>]"
(139): here
instantiation of "ctcStatus_t reduce(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::negate<float, float>, Rof=ctc_helper::add<float, float>]"
(149): here
/home/aj/repo/warp-ctc/src/reduce.cu(44): warning: function "__shfl_down(float, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(278): here was declared deprecated ("__shfl_down() is deprecated in favor of __shfl_down_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
detected during:
instantiation of "T CTAReduce<NT, T, Rop>::reduce(int, T, CTAReduce<NT, T, Rop>::Storage &, int, Rop) [with NT=128, T=float, Rop=ctc_helper::maximum<float, float>]"
(76): here
instantiation of "void reduce_rows<NT,Iop,Rop,T>(Iop, Rop, const T *, T *, int, int) [with NT=128, Iop=ctc_helper::identity<float, float>, Rop=ctc_helper::maximum<float, float>, T=float]"
(124): here
instantiation of "void ReduceHelper::impl(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::identity<float, float>, Rof=ctc_helper::maximum<float, float>]"
(139): here
instantiation of "ctcStatus_t reduce(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::identity<float, float>, Rof=ctc_helper::maximum<float, float>]"
(157): here
/home/aj/repo/warp-ctc/src/reduce.cu(44): warning: function "__shfl_down(float, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(278): here was declared deprecated ("__shfl_down() is deprecated in favor of __shfl_down_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
detected during:
instantiation of "T CTAReduce<NT, T, Rop>::reduce(int, T, CTAReduce<NT, T, Rop>::Storage &, int, Rop) [with NT=128, T=float, Rop=ctc_helper::add<float, float>]"
(76): here
instantiation of "void reduce_rows<NT,Iop,Rop,T>(Iop, Rop, const T *, T *, int, int) [with NT=128, Iop=ctc_helper::negate<float, float>, Rop=ctc_helper::add<float, float>, T=float]"
(124): here
instantiation of "void ReduceHelper::impl(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::negate<float, float>, Rof=ctc_helper::add<float, float>]"
(139): here
instantiation of "ctcStatus_t reduce(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::negate<float, float>, Rof=ctc_helper::add<float, float>]"
(149): here
/home/aj/repo/warp-ctc/src/reduce.cu(44): warning: function "__shfl_down(float, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(278): here was declared deprecated ("__shfl_down() is deprecated in favor of __shfl_down_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
detected during:
instantiation of "T CTAReduce<NT, T, Rop>::reduce(int, T, CTAReduce<NT, T, Rop>::Storage &, int, Rop) [with NT=128, T=float, Rop=ctc_helper::maximum<float, float>]"
(76): here
instantiation of "void reduce_rows<NT,Iop,Rop,T>(Iop, Rop, const T *, T *, int, int) [with NT=128, Iop=ctc_helper::identity<float, float>, Rop=ctc_helper::maximum<float, float>, T=float]"
(124): here
instantiation of "void ReduceHelper::impl(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::identity<float, float>, Rof=ctc_helper::maximum<float, float>]"
(139): here
instantiation of "ctcStatus_t reduce(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::identity<float, float>, Rof=ctc_helper::maximum<float, float>]"
(157): here
[ 20%] Building NVCC (Device) object CMakeFiles/warpctc.dir/src/warpctc_generated_ctc_entrypoint.cu.o
/home/aj/repo/warp-ctc/include/contrib/moderngpu/include/device/intrinsics.cuh(115): warning: function "__shfl_up(float, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(261): here was declared deprecated ("__shfl_up() is not valid on compute_70 and above, and should be replaced with __shfl_up_sync().To continue using __shfl_up(), specify virtual architecture compute_60 when targeting sm_70 and above, for example, using the pair of compiler options: -arch=compute_60 -code=sm_70.")
/home/aj/repo/warp-ctc/include/contrib/moderngpu/include/device/intrinsics.cuh(125): warning: function "__shfl_up(int, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(175): here was declared deprecated ("__shfl_up() is not valid on compute_70 and above, and should be replaced with __shfl_up_sync().To continue using __shfl_up(), specify virtual architecture compute_60 when targeting sm_70 and above, for example, using the pair of compiler options: -arch=compute_60 -code=sm_70.")
/home/aj/repo/warp-ctc/include/contrib/moderngpu/include/device/intrinsics.cuh(126): warning: function "__shfl_up(int, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(175): here was declared deprecated ("__shfl_up() is not valid on compute_70 and above, and should be replaced with __shfl_up_sync().To continue using __shfl_up(), specify virtual architecture compute_60 when targeting sm_70 and above, for example, using the pair of compiler options: -arch=compute_60 -code=sm_70.")
/home/aj/repo/warp-ctc/include/contrib/moderngpu/include/device/intrinsics.cuh(115): warning: function "__shfl_up(float, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(261): here was declared deprecated ("__shfl_up() is deprecated in favor of __shfl_up_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
/home/aj/repo/warp-ctc/include/contrib/moderngpu/include/device/intrinsics.cuh(125): warning: function "__shfl_up(int, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(175): here was declared deprecated ("__shfl_up() is deprecated in favor of __shfl_up_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
/home/aj/repo/warp-ctc/include/contrib/moderngpu/include/device/intrinsics.cuh(126): warning: function "__shfl_up(int, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(175): here was declared deprecated ("__shfl_up() is deprecated in favor of __shfl_up_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
/home/aj/repo/warp-ctc/include/contrib/moderngpu/include/device/intrinsics.cuh(115): warning: function "__shfl_up(float, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(261): here was declared deprecated ("__shfl_up() is deprecated in favor of __shfl_up_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
/home/aj/repo/warp-ctc/include/contrib/moderngpu/include/device/intrinsics.cuh(125): warning: function "__shfl_up(int, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(175): here was declared deprecated ("__shfl_up() is deprecated in favor of __shfl_up_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
/home/aj/repo/warp-ctc/include/contrib/moderngpu/include/device/intrinsics.cuh(126): warning: function "__shfl_up(int, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(175): here was declared deprecated ("__shfl_up() is deprecated in favor of __shfl_up_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
/home/aj/repo/warp-ctc/include/contrib/moderngpu/include/device/intrinsics.cuh(115): warning: function "__shfl_up(float, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(261): here was declared deprecated ("__shfl_up() is deprecated in favor of __shfl_up_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
/home/aj/repo/warp-ctc/include/contrib/moderngpu/include/device/intrinsics.cuh(125): warning: function "__shfl_up(int, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(175): here was declared deprecated ("__shfl_up() is deprecated in favor of __shfl_up_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
/home/aj/repo/warp-ctc/include/contrib/moderngpu/include/device/intrinsics.cuh(126): warning: function "__shfl_up(int, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(175): here was declared deprecated ("__shfl_up() is deprecated in favor of __shfl_up_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
/home/aj/repo/warp-ctc/include/contrib/moderngpu/include/device/intrinsics.cuh(115): warning: function "__shfl_up(float, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(261): here was declared deprecated ("__shfl_up() is deprecated in favor of __shfl_up_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
/home/aj/repo/warp-ctc/include/contrib/moderngpu/include/device/intrinsics.cuh(125): warning: function "__shfl_up(int, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(175): here was declared deprecated ("__shfl_up() is deprecated in favor of __shfl_up_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
/home/aj/repo/warp-ctc/include/contrib/moderngpu/include/device/intrinsics.cuh(126): warning: function "__shfl_up(int, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(175): here was declared deprecated ("__shfl_up() is deprecated in favor of __shfl_up_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
/home/aj/repo/warp-ctc/include/contrib/moderngpu/include/device/intrinsics.cuh(115): warning: function "__shfl_up(float, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(261): here was declared deprecated ("__shfl_up() is deprecated in favor of __shfl_up_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
/home/aj/repo/warp-ctc/include/contrib/moderngpu/include/device/intrinsics.cuh(125): warning: function "__shfl_up(int, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(175): here was declared deprecated ("__shfl_up() is deprecated in favor of __shfl_up_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
/home/aj/repo/warp-ctc/include/contrib/moderngpu/include/device/intrinsics.cuh(126): warning: function "__shfl_up(int, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(175): here was declared deprecated ("__shfl_up() is deprecated in favor of __shfl_up_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
/home/aj/repo/warp-ctc/include/contrib/moderngpu/include/device/intrinsics.cuh(115): warning: function "__shfl_up(float, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(261): here was declared deprecated ("__shfl_up() is deprecated in favor of __shfl_up_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
/home/aj/repo/warp-ctc/include/contrib/moderngpu/include/device/intrinsics.cuh(125): warning: function "__shfl_up(int, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(175): here was declared deprecated ("__shfl_up() is deprecated in favor of __shfl_up_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
/home/aj/repo/warp-ctc/include/contrib/moderngpu/include/device/intrinsics.cuh(126): warning: function "__shfl_up(int, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(175): here was declared deprecated ("__shfl_up() is deprecated in favor of __shfl_up_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
/home/aj/repo/warp-ctc/include/contrib/moderngpu/include/device/intrinsics.cuh(115): warning: function "__shfl_up(float, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(261): here was declared deprecated ("__shfl_up() is deprecated in favor of __shfl_up_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
/home/aj/repo/warp-ctc/include/contrib/moderngpu/include/device/intrinsics.cuh(125): warning: function "__shfl_up(int, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(175): here was declared deprecated ("__shfl_up() is deprecated in favor of __shfl_up_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
/home/aj/repo/warp-ctc/include/contrib/moderngpu/include/device/intrinsics.cuh(126): warning: function "__shfl_up(int, unsigned int, int)"
/usr/local/cuda/include/sm_30_intrinsics.hpp(175): here was declared deprecated ("__shfl_up() is deprecated in favor of __shfl_up_sync() and may be removed in a future release (Use -Wno-deprecated-declarations to suppress this warning).")
Scanning dependencies of target warpctc
[ 30%] Linking CXX shared library libwarpctc.so
[ 30%] Built target warpctc
Scanning dependencies of target test_cpu
[ 40%] Building CXX object CMakeFiles/test_cpu.dir/tests/test_cpu.cpp.o
[ 50%] Linking CXX executable test_cpu
[ 50%] Built target test_cpu
[ 60%] Building NVCC (Device) object CMakeFiles/test_gpu.dir/tests/test_gpu_generated_test_gpu.cu.o
Scanning dependencies of target test_gpu
[ 70%] Linking CXX executable test_gpu
[ 70%] Built target test_gpu
Scanning dependencies of target warp_ctc
[ 80%] Building CXX object CMakeFiles/warp_ctc.dir/torch_binding/binding.cpp.o
[ 90%] Building C object CMakeFiles/warp_ctc.dir/torch_binding/utils.c.o
[100%] Linking CXX shared module libwarp_ctc.so
[100%] Built target warp_ctc

also sudo make install is done with no error:

[ 30%] Built target warpctc
[ 50%] Built target test_cpu
[ 70%] Built target test_gpu
[100%] Built target warp_ctc
Install the project...
-- Install configuration: ""
-- Up-to-date: /home/aj/torch/install/lib/libwarpctc.so
-- Up-to-date: /home/aj/torch/install/include/ctc.h
-- Up-to-date: /home/aj/torch/install/lib/libwarpctc.so
-- Up-to-date: /home/aj/torch/install/../../../../usr/local/lib/libwarp_ctc.so
-- Up-to-date: /home/aj/torch/install/../../../../usr/local/lua/warp_ctc/init.lua

I install pytorch_binding by:

aj@aj-pc:~/repo/warp-ctc/build$ cd ../pytorch_binding/
aj@aj-pc:~/repo/warp-ctc/pytorch_binding$ CUDA_HOME=/usr/local/cuda sudo -E python3 setup.py install
generating build/_warp_ctc.c
(already up-to-date)
not modified: 'build/_warp_ctc.c'
running install
running build
running build_py
running build_ext
running install_lib
running install_egg_info
Removing /usr/local/lib/python3.5/dist-packages/warpctc_pytorch-0.1.egg-info
Writing /usr/local/lib/python3.5/dist-packages/warpctc_pytorch-0.1.egg-info
aj@aj-pc:~/repo/warp-ctc/pytorch_binding$ 

P.S.: I downgrade gcc and g++ compiler to 4.8.5 due to building problems.

[mohamad-hasan-sohan-ajini]

hurray... I solved the problem after two days of hard work.

I tried lots of things, so I can not tell exactly what solved the problem. But there are some hints I think are important to solve the issue:

1- gcc and g++ compiler version must be <= 4.9. Compiling warp-ctc with greater versions cause error.

2- I think there are some problems with cuda 9.1. I finally accomplished with cuda 9.0. (but I'm not sure about it)

3-Your fortran version (needed to compile torch) must be the same as g++ and gcc version.

4- Add build folder to LD_LIBRARY_PATH and run ./tets_cp and ./test_gpu. It helps to know if the warp-ctc compilation has problem or not.

thanks to @SeanNaren for his attention. Please correct me if I'm wrong.

[SeanNaren]

@Hajix thanks for posting up solutions, its always been tricky getting warp-ctc installed. I think 1) is crucial. 2) I'm not entirely sure about since I've compiled warp-ctc successfully on CUDA 9.1! Glad your problem is fixed nonetheless