baidu-research / persistent-rnn Goto Github PK

When I tried to build project with
scons mode=release install=true
it gives the following error:
scons: Reading SConscript files ...
Building NVCC tool
scons: warning: Two different environments were specified for target src/detail/matrix/copy_operations.os, but they appear to have the same action: $SHCXX -o $TARGET -c $SHCXXFLAGS $SHCCFLAGS $_CCCOMCOM $SOURCES
File "/home/ybk/workspace/persistent-rnn/SConscript", line 45, in <module> scons: *** Multiple ways to build the same target were specified for: src/detail/matrix/copy_operations.os (from ['/home/ybk/workspace/persistent-rnn/src/detail/matrix/copy_operations.cu'] and from ['src/detail/matrix/copy_operations.cpp'])
File "/home/ybk/workspace/persistent-rnn/SConscript", line 45, in <module>

So I tried another way to build it with
source ubuntu-64-environment.sh
sudo scons install=true
it gives another error like:
scons:Reading SConscript files ...
Building NVCC tool
Installing Persistent RNN Kernels
scons: done reading SConscript files.
scons: Building targets ...
nvcc -o .debug_build/benchmark/benchmark-atomic-throughput.o -c -Xcompiler -g -Xcompiler -DPRNN_DEBUG -Xcompiler -Wall -Xcompiler -Werror -I /usr/local/cuda/include -I .debug_build/include -I include -gencode=arch=compute_30,code=sm_30 -G -std=c++11 "-D CUDA_ARCH_MAJOR=3" -Xcompiler=-Wno-unused-function -Xcompiler=-Wno-unused-private-field benchmark/benchmark-atomic-throughput.cu
In file included from /usr/local/cuda/include/cuda_runtime.h:112:0, from <command-line>:0: /usr/local/cuda/include/common_functions.h:87:15: fatal error: new: No such file or directory #include <new>
compilation terminated.
scons: *** [.debug_build/benchmark/benchmark-atomic-throughput.o] Error 1
scons: building terminated because of errors.

At first I thought it was the problem of clang++ to find c++ include files, so i write an example named new_feature to test whether clang++ can do #include <new>
#include <iostream>
#include <string>
#include <new>
class MyClass
{
public:
std::string s ="Hello, world\n"; // Non-static data member initializer
};
int main()
{
std::cout << MyClass().s;
}
Then I run it with:
clang++ -std=c++11 -stdlib=libc++ -Werror -Weverything -Wno-disabled-macro-expansion -Wno-float-equal -Wno-c++98-compat -Wno-c++98-compat-pedantic -Wno-global-constructors -Wno-exit-time-destructors -Wno-missing-prototypes -Wno-padded -Wno-old-style-cast -lc++ -lc++abi new_feature.cpp
and it runs well and gives the output file.
So I don't know why the project gives me an error like that when I try to build it...

hi, I used prnn on my P40(30sm and 128sp) using same config with titan x(sm>=24, major=5). When I used mini-batch=4, layer-size=1152, layers=1, timesteps=300, it only has 1.3TFlops. In my opinion, P40 is faster than titan x. So it should has 2.8TFlops the same with titan x. Why dones‘t it. Error config or some other reasons？ Please help me find the mistakes.

I am using 980 Ti, Ubuntu 16.04 LTS, CUDA 7.5

Below is the gdb printout

Thread 1 "persistent-rnn-" received signal SIGBUS, Bus error.
__memmove_avx_unaligned ()
at ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S:146
146 ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S: No such file or directory.
(gdb) bt
#0 __memmove_avx_unaligned ()

at ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S:146

#1 0x00007ffff532c83f in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#2 0x00007ffff53cf81e in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#3 0x00007ffff53cfa4b in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#4 0x00007ffff53d0b5f in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#5 0x00007ffff514e1a2 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#6 0x00007ffff514e8c5 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#7 0x00007ffff544cda2 in cuMemcpyAsync ()

from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#8 0x00007ffff623a33d in ?? () from /usr/local/cuda/lib64/libcudart.so
#9 0x00007ffff621ccbb in ?? () from /usr/local/cuda/lib64/libcudart.so
#10 0x00007ffff624f628 in cudaMemcpyAsync ()

from /usr/local/cuda/lib64/libcudart.so
#11 0x00007ffff776ae7b in prnn::parallel::CudaRuntimeLibrary::cudaMemcpyAsync(void_, void const_, unsigned long, prnn::parallel::CudaRuntimeLibrary::cudaMemcpyKind, void*) ()

from /home/mewang/projects/persistent-rnn/.release_build/libprnn.so
#12 0x00007ffff7841f9b in void prnn::rnn::detail::dispatchForwardPropRecurrent<prnn::matrix::RectifiedLinear, prnn::rnn::RecurrentArchitectureParameters<float, prnn::rnn::TileConfig<1, 8, 8, 4, 4, 2, 4, 0, float> > >(prnn::rnn::RecurrentArchitectureParameters<float, prnn::rnn::TileConfig<1, 8, 8, 4, 4, 2, 4, 0, float> >::RealType_, prnn::rnn::RecurrentArchitectureParameters<float, prnn::rnn::TileConfig<1, 8, 8, 4, 4, 2, 4, 0, float> >::RealType const_, prnn::rnn::RecurrentArchitectureParameters<float, prnn::rnn::TileConfig<1, 8, 8, 4, 4, 2, 4, 0, float> >::RealType*, prnn::rnn::RecurrentArchitectureParameters<float, prnn::rnn::TileConfig<1, 8, 8, 4, 4, 2, 4, 0, float> > const&) ()

from /home/mewang/projects/persistent-rnn/.release_build/libprnn.so
#13 0x00007ffff7842f8c in void prnn::rnn::detail::forwardPropRecurrent<prnn::matrix::RectifiedLinear, prnn::matrix::SinglePrecision, (prnn::RecurrentLayerDirection)0>(prnn::matrix::DynamicView const&, prnn::matrix::ConstDynamicView const&, prnn::matrix::DynamicView const&, prnn::RecurrentOpsHandle const&, std::tupleprnn::matrix::SinglePrecision const&) ()

from /home/mewang/projects/persistent-rnn/.release_build/libprnn.so
#14 0x00007ffff7840062 in prnn::rnn::detail::forwardPropRecurrentOverActivationFunctions(prnn::matrix::DynamicView const&, prnn::matrix::ConstDynamicView const&, prnn::matrix::DynamicView const&, prnn::RecurrentOpsHandle const&) ()

from /home/mewang/projects/persistent-rnn/.release_build/libprnn.so
#15 0x00007ffff7752ecf in prnnRNNForward ()

from /home/mewang/projects/persistent-rnn/.release_build/libprnn.so
#16 0x0000000000407d8b in TestCForwardOps(Options const&) ()
#17 0x0000000000406ccb in RunTest(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, void (*)(Options const&), Options&)

()

#18 0x00000000004049e1 in main ()

I get the following output when I follow the directions in the README file...

What should I do to fix this problem? Thanks!

Hey!

I'm having difficulties getting the library to work correctly with my GTX 1080.

It doesn't seem to recognize that my graphics card is of sm_60 and arch major 6. If i manually change variables in the nvcc.py file to force sm_60 it does compile.

This goes into my next issue which is the cudaDevAttrMultiProcessorCount is returning 20 when calling getMaximumSizeRNNForThisGPU, which means I don't get the tile sizes associated with pascal.

Also I had to make a modification here, I assume line 54 in recurrent_ops.cu:

typedef TileConfig<56, 2432, 2560, 352, 320, 11, 16, direction, T> TileSize;

is supposed to be:

typedef TileConfig<56, 2432, 2560, 352, 320, 11, 16, direction, float16> TileSize;

Hopefully can solve the issue, let me know if any more information is needed!

I'm running Ubuntu 14.04 using CUDA 8.0, V8.0.26 (RC) with driver (installed after CUDA as suggested for 1080 drivers) 367.35.

Great work putting this together!

I am trying to run PRNN on a GeForce RTX 2080(46 SM, 7.7 compute).
I've tried the following tile sizes,
TileConfig<24, 1152, 1152, 192, 288, 6, 36, direction, T>
TileConfig<32, 1024, 1024, 128, 256, 4, 32, direction, T>
TileConfig<32, 1024, 1024, 64, 512, 1, 32, direction, T>
TileConfig<40, 640, 640, 80, 128, 5, 4, direction, T>

Running benchmark using any of these with batchsize=4, timesteps=20, and layer sizes max for each tile configuration, the fastest I can get is 0.00478542 TFLOPS/s in the forward run.

Are the tile sizes inappropriate or is the issue something else.

Thank you.

Hey! When building the library, running scons mode=release install=true terminates with this error:

src/detail/rnn/recurrent_ops.cu: In function 'size_t prnn::rnn::getForwardPropScratchSize(const prnn::RecurrentOpsHandle&, const prnn::matrix::Precision&)':
src/detail/rnn/recurrent_ops.cu:789:6: error: variable 'scratchDimension' set but not used [-Werror=unused-but-set-variable]
     auto scratchDimension = extendDimensions(dimension, precision);
      ^
At global scope:
cc1plus: error: unrecognized command line option "-Wno-unused-private-field" [-Werror]
cc1plus: all warnings being treated as errors
scons: *** [.release_build/src/detail/rnn/recurrent_ops.os] Error 1
scons: building terminated because of errors.

Running on Ubuntu 14.04 with a GTX 1080. Thanks!

Gpu: titanX
ubuntu 14.04
cuda 3.007

I get the following error when running persistent-rnn-benchmark
I think libcudnn.so is used for this particular function call

xyz@ubuntu15:~/bin$ ./persistent-rnn-benchmark 
terminate called after throwing an instance of 'std::runtime_error'
  what():  Failed to load function 'cudnnCreateDropoutDescriptor' from dynamic library.
Aborted (core dumped)

Please let me know if you need more details to resolve this issue
@gdiamos

I'm considering adding TensorFlow bindings to persistent-rnn, and I've been reading the codebase for my own understanding.

In https://github.com/baidu-research/persistent-rnn/blob/master/include/prnn/detail/rnn/synchronizer.h#L86

    bool get_failed_() const {
        index_t failed = 0;

        prnn::parallel::CudaRuntimeLibrary::cudaMemcpyAsync(&failed,
            get_barrier_failed_flag(), sizeof(index_t),
            prnn::parallel::CudaRuntimeLibrary::cudaMemcpyDefault, stream_);

        return failed != 0;
    }

It seems to me that this has has a race condition between function return (failed going out of scope) and cudaMemcpyAsync completing. Is this the case? If not, what is causing synchronization of the memcpy before function return?
There are several other similar examples to this in Synchronizer.

Thanks for any comment on this "maybe not an issue but just a question about code" issue :)

I followed the suggestion of removing line 789 from src/detail/rnn/recurrent_ops.cu which solved the scratchDimension error but I ran into another error after
sarunac4@TitanX01:~/baidu/persistent-rnn$ scons mode=release install=true scons: Reading SConscript files ... Building NVCC tool Installing Persistent RNN Kernels scons: done reading SConscript files. scons: Building targets ... nvcc -o .release_build/benchmark/benchmark-atomic-throughput.o -c -Xcompiler -O3 -Xcompiler -Wall -Xcompiler -Werror -I /usr/local/cuda/include -I .release_build/include -I include -gencode=arch=compute_30,code=sm_30 -std=c++11 "-D CUDA_ARCH_MAJOR=3" -Xcompiler=-Wno-unused-function -Xcompiler=-Wno-unused-private-field benchmark/benchmark-atomic-throughput.cu g++ -o .release_build/src/persistent_rnn.os -c -O3 -Wall -Werror -std=c++11 -fPIC -I/usr/local/cuda/include -I.release_build/include -Iinclude src/persistent_rnn.cpp g++ -o .release_build/src/persistent_rnn_high_level.os -c -O3 -Wall -Werror -std=c++11 -fPIC -I/usr/local/cuda/include -I.release_build/include -Iinclude src/persistent_rnn_high_level.cpp g++ -o .release_build/src/detail/matrix/allocation.os -c -O3 -Wall -Werror -std=c++11 -fPIC -I/usr/local/cuda/include -I.release_build/include -Iinclude src/detail/matrix/allocation.cpp g++ -o .release_build/src/detail/matrix/atlas_library.os -c -O3 -Wall -Werror -std=c++11 -fPIC -I/usr/local/cuda/include -I.release_build/include -Iinclude src/detail/matrix/atlas_library.cpp g++ -o .release_build/src/detail/matrix/blas_operations.os -c -O3 -Wall -Werror -std=c++11 -fPIC -I/usr/local/cuda/include -I.release_build/include -Iinclude src/detail/matrix/blas_operations.cpp g++ -o .release_build/src/detail/matrix/cublas_library.os -c -O3 -Wall -Werror -std=c++11 -fPIC -I/usr/local/cuda/include -I.release_build/include -Iinclude src/detail/matrix/cublas_library.cpp g++ -o .release_build/src/detail/matrix/curand_library.os -c -O3 -Wall -Werror -std=c++11 -fPIC -I/usr/local/cuda/include -I.release_build/include -Iinclude src/detail/matrix/curand_library.cpp g++ -o .release_build/src/detail/matrix/float_iterator.os -c -O3 -Wall -Werror -std=c++11 -fPIC -I/usr/local/cuda/include -I.release_build/include -Iinclude src/detail/matrix/float_iterator.cpp g++ -o .release_build/src/detail/matrix/float_reference.os -c -O3 -Wall -Werror -std=c++11 -fPIC -I/usr/local/cuda/include -I.release_build/include -Iinclude src/detail/matrix/float_reference.cpp g++ -o .release_build/src/detail/matrix/matrix.os -c -O3 -Wall -Werror -std=c++11 -fPIC -I/usr/local/cuda/include -I.release_build/include -Iinclude src/detail/matrix/matrix.cpp g++ -o .release_build/src/detail/matrix/matrix_transformations.os -c -O3 -Wall -Werror -std=c++11 -fPIC -I/usr/local/cuda/include -I.release_build/include -Iinclude src/detail/matrix/matrix_transformations.cpp g++ -o .release_build/src/detail/matrix/operation.os -c -O3 -Wall -Werror -std=c++11 -fPIC -I/usr/local/cuda/include -I.release_build/include -Iinclude src/detail/matrix/operation.cpp g++ -o .release_build/src/detail/matrix/precision.os -c -O3 -Wall -Werror -std=c++11 -fPIC -I/usr/local/cuda/include -I.release_build/include -Iinclude src/detail/matrix/precision.cpp g++ -o .release_build/src/detail/matrix/random_operations.os -c -O3 -Wall -Werror -std=c++11 -fPIC -I/usr/local/cuda/include -I.release_build/include -Iinclude src/detail/matrix/random_operations.cpp g++ -o .release_build/src/detail/parallel/cuda.os -c -O3 -Wall -Werror -std=c++11 -fPIC -I/usr/local/cuda/include -I.release_build/include -Iinclude src/detail/parallel/cuda.cpp g++ -o .release_build/src/detail/parallel/cuda_driver.os -c -O3 -Wall -Werror -std=c++11 -fPIC -I/usr/local/cuda/include -I.release_build/include -Iinclude src/detail/parallel/cuda_driver.cpp g++ -o .release_build/src/detail/parallel/cuda_runtime_library.os -c -O3 -Wall -Werror -std=c++11 -fPIC -I/usr/local/cuda/include -I.release_build/include -Iinclude src/detail/parallel/cuda_runtime_library.cpp g++ -o .release_build/src/detail/parallel/memory.os -c -O3 -Wall -Werror -std=c++11 -fPIC -I/usr/local/cuda/include -I.release_build/include -Iinclude src/detail/parallel/memory.cpp g++ -o .release_build/src/detail/parallel/parallel_for.os -c -O3 -Wall -Werror -std=c++11 -fPIC -I/usr/local/cuda/include -I.release_build/include -Iinclude src/detail/parallel/parallel_for.cpp g++ -o .release_build/src/detail/parallel/synchronization.os -c -O3 -Wall -Werror -std=c++11 -fPIC -I/usr/local/cuda/include -I.release_build/include -Iinclude src/detail/parallel/synchronization.cpp g++ -o .release_build/src/detail/util/knob_file.os -c -O3 -Wall -Werror -std=c++11 -fPIC -I/usr/local/cuda/include -I.release_build/include -Iinclude src/detail/util/knob_file.cpp g++ -o .release_build/src/detail/util/knobs.os -c -O3 -Wall -Werror -std=c++11 -fPIC -I/usr/local/cuda/include -I.release_build/include -Iinclude src/detail/util/knobs.cpp g++ -o .release_build/src/detail/util/logger.os -c -O3 -Wall -Werror -std=c++11 -fPIC -I/usr/local/cuda/include -I.release_build/include -Iinclude src/detail/util/logger.cpp g++ -o .release_build/src/detail/util/system_compatibility.os -c -O3 -Wall -Werror -std=c++11 -fPIC -I/usr/local/cuda/include -I.release_build/include -Iinclude src/detail/util/system_compatibility.cpp g++ -o .release_build/src/detail/rnn/recurrent_ops_handle.os -c -O3 -Wall -Werror -std=c++11 -fPIC -I/usr/local/cuda/include -I.release_build/include -Iinclude src/detail/rnn/recurrent_ops_handle.cpp nvcc -o .release_build/src/detail/matrix/copy_operations.os -c -shared -Xcompiler -O3 -Xcompiler -Wall -Xcompiler -Werror -Xcompiler -fPIC -I /usr/local/cuda/include -I .release_build/include -I include -gencode=arch=compute_30,code=sm_30 -std=c++11 "-D CUDA_ARCH_MAJOR=3" -Xcompiler=-Wno-unused-function -Xcompiler=-Wno-unused-private-field src/detail/matrix/copy_operations.cu nvcc -o .release_build/src/detail/matrix/matrix_operations.os -c -shared -Xcompiler -O3 -Xcompiler -Wall -Xcompiler -Werror -Xcompiler -fPIC -I /usr/local/cuda/include -I .release_build/include -I include -gencode=arch=compute_30,code=sm_30 -std=c++11 "-D CUDA_ARCH_MAJOR=3" -Xcompiler=-Wno-unused-function -Xcompiler=-Wno-unused-private-field src/detail/matrix/matrix_operations.cu nvcc -o .release_build/src/detail/rnn/recurrent_ops.os -c -shared -Xcompiler -O3 -Xcompiler -Wall -Xcompiler -Werror -Xcompiler -fPIC -I /usr/local/cuda/include -I .release_build/include -I include -gencode=arch=compute_30,code=sm_30 -std=c++11 "-D CUDA_ARCH_MAJOR=3" -Xcompiler=-Wno-unused-function -Xcompiler=-Wno-unused-private-field src/detail/rnn/recurrent_ops.cu g++ -o .release_build/libprnn.so -shared -Wl,-rpath=/usr/local/cuda/lib64 .release_build/src/persistent_rnn.os .release_build/src/persistent_rnn_high_level.os .release_build/src/detail/matrix/allocation.os .release_build/src/detail/matrix/atlas_library.os .release_build/src/detail/matrix/blas_operations.os .release_build/src/detail/matrix/cublas_library.os .release_build/src/detail/matrix/curand_library.os .release_build/src/detail/matrix/float_iterator.os .release_build/src/detail/matrix/float_reference.os .release_build/src/detail/matrix/matrix.os .release_build/src/detail/matrix/matrix_transformations.os .release_build/src/detail/matrix/operation.os .release_build/src/detail/matrix/precision.os .release_build/src/detail/matrix/random_operations.os .release_build/src/detail/parallel/cuda.os .release_build/src/detail/parallel/cuda_driver.os .release_build/src/detail/parallel/cuda_runtime_library.os .release_build/src/detail/parallel/memory.os .release_build/src/detail/parallel/parallel_for.os .release_build/src/detail/parallel/synchronization.os .release_build/src/detail/util/knob_file.os .release_build/src/detail/util/knobs.os .release_build/src/detail/util/logger.os .release_build/src/detail/util/system_compatibility.os .release_build/src/detail/rnn/recurrent_ops_handle.os .release_build/src/detail/matrix/copy_operations.os .release_build/src/detail/matrix/matrix_operations.os .release_build/src/detail/rnn/recurrent_ops.os -L/usr/local/cuda/lib64 -L/usr/local/lib -lcudart_static -lrt -ldl Install file: ".release_build/libprnn.so" as "/usr/local/lib/libprnn.so" scons: *** [/usr/local/lib/libprnn.so] /usr/local/lib/libprnn.so: Permission denied scons: building terminated because of errors.
The system is running on
OS: Ubuntu 14.04
CPU: Intel(R) Xeon(R) CPU E5-1607
GPU: GeForce GTX TITAN X

Running tests for RNN (512, 7, 2, 1) (ReLU, forward, lstm, linear input, best)
Test 'C Interface Tensor Test' Passed
Test 'C Interface RNN Test' Passed
Test 'C Interface Forward Ops Test' Failed with error 'invalid device function'

I am running on NVIDIA P100.
Could you please let me know how to go about correcting this error... Thank you.

I have been trying to compile the benchmarks for some time now and always get errors of the flavor in the attached file. Apparently GCC porting has been responsible for errors like this, but I also notice that persistent-rnn/include/prnn/detail/parallel/assert.h redefines assert() right in the middle of things. I haven't looked all the way down to see if this redefinition is the source of the problem but would appreciate any insight into what's going on. Thanks.

What I've tried: #include <cassert>, #include <assert.h>, setting/unsetting the NDEBUG macro, and using -fpermissive in compilation.

pcompile_output.txt