Training error in main.py about tensorflow-wavenet

[ibab]

That's strange. Which version of TensorFlow are you using?
It might also be a Python 2 problem (I'm developing on Python 3).

[polyrhythmatic]

I was wondering if that was the case. I just tried using python 3.5 with tensorflow 0.10.0 and got the following error:

I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties: 
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.076
pciBusID 0000:01:00.0
Total memory: 12.00GiB
Free memory: 11.51GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:838] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:01:00.0)
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/tensor_shape.py", line 563, in merge_with
    self.assert_same_rank(other)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/tensor_shape.py", line 609, in assert_same_rank
    "Shapes %s and %s must have the same rank" % (self, other))
ValueError: Shapes (?, ?, ?, 256) and (?, 256) must have the same rank

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 129, in <module>
    main()
  File "main.py", line 83, in main
    loss = net.loss(audio_batch)
  File "/home/seth/Development/tensorflow-wavenet/wavenet.py", line 105, in loss
    loss = tf.nn.softmax_cross_entropy_with_logits(raw_output, tf.reshape(shifted, [-1, self.channels]))
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/nn_ops.py", line 491, in softmax_cross_entropy_with_logits
    precise_logits, labels, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 1427, in _softmax_cross_entropy_with_logits
    features=features, labels=labels, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 703, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 2319, in create_op
    set_shapes_for_outputs(ret)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1711, in set_shapes_for_outputs
    shapes = shape_func(op)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/nn_ops.py", line 603, in _SoftmaxCrossEntropyWithLogitsShape
    input_shape = logits_shape.merge_with(labels_shape).with_rank(2)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/tensor_shape.py", line 570, in merge_with
    (self, other))
ValueError: Shapes (?, ?, ?, 256) and (?, 256) are not compatible

Im running ubuntu 16.04 if that matters.

[adamduracz]

Getting a similar error to @polyrhythmatic with Python 2.7 and TF 0.10.0:

> python main.py
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.1.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.dylib locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] OS X does not support NUMA - returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties:
name: GeForce GT 650M
major: 3 minor: 0 memoryClockRate (GHz) 0.9
pciBusID 0000:01:00.0
Total memory: 1023.69MiB
Free memory: 53.51MiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:838] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GT 650M, pci bus id: 0000:01:00.0)
Traceback (most recent call last):
  File "main.py", line 129, in <module>
    main()
  File "main.py", line 83, in main
    loss = net.loss(audio_batch)
  File "/Users/adam/work/python/tensorflow-wavenet/wavenet.py", line 105, in loss
    loss = tf.nn.softmax_cross_entropy_with_logits(raw_output, tf.reshape(shifted, [-1, self.channels]))
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/nn_ops.py", line 491, in softmax_cross_entropy_with_logits
    precise_logits, labels, name=name)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 1427, in _softmax_cross_entropy_with_logits
    features=features, labels=labels, name=name)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 703, in apply_op
    op_def=op_def)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2319, in create_op
    set_shapes_for_outputs(ret)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1711, in set_shapes_for_outputs
    shapes = shape_func(op)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/nn_ops.py", line 603, in _SoftmaxCrossEntropyWithLogitsShape
    input_shape = logits_shape.merge_with(labels_shape).with_rank(2)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/tensor_shape.py", line 570, in merge_with
    (self, other))
ValueError: Shapes (?, ?, ?, 256) and (?, 256) are not compatible

[polyrhythmatic]

So I don't really have any idea what I'm doing, but it seems to me that maybe the issue is with line 105 in wavenet and some sort of shape mismatch between the arguments to tf.nn.softmax_cross_entropy_with_logits

with no arguments to

python main.py

I get

    "Shapes %s and %s must have the same rank" % (self, other))
ValueError: Shapes (?, ?, ?, 256) and (?, 256) must have the same rank

but when I provide a channel argument of 3

python main.py --channels 3

One of the shapes is more defined

    "Shapes %s and %s must have the same rank" % (self, other))
ValueError: Shapes (3, ?, ?, 256) and (?, 256) must have the same rank

Not sure if this makes any difference.

[ibab]

I'll try to fix this tomorrow.
It might be because I've written the network using the current git master of TensorFlow, or there might be a bug.

[lelayf]

I have the same error here TF r0.10 on Mac OS X and python 3.5.2 in virtualenv.

[finlay-liu]

I get the same error with @adamduracz...

[ibab]

The shape error resulted from a small API difference in TF 0.10 and the current git TF.
The fix is in e67b137.
I'll keep developing on TF 0.10 from now :)

I'm not sure what's causing the first error in this issue, so I'm keeping it open.

[nonstop99]

Hi, I was getting the same error as above, I tried the latest files from the git but now i'm getting..

osboxes@osboxes:~/Documents/wavenet/wavenet$ sudo python main.py Traceback (most recent call last): File "main.py", line 129, in <module> main() File "main.py", line 119, in main summary, loss_value, _ = sess.run([summaries, loss, optim]) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 710, in run run_metadata_ptr) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 908, in _run feed_dict_string, options, run_metadata) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 958, in _do_run target_list, options, run_metadata) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 978, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors.OutOfRangeError: PaddingFIFOQueue '_3_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0) [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]] Caused by op u'create_inputs/padding_fifo_queue_DequeueMany', defined at: File "main.py", line 129, in <module> main() File "main.py", line 78, in main audio_batch, _ = queue.dequeue_many(args.batch_size) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/data_flow_ops.py", line 435, in dequeue_many self._queue_ref, n=n, component_types=self._dtypes, name=name) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 867, in _queue_dequeue_many timeout_ms=timeout_ms, name=name) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 703, in apply_op op_def=op_def) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2317, in create_op original_op=self._default_original_op, op_def=op_def) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1239, in __init__ self._traceback = _extract_stack()

[adamduracz]

@ibab e67b137 indeed resolved the issue I posted, but that was hiding another one:

> python main.py
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.1.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.dylib locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] OS X does not support NUMA - returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties:
name: GeForce GT 650M
major: 3 minor: 0 memoryClockRate (GHz) 0.9
pciBusID 0000:01:00.0
Total memory: 1023.69MiB
Free memory: 402.64MiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:838] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GT 650M, pci bus id: 0000:01:00.0)
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
Traceback (most recent call last):
  File "main.py", line 129, in <module>
    main()
  File "main.py", line 119, in main
    summary, loss_value, _ = sess.run([summaries, loss, optim])
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 710, in run
    run_metadata_ptr)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 908, in _run
    feed_dict_string, options, run_metadata)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 958, in _do_run
    target_list, options, run_metadata)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 978, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.OutOfRangeError: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
Caused by op u'create_inputs/padding_fifo_queue_DequeueMany', defined at:
  File "main.py", line 129, in <module>
    main()
  File "main.py", line 78, in main
    audio_batch, _ = queue.dequeue_many(args.batch_size)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/data_flow_ops.py", line 435, in dequeue_many
    self._queue_ref, n=n, component_types=self._dtypes, name=name)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 867, in _queue_dequeue_many
    timeout_ms=timeout_ms, name=name)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 703, in apply_op
    op_def=op_def)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2317, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/Users/adam/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1239, in __init__
    self._traceback = _extract_stack()

[nonstop99]

I'm using a virtulbox image so I can make that available if it helps?

[ibab]

That looks like it's not finding the data directory.
I'll make sure to add a check so that there's a better error message in this case.

[nonstop99]

Ok is there anything we need to do to fix it?

btw, I tried using Python 3.5 but got the same error

[polyrhythmatic]

@ibab thanks for all your work on this! I've tried specifying the data directory and still can't seem to get anywhere with it. Any recommendations?

[ibab]

I've added a bit of code that makes sure that audio files have been found and prints the pattern that they are globbed with.

[nonstop99]

@ibab

here's the output...

` sudo python3.5 train.py
Found 4706 audio files
Traceback (most recent call last):
File "/home/osboxes/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 965, in _do_call
return fn(*args)
File "/home/osboxes/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 947, in _run_fn
status, run_metadata)
File "/usr/lib/python3.5/contextlib.py", line 66, in exit
next(self.gen)
File "/home/osboxes/.local/lib/python3.5/site-packages/tensorflow/python/framework/errors.py", line 450, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors.OutOfRangeError: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
[[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train.py", line 151, in
main()
File "train.py", line 136, in main
summary, loss_value, _ = sess.run([summaries, loss, optim])
File "/home/osboxes/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 710, in run
run_metadata_ptr)
File "/home/osboxes/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 908, in _run
feed_dict_string, options, run_metadata)
File "/home/osboxes/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 958, in _do_run
target_list, options, run_metadata)
File "/home/osboxes/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 978, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.OutOfRangeError: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
[[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
Caused by op 'create_inputs/padding_fifo_queue_DequeueMany', defined at:
File "train.py", line 151, in
main()
File "train.py", line 90, in main
audio_batch, _ = queue.dequeue_many(args.batch_size)
File "/home/osboxes/.local/lib/python3.5/site-packages/tensorflow/python/ops/data_flow_ops.py", line 435, in dequeue_many
self._queue_ref, n=n, component_types=self._dtypes, name=name)
File "/home/osboxes/.local/lib/python3.5/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 867, in _queue_dequeue_many
timeout_ms=timeout_ms, name=name)
File "/home/osboxes/.local/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 703, in apply_op
op_def=op_def)
File "/home/osboxes/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2317, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/osboxes/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1239, in init
self._traceback = _extract_stack()`

[finlay-liu]

Thx. I fetch the latest code any solve my problem, now i can train the model in tf.
I put the VCTK-Corpus folder in the root of the code, it runs ok. @polyrhythmatic

[polyrhythmatic]

@finlay-liu thanks - already have the files in root and still getting an error.

@ibab I'm on the latest commit and still having issues 3e4da39

running

python3 train.py

I get the following output

I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties: 
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.076
pciBusID 0000:01:00.0
Total memory: 12.00GiB
Free memory: 11.52GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:838] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:01:00.0)
Found 44257 audio files
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 965, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 947, in _run_fn
    status, run_metadata)
  File "/usr/lib/python3.5/contextlib.py", line 66, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors.py", line 450, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors.OutOfRangeError: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train.py", line 151, in <module>
    main()
  File "train.py", line 136, in main
    summary, loss_value, _ = sess.run([summaries, loss, optim])
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 710, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 908, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 958, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 978, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.OutOfRangeError: PaddingFIFOQueue '_0_create_inputs/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
     [[Node: create_inputs/padding_fifo_queue_DequeueMany = QueueDequeueMany[_class=["loc:@create_inputs/padding_fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](create_inputs/padding_fifo_queue, create_inputs/padding_fifo_queue_DequeueMany/n)]]
Caused by op 'create_inputs/padding_fifo_queue_DequeueMany', defined at:
  File "train.py", line 151, in <module>
    main()
  File "train.py", line 90, in main
    audio_batch, _ = queue.dequeue_many(args.batch_size)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/data_flow_ops.py", line 435, in dequeue_many
    self._queue_ref, n=n, component_types=self._dtypes, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 867, in _queue_dequeue_many
    timeout_ms=timeout_ms, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 703, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 2317, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1239, in __init__
    self._traceback = _extract_stack()

Any idea what this could be?

[finlay-liu]

@polyrhythmatic i use py2 it's ok, and i think it maybe the the version problem.

[nonstop99]

@finlay-liu I've tried with py2.7 and py3.5 but i'm getting the same results as @polyrhythmatic

[polyrhythmatic]

@finlay-liu thanks but I have tried python 2.7 as well and it does not work.

[ibab]

It just occurred to me that this could be caused by ffmpeg not being installed.
Could you check whether it's installed and in your PATH?
I'll add a note to the README.md that it is required.

[spiantino]

brew install ffmpeg fixed it for me (on OS X), but now I'm running out of memory. I suppose 6 GB is not enough (I have a 980 Ti)

[ibab]

@spiantino: The dilated convolution uses an excessively large amount of memory at the moment.
It's on my list of things that need to be fixed for the implementation is "complete".

If you still want to test the network, you can reduce the number of layers in the train.py script.

[spiantino]

Had to go all the way down to dilations = [1, 2, 4] but loss seems to be decreasing. Probably not going to learn very much, but we'll see

[ibab]

I've added a memory-related issue here: #4

[ibab]

I think I can close this issue now, as it seems that it was caused by missing the ffmpeg library.
If you are still having problems that match the ones above, feel free to comment here and I will reopen the issue.

[adamduracz]

@ibab training works for me after brew install ffmpeg. Thanks!

[polyrhythmatic]

Thanks - this worked a charm yesterday! Just had to update cudnn to 5 today and now I'm up and training (with python 2.7 as well). I had assumed that tensorflow installed ffmpeg by default. Great work :)