The ctc-connectionist-temporal-classification from mpezeshki

example not running

I'm trying to run your ctc example but i get the following error:

Building model ...
/Users/sorensonderby/Documents/phd/RNN/Theano/theano/scan_module/scan_perform_ext.py:133: RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility
  from scan_perform.scan_perform import *
Bulding DataStream ...
Bulding training process...
INFO:blocks.algorithms:Taking the cost gradient
INFO:blocks.algorithms:The cost gradient computation graph is built
Starting training ...
INFO:blocks.main_loop:Entered the main loop
INFO:blocks.algorithms:Initializing the training algorithm
INFO:blocks.algorithms:The training algorithm is initialized
ERROR:blocks.main_loop:Error occured during training.

Blocks will attempt to run `on_error` extensions, potentially saving data, before exiting and reraising the error. Note that the usual `after_training` extensions will *not* be run. The original error will be re-raised and also stored in the training log. Press CTRL + C to halt Blocks immediately.

-------------------------------------------------------------------------------
BEFORE FIRST EPOCH
-------------------------------------------------------------------------------
Training status:
     batch_interrupt_received: False
     epoch_interrupt_received: False
     epoch_started: True
     epochs_done: 0
     iterations_done: 0
     received_first_batch: False
     training_started: True
Log records from the iteration 0:

Traceback (most recent call last):
  File "/Users/sorensonderby/Documents/phd/RNN/CTC-Connectionist-Temporal-Classification/test_ctc.py", line 122, in <module>
    main_loop.run()
  File "/Users/sorensonderby/Documents/phd/RNN/blocks/blocks/main_loop.py", line 192, in run
    reraise_as(e)
  File "/Users/sorensonderby/Documents/phd/RNN/blocks/blocks/utils/__init__.py", line 225, in reraise_as
    six.reraise(type(new_exc), new_exc, orig_exc_traceback)
  File "/Users/sorensonderby/Documents/phd/RNN/blocks/blocks/main_loop.py", line 178, in run
    while self._run_epoch():
  File "/Users/sorensonderby/Documents/phd/RNN/blocks/blocks/main_loop.py", line 227, in _run_epoch
    while self._run_iteration():
  File "/Users/sorensonderby/Documents/phd/RNN/blocks/blocks/main_loop.py", line 247, in _run_iteration
    self.algorithm.process_batch(batch)
  File "/Users/sorensonderby/Documents/phd/RNN/blocks/blocks/algorithms/__init__.py", line 234, in process_batch
    self._function(*ordered_batch)
  File "/Users/sorensonderby/Documents/phd/RNN/Theano/theano/compile/function_module.py", line 517, in __call__
    allow_downcast=s.allow_downcast)
  File "/Users/sorensonderby/Documents/phd/RNN/Theano/theano/tensor/type.py", line 130, in filter
    raise TypeError(err_msg, data)
TypeError: ('Bad input argument to theano function with name "/Users/sorensonderby/Documents/phd/RNN/blocks/blocks/algorithms/__init__.py:224"  at index 0(0-based), TensorType(float32, matrix) cannot store a value of dtype float64 without risking loss of precision. If you do not mind this loss, you can: 1) explicitly cast your data to float32, or 2) set "allow_input_downcast=True" when calling "function"., [[ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.]\n [ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.]\n [ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.]\n [ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.]\n [ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.]\n [ 0.  0.  0.  0.  1.  1.  1.  1.  1.  1.]]\n\nOriginal exception:\n\tTypeError: Bad input argument to theano function with name "/Users/sorensonderby/Documents/phd/RNN/blocks/blocks/algorithms/__init__.py:224"  at index 0(0-based), TensorType(float32, matrix) cannot store a value of dtype float64 without risking loss of precision. If you do not mind this loss, you can: 1) explicitly cast your data to float32, or 2) set "allow_input_downcast=True" when calling "function"., [[ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.]\n [ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.]\n [ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.]\n [ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.]\n [ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.]\n [ 0.  0.  0.  0.  1.  1.  1.  1.  1.  1.]]', 'TensorType(float32, matrix) cannot store a value of dtype float64 without risking loss of precision. If you do not mind this loss, you can: 1) explicitly cast your data to float32, or 2) set "allow_input_downcast=True" when calling "function".', array([[ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.],
       [ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.],
       [ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.],
       [ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.],
       [ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.],
       [ 0.,  0.,  0.,  0.,  1.,  1.,  1.,  1.,  1.,  1.]]))

Which i think i can workaround by setting allow_input_downcast=True in line 224 in blocks/algorithms/__init__.py

But then i get another error:

Building model ...
/Users/sorensonderby/Documents/phd/RNN/Theano/theano/scan_module/scan_perform_ext.py:133: RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility
  from scan_perform.scan_perform import *
Bulding DataStream ...
Bulding training process...
INFO:blocks.algorithms:Taking the cost gradient
INFO:blocks.algorithms:The cost gradient computation graph is built
INFO:blocks.main_loop:Entered the main loop
INFO:blocks.algorithms:Initializing the training algorithm
Starting training ...

INFO:blocks.algorithms:The training algorithm is initialized
-------------------------------------------------------------------------------
BEFORE FIRST EPOCH
-------------------------------------------------------------------------------
Training status:
     batch_interrupt_received: False
     epoch_interrupt_received: False
     epoch_started: True
     epochs_done: 0
     iterations_done: 0
     received_first_batch: False
     training_started: True
Log records from the iteration 0:

ERROR:blocks.main_loop:Error occured during training.

Blocks will attempt to run `on_error` extensions, potentially saving data, before exiting and reraising the error. Note that the usual `after_training` extensions will *not* be run. The original error will be re-raised and also stored in the training log. Press CTRL + C to halt Blocks immediately.
Traceback (most recent call last):
  File "/Users/sorensonderby/Documents/phd/RNN/CTC-Connectionist-Temporal-Classification/test_ctc.py", line 122, in <module>
    main_loop.run()
  File "/Users/sorensonderby/Documents/phd/RNN/blocks/blocks/main_loop.py", line 192, in run
    reraise_as(e)
  File "/Users/sorensonderby/Documents/phd/RNN/blocks/blocks/utils/__init__.py", line 225, in reraise_as
    six.reraise(type(new_exc), new_exc, orig_exc_traceback)
  File "/Users/sorensonderby/Documents/phd/RNN/blocks/blocks/main_loop.py", line 178, in run
    while self._run_epoch():
  File "/Users/sorensonderby/Documents/phd/RNN/blocks/blocks/main_loop.py", line 227, in _run_epoch
    while self._run_iteration():
  File "/Users/sorensonderby/Documents/phd/RNN/blocks/blocks/main_loop.py", line 247, in _run_iteration
    self.algorithm.process_batch(batch)
  File "/Users/sorensonderby/Documents/phd/RNN/blocks/blocks/algorithms/__init__.py", line 234, in process_batch
    self._function(*ordered_batch)
  File "/Users/sorensonderby/Documents/phd/RNN/Theano/theano/compile/function_module.py", line 610, in __call__
    storage_map=self.fn.storage_map)
  File "/Users/sorensonderby/Documents/phd/RNN/Theano/theano/compile/function_module.py", line 599, in __call__
    outputs = self.fn()
TypeError: expected type_num 7 (NPY_INT64) got 12
Apply node that caused the error: Elemwise{Add}[(0, 1)](Viterbi, shared_Viterbi)
Inputs types: [TensorType(int64, vector), TensorType(int64, vector)]
Inputs shapes: [(0,), (7,)]
Inputs strides: [(8,), (8,)]
Inputs values: [array([], dtype=float64), 'not shown']

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

Original exception:
    TypeError: expected type_num 7 (NPY_INT64) got 12
Apply node that caused the error: Elemwise{Add}[(0, 1)](Viterbi, shared_Viterbi)
Inputs types: [TensorType(int64, vector), TensorType(int64, vector)]
Inputs shapes: [(0,), (7,)]
Inputs strides: [(8,), (8,)]
Inputs values: [array([], dtype=float64), 'not shown']

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

Can you add a few notes explaning what S, T, B, D, L, C and F are?

Maybe you could also explain the input format for apply(cls, y, y_hat, y_mask, y_hat_mask, scale='log_scale') ?

Is it correct that:

y : one-hot encoded labels, LABEL_LENGTH x BATCH_SIZE
y_hat : predictions: INPUT_SEQUENCE_LENGTH x BATCH_SIZE x {num_classes + blank}
y_mask : binary mask? shp? I assume that one is used for included sequences?
y_mask_hat : Used to mask if the input is not INPUT_SEQUENCE_LENGTH?

Where INPUT_SEQUENCE_LENGTH is the length of the input sequences (30 for the example data) and LABEL_LENGTH is the label sequence for each target. Is LABEL_LENGTH padded if the true label length vary?

-Søren

p(blank symbol) >> p(non-blank symbol) during NN-CTC training

Hi all

I want to discuss some issue regarding training DNN/CNN-CTC for speech recognition. (Wall Street Journal Corpus). I modeled output unit as characters.

I observed that CTC objective function was increasing and finally converged during training.

But I also observed that final NN outputs have clear tendency : p(blank symbol) >> p(non-blank symbol) for all speech time frame as following figure

In Alex Graves' paper, trained RNN should have high p(non-blank) at some point like following figure

Do you have same situation when you train NN-CTC for sequence labeling problem? I am suspecting that the reason is I use MLP/CNN instead of RNN, but I can't clearly explain why this can be a reason.
Any idea about this result?

Thank you for reading my question.

python test_ctc.py failed!

ubgpu@ubgpu:/github/CTC-Connectionist-Temporal-Classification$
ubgpu@ubgpu:/github/CTC-Connectionist-Temporal-Classification$ sudo python test_ctc.py
[sudo] password for ubgpu:
Using gpu device 0: GeForce GTX 970
Building model ...
/usr/local/lib/python2.7/dist-packages/theano/scan_module/scan_perform_ext.py:133: RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility
from scan_perform.scan_perform import *
Bulding DataStream ...
Bulding training process...
INFO:blocks.algorithms:Taking the cost gradient
INFO:blocks.algorithms:The cost gradient computation graph is built
Starting training ...
INFO:blocks.main_loop:Entered the main loop
INFO:blocks.algorithms:Initializing the training algorithm
ERROR (theano.gof.opt): Optimization failure due to: local_gpu_lazy_ifelse
ERROR:theano.gof.opt:Optimization failure due to: local_gpu_lazy_ifelse
ERROR (theano.gof.opt): TRACEBACK:
ERROR:theano.gof.opt:TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 1493, in process_node
replacements = lopt.transform(node)
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/opt.py", line 460, in local_gpu_lazy_ifelse
outs[i] = gpu_from_host(outs[i])
File "/usr/local/lib/python2.7/dist-packages/theano/gof/op.py", line 507, in call
node = self.make_node(_inputs, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/basic_ops.py", line 133, in make_node
dtype=x.dtype)()])
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/type.py", line 69, in init
(self.class.name, dtype, name))
TypeError: CudaNdarrayType only supports dtype float32 for now. Tried using dtype int64 for variable None

ERROR:theano.gof.opt:Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 1493, in process_node
replacements = lopt.transform(node)
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/opt.py", line 460, in local_gpu_lazy_ifelse
outs[i] = gpu_from_host(outs[i])
File "/usr/local/lib/python2.7/dist-packages/theano/gof/op.py", line 507, in call
node = self.make_node(_inputs, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/basic_ops.py", line 133, in make_node
dtype=x.dtype)()])
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/type.py", line 69, in init
(self.class.name, dtype, name))
TypeError: CudaNdarrayType only supports dtype float32 for now. Tried using dtype int64 for variable None

ERROR (theano.gof.opt): Optimization failure due to: local_gpu_lazy_ifelse
ERROR:theano.gof.opt:Optimization failure due to: local_gpu_lazy_ifelse
ERROR (theano.gof.opt): TRACEBACK:
ERROR:theano.gof.opt:TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 1493, in process_node
replacements = lopt.transform(node)
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/opt.py", line 460, in local_gpu_lazy_ifelse
outs[i] = gpu_from_host(outs[i])
File "/usr/local/lib/python2.7/dist-packages/theano/gof/op.py", line 507, in call
node = self.make_node(_inputs, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/basic_ops.py", line 133, in make_node
dtype=x.dtype)()])
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/type.py", line 69, in init
(self.class.name, dtype, name))
TypeError: CudaNdarrayType only supports dtype float32 for now. Tried using dtype int32 for variable None

ERROR:theano.gof.opt:Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 1493, in process_node
replacements = lopt.transform(node)
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/opt.py", line 460, in local_gpu_lazy_ifelse
outs[i] = gpu_from_host(outs[i])
File "/usr/local/lib/python2.7/dist-packages/theano/gof/op.py", line 507, in call
node = self.make_node(_inputs, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/basic_ops.py", line 133, in make_node
dtype=x.dtype)()])
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/type.py", line 69, in init
(self.class.name, dtype, name))
TypeError: CudaNdarrayType only supports dtype float32 for now. Tried using dtype int32 for variable None

<<!! BUG IN FGRAPH.REPLACE OR A LISTENER !!>> <type 'exceptions.TypeError'> ('The type of the replacement must be compatible with the type of the original Variable.', Sum{axis=[0], acc_dtype=float64}.0, HostFromGpu.0, TensorType(int32, vector), TensorType(float32, vector), 'local_gpu_careduce') local_gpu_careduce
ERROR (theano.gof.opt): Optimization failure due to: local_gpu_careduce
ERROR:theano.gof.opt:Optimization failure due to: local_gpu_careduce
ERROR (theano.gof.opt): TRACEBACK:
ERROR:theano.gof.opt:TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 1527, in process_node
fgraph.replace_all_validate(repl_pairs, reason=lopt)
File "/usr/local/lib/python2.7/dist-packages/theano/gof/toolbox.py", line 259, in replace_all_validate
fgraph.replace(r, new_r, reason=reason, verbose=False)
File "/usr/local/lib/python2.7/dist-packages/theano/gof/fg.py", line 474, in replace
str(reason))
TypeError: ('The type of the replacement must be compatible with the type of the original Variable.', Sum{axis=[0], acc_dtype=float64}.0, HostFromGpu.0, TensorType(int32, vector), TensorType(float32, vector), 'local_gpu_careduce')

ERROR:theano.gof.opt:Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 1527, in process_node
fgraph.replace_all_validate(repl_pairs, reason=lopt)
File "/usr/local/lib/python2.7/dist-packages/theano/gof/toolbox.py", line 259, in replace_all_validate
fgraph.replace(r, new_r, reason=reason, verbose=False)
File "/usr/local/lib/python2.7/dist-packages/theano/gof/fg.py", line 474, in replace
str(reason))
TypeError: ('The type of the replacement must be compatible with the type of the original Variable.', Sum{axis=[0], acc_dtype=float64}.0, HostFromGpu.0, TensorType(int32, vector), TensorType(float32, vector), 'local_gpu_careduce')

ERROR (theano.gof.opt): Optimization failure due to: local_gpu_lazy_ifelse
ERROR:theano.gof.opt:Optimization failure due to: local_gpu_lazy_ifelse
ERROR (theano.gof.opt): TRACEBACK:
ERROR:theano.gof.opt:TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 1493, in process_node
replacements = lopt.transform(node)
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/opt.py", line 460, in local_gpu_lazy_ifelse
outs[i] = gpu_from_host(outs[i])
File "/usr/local/lib/python2.7/dist-packages/theano/gof/op.py", line 507, in call
node = self.make_node(_inputs, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/basic_ops.py", line 133, in make_node
dtype=x.dtype)()])
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/type.py", line 69, in init
(self.class.name, dtype, name))
TypeError: CudaNdarrayType only supports dtype float32 for now. Tried using dtype int64 for variable None