solivr / tf-crnn

TensorFlow convolutional recurrent neural network (CRNN) for text recognition

Text recognition with Convolutional Recurrent Neural Network and TensorFlow 2.0 (tf2-crnn)

Implementation of a Convolutional Recurrent Neural Network (CRNN) for image-based sequence recognition tasks, such as scene text recognition and OCR.

This implementation is based on Tensorflow 2.0 and uses tf.keras and tf.data modules to build the model and to handle input data.

To access the previous version implementing Shi et al. paper, go to the v.0.5.2 tag.

Installation

tf_crnn makes use of tensorflow-gpu package (so CUDA and cuDNN are needed).

You can install it using the environment.yml file provided and use it within an environment.

conda env create -f environment.yml

See also the docs for more information.

Try it

Train a model with IAM dataset.

Create an account

Create an account on the official IAM dataset page in order to access the data. Export your credentials as enviornment variables, they will be used by the download script.

export IAM_USER=<your-username>
export IAM_PWD=<your-password>

Generate the data in the correct format

cd hlp
python prepare_iam.py --download_dir ../data/iam --generated_data_dir ../data/iam/generated
cd ..

Train the model

python training.py with config.json

More details in the documentation.

tf-crnn's Issues

Error while attempting to train a new model

I met a problem when I attempted to train a new model. I have no idea where the error comes from. Here is the error message. Thanks!

Traceback (most recent call last):
File "train.py", line 97, in
image_summaries=True))
File "/usr/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 302, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/usr/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 783, in _train_model
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 537, in exit
self._close_internal(exception_type)
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 574, in _close_internal
self._sess.close()
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 820, in close
self._sess.close()
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 941, in close
ignore_live_threads=True)
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/usr/lib/python3.6/site-packages/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/usr/lib/python3.6/site-packages/tensorflow/python/training/queue_runner_impl.py", line 238, in _run
enqueue_callable()
File "/usr/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1231, in _single_operation_run
target_list_as_strings, status, None)
File "/usr/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: paddings must be less than the dimension size: 4, 2 not less than 2
[[Node: DataAugmentation/random_padding = MirrorPad[T=DT_UINT8, Tpaddings=DT_INT32, mode="REFLECT", _device="/job:localhost/replica:0/task:0/device:CPU:0"](image_decoding/Merge, DataAugmentation/random_padding/paddings)]]

The speed between CPU and GPU

when I predict image with the output model, I find CPU is faster than GPU
the code is

import tensorflow as tf
from src.loader import PredictionModel
from scipy.misc import imread
import time
import os

def predict(model_dir, image,gpu_id = 0):
    os.environ['CUDA_VISIBLE_DEVICES'] = str(gpu_id)
    config_sess = tf.ConfigProto()
    config_sess.gpu_options.per_process_gpu_memory_fraction = 0.2

    with tf.Session(config=config_sess) as sess:
        start = time.time()
        model = PredictionModel(model_dir,session=sess)
        predictions = model.predict(image)
        transcription = predictions['words']
        score = predictions['score']
        return [transcription[0].decode(), score, time.time() - start]


if __name__ == '__main__':
    model_dir = 'ckpt2model/export/1511402179/'
    image = imread('/data/zhoujun/data/print_char/val1/3_song.jpg', mode='L')[:, :, None]
    result = predict(model_dir, image,0)
    print(result)

and gpu cost 3.9， cpu cost 0.9s

Very slow when running estimator.evaluate, gpu usage is always zero

The training model took 27 hours (one million data) and one model required 4 hours (100,000 data) when testing the model

prediction with estimator

I encountered some problems when predicting using estimator with following code:
`
import argparse
import os, time
import csv
import numpy as np
try:
import better_exceptions
except ImportError:
pass
from tqdm import trange
import tensorflow as tf
from src.model import crnn_fn
from src.data_handler import data_loader
from src.data_handler import preprocess_image_for_prediction

from src.config import Params, Alphabet, import_params_from_json

parameters = Params(train_batch_size=100,
eval_batch_size=100,
learning_rate=1e-3, # 1e-3 recommended
learning_decay_rate=0.95,
learning_decay_steps=5000,
evaluate_every_epoch=5,
save_interval=5e3,
input_shape=(32, 304),
optimizer='adam',
digits_only=True,
alphabet=Alphabet.LETTERS_DIGITS_EXTENDED,
alphabet_decoding='same',
csv_delimiter=';',
csv_files_eval='./output_numbers/testlabels_abs.csv',
csv_files_train='./output_numbers/trainlabels_abs.csv',
output_model_dir='./estimator/',
n_epochs=1,
gpu=''
)
model_params = {
'Params': parameters,
}

parameters.export_experiment_params()

os.environ['CUDA_VISIBLE_DEVICES'] = parameters.gpu
config_sess = tf.ConfigProto()
config_sess.gpu_options.per_process_gpu_memory_fraction = 0.8
config_sess.gpu_options.allow_growth = True

est_config = tf.estimator.RunConfig()
est_config.replace(keep_checkpoint_max=10,
save_checkpoints_steps=parameters.save_interval,
session_config=config_sess,
save_checkpoints_secs=None,
save_summary_steps=1000,
model_dir=parameters.output_model_dir)

estimator = tf.estimator.Estimator(model_fn=crnn_fn,
params=model_params,
model_dir=parameters.output_model_dir,
config=est_config
)

pred=estimator.predict(input_fn=data_loader(csv_filename='./output_numbers/testlabels_abs.csv',params=parameters))

for i in enumerate(pred):
print(i)
`
The error is like:

(tf36) C:\Users\lance\Documents\Github\tf-crnn>python predict.py
Traceback (most recent call last):
File "predict.py", line 91, in
for i,j in enumerate(a):
File "C:\Users\lance\Miniconda3\envs\tf36\lib\site-packages\tensorflow\python\estimator\estimator.py", line 425, in predict
for i in range(self._extract_batch_length(preds_evaluated)):
File "C:\Users\lance\Miniconda3\envs\tf36\lib\site-packages\tensorflow\python\estimator\estimator.py", line 592, in _extract_batch_length
'different batch length then others.' % key)
ValueError: Batch length of predictions should be same. raw_predictions has different batch length then others.

Wonder if anyone could help with it. Thanks.

working with numbers only

Hi @solivr

i would like to recognize numbers only , do i have to change in the parameters

parameters = Params(train_batch_size=128,
                            eval_batch_size=128,
                            learning_rate=1e-3,  # 1e-3 recommended
                            learning_decay_rate=0.95,
                            learning_decay_steps=5000,
                            evaluate_every_epoch=5,
                            save_interval=5e3,
                            input_shape=(32, 304),
                            optimizer='adam',
                            digits_only=False,
                            alphabet=Alphabet.LETTERS_DIGITS_EXTENDED,
                            alphabet_decoding='same',
                            csv_delimiter=';',
                            csv_files_eval=args.get('csv_files_eval'),
                            csv_files_train=args.get('csv_files_train'),
                            output_model_dir=args.get('output_model_dir'),
                            n_epochs=args.get('nb_epochs'),
                            gpu=args.get('gpu')
                            )  

is it mandatory to change digits_only to True

'Params' object has no attribute 'get'

After trained model. I run python export_model.py and got error

/home/zero/tensorflow/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Traceback (most recent call last):
  File "export_model.py", line 45, in <module>
    serving_input_receiver_fn=preprocess_image_for_prediction(min_width=10))
  File "/home/zero/tensorflow/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 436, in export_savedmodel
    mode=model_fn_lib.ModeKeys.PREDICT)
  File "/home/zero/tensorflow/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 615, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "/home/zero/tf-crnn/src/model.py", line 382, in crnn_fn
    parameters = params.get('Params')
AttributeError: 'Params' object has no attribute 'get'

I run in Ubuntu 16 with cpu and tensorflow 1.3.
Please let me know if anyone has solved this.

Load Saved Model to predict only one image once cause error

below is my code
`
#!/usr/bin/env python

if name == "main":
with tf.Session(graph=tf.Graph()) as sess:
print("loading model from model dir ...")
model = PredictionModel("model0/export/1523419345/", sess)
im = Image.open("oo-1.png").convert('L')
im = numpy.array(im.getdata(),numpy.uint8).reshape(im.size[0], im.size[1], 1)
print(im.shape)
ret = model.predict(im)
print ("predict result : {}" % (ret))

`
error
2018-04-11 04:13:01.785282: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Less leaves in the beam search than requested.
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1327, in _do_call
return fn(*args)
File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1306, in _run_fn
status, run_metadata)
File "/opt/conda/lib/python3.6/contextlib.py", line 88, in exit
next(self.gen)
File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Less leaves in the beam search than requested.
[[Node: code2str_conversion/CTCBeamSearchDecoder = CTCBeamSearchDecoder[beam_width=100, merge_repeated=false, top_paths=2, _device="/job:localhost/replica:0/task:0/cpu:0"](deep_bidirectional_lstm/transpose_time_major, code2str_conversion/Cast_1)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "predict.py", line 24, in
ret = model.predict(im)
File "/home/jovyan/work/tf-crnn/src/loader.py", line 22, in predict
return self.session.run(output, feed_dict={self._input_dict['images']: image})
File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 895, in run
run_metadata_ptr)
File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1124, in _run
feed_dict_tensor, options, run_metadata)
File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run
options, run_metadata)
File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Less leaves in the beam search than requested.
[[Node: code2str_conversion/CTCBeamSearchDecoder = CTCBeamSearchDecoder[beam_width=100, merge_repeated=false, top_paths=2, _device="/job:localhost/replica:0/task:0/cpu:0"](deep_bidirectional_lstm/transpose_time_major, code2str_conversion/Cast_1)]]

Caused by op 'code2str_conversion/CTCBeamSearchDecoder', defined at:
File "predict.py", line 18, in
model = PredictionModel("model0/export/1523419345/", sess)
File "/home/jovyan/work/tf-crnn/src/loader.py", line 14, in init
self.model = tf.saved_model.loader.load(self.session, ['serve'], model_dir)
File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/saved_model/loader_impl.py", line 216, in load
saver = tf_saver.import_meta_graph(meta_graph_def_to_load, **saver_kwargs)
File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1698, in import_meta_graph
**kwargs)
File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/meta_graph.py", line 656, in import_scoped_meta_graph
producer_op_list=producer_op_list)
File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 313, in import_graph_def
op_def=op_def)
File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1204, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Less leaves in the beam search than requested.
[[Node: code2str_conversion/CTCBeamSearchDecoder = CTCBeamSearchDecoder[beam_width=100, merge_repeated=false, top_paths=2, _device="/job:localhost/replica:0/task:0/cpu:0"](deep_bidirectional_lstm/transpose_time_major, code2str_conversion/Cast_1)]]

How to apply the code to Chinese

if I want to apply the code to Chinese, how to modify the python script?

there are errors about the function definition：import_params_from_json

def import_params_from_json(model_directory: str=None, json_filename: str=None) -> dict:
i must modify it:
def import_params_from_json(model_directory, json_filename):
how can i define the function like the: def import_params_from_json(model_directory: str=None, json_filename: str=None) -> dict:

train and print accuracy error

I use a small dataset(train:1000,test:1000) to test the code, and I meet a error when train the mode at the last batch.the batchsize of train and evaluation is 128

W tensorflow/core/framework/op_kernel.cc:1192] Out of range: RandomShuffleQueue '_3_prepared_batch_queue/random_shuffle_queue' is closed and has insufficient elements (requested 128, current size 8)
         [[Node: prepared_batch_queue = QueueDequeueManyV2[component_types=[DT_STRING, DT_FLOAT, DT_INT32, DT_STRING], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](prepared_batch_queue/random_shuffle_queue, prepared_batch_queue/n)]]

and there is no any accuracy information output

if mode == tf.estimator.ModeKeys.EVAL:
        print('start to evaluation')
        with tf.name_scope('evaluation'):
            CER = tf.metrics.mean(tf.edit_distance(sparse_code_pred, tf.cast(sparse_code_target, dtype=tf.int64)), name='CER')

            # Convert label codes to decoding alphabet to compare predicted and groundtrouth words
            target_chars = table_int2str.lookup(tf.cast(sparse_code_target, tf.int64))
            target_words = get_words_from_chars(target_chars.values, seq_lengths_labels)
            accuracy = tf.metrics.accuracy(target_words, predictions_dict['words'], name='accuracy')

            eval_metric_ops = {
                               'eval/accuracy': accuracy,
                               'eval/CER': CER,
                               }
        CER = tf.Print(CER, [CER], message='-- CER : ')
        accuracy = tf.Print(accuracy, [accuracy], message='-- Accuracy : ')
        tf.summary.scalar('accuracy/val_accuracy', accuracy)
        print('end to evaluation')

error in tf1.3

when I run this code in the ubuntu16.04 with tensorflow 1.3, I meet the flow error, the dataset I used is mnist.
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1327, in _do_call
return fn(*args)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1306, in _run_fn
status, run_metadata)
File "/usr/lib/python3.5/contextlib.py", line 66, in exit
next(self.gen)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.OutOfRangeError: RandomShuffleQueue '_3_prepared_batch_queue/random_shuffle_queue' is closed and has insufficient elements (requested 128, current size 0)
[[Node: prepared_batch_queue = QueueDequeueManyV2[component_types=[DT_STRING, DT_FLOAT, DT_INT32, DT_STRING], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](prepared_batch_queue/random_shuffle_queue, prepared_batch_queue/n)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 686, in _train_model
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 518, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 862, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 818, in run
return self._sess.run(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 972, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 818, in run
return self._sess.run(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 895, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1124, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1321, in _do_run
options, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1340, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: RandomShuffleQueue '_3_prepared_batch_queue/random_shuffle_queue' is closed and has insufficient elements (requested 128, current size 0)
[[Node: prepared_batch_queue = QueueDequeueManyV2[component_types=[DT_STRING, DT_FLOAT, DT_INT32, DT_STRING], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](prepared_batch_queue/random_shuffle_queue, prepared_batch_queue/n)]]

Caused by op 'prepared_batch_queue', defined at:
File "train.py", line 97, in
image_summaries=True))
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 241, in train
loss = self._train_model(input_fn=input_fn, hooks=hooks)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 628, in _train_model
input_fn, model_fn_lib.ModeKeys.TRAIN)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 499, in _get_features_and_labels_from_input_fn
result = self._call_input_fn(input_fn, mode)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 585, in _call_input_fn
return input_fn(**kwargs)
File "/data1/zj/tf-crnn/src/data_handler.py", line 37, in input_fn
name='prepared_batch_queue')
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/input.py", line 1220, in shuffle_batch
name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/input.py", line 791, in _shuffle_batch
dequeued = queue.dequeue_many(batch_size, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/data_flow_ops.py", line 457, in dequeue_many
self._queue_ref, n=n, component_types=self._dtypes, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 1342, in _queue_dequeue_many_v2
timeout_ms=timeout_ms, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1204, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

OutOfRangeError (see above for traceback): RandomShuffleQueue '_3_prepared_batch_queue/random_shuffle_queue' is closed and has insufficient elements (requested 128, current size 0)
[[Node: prepared_batch_queue = QueueDequeueManyV2[component_types=[DT_STRING, DT_FLOAT, DT_INT32, DT_STRING], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](prepared_batch_queue/random_shuffle_queue, prepared_batch_queue/n)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train.py", line 97, in
image_summaries=True))
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 241, in train
loss = self._train_model(input_fn=input_fn, hooks=hooks)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 686, in _train_model
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 534, in exit
self._close_internal(exception_type)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 569, in _close_internal
self._sess.close()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 811, in close
self._sess.close()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 908, in close
ignore_live_threads=True)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
raise value
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/queue_runner_impl.py", line 238, in _run
enqueue_callable()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1235, in _single_operation_run
target_list_as_strings, status, None)
File "/usr/lib/python3.5/contextlib.py", line 66, in exit
next(self.gen)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Expect 2 fields but have 1 in record 0
[[Node: csv_reading_op = DecodeCSV[OUT_TYPE=[DT_STRING, DT_STRING], field_delim=";", use_quote_delim=true, _device="/job:localhost/replica:0/task:0/cpu:0"](file_reading_op:1, csv_reading_op/record_defaults_0, csv_reading_op/record_defaults_1)]]

and when I change the csv_delimiter in train.py from ';' to ' ' , the error become that:

Invalid argument: All labels must be nonnegative integers, batch: 0 labels: -1,-1,-1,-1,-1,-1,-1
2017-10-16 15:02:48.615988: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: All labels must be nonnegative integers, batch: 0 labels: -1,-1,-1,-1,-1,-1,-1
[[Node: CTCLoss = CTCLoss[ctc_merge_repeated=true, ignore_longer_outputs_than_inputs=true, preprocess_collapse_repeated=false, _device="/job:localhost/replica:0/task:0/cpu:0"](deep_bidirectional_lstm/transpose_time_major/_713, str2code_conversion/StringSplit, str2code_conversion/hash_table_Lookup, Cast_3/_791)]]
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1327, in _do_call
return fn(*args)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1306, in _run_fn
status, run_metadata)
File "/usr/lib/python3.5/contextlib.py", line 66, in exit
next(self.gen)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: All labels must be nonnegative integers, batch: 0 labels: -1,-1,-1,-1,-1,-1,-1
[[Node: CTCLoss = CTCLoss[ctc_merge_repeated=true, ignore_longer_outputs_than_inputs=true, preprocess_collapse_repeated=false, _device="/job:localhost/replica:0/task:0/cpu:0"](deep_bidirectional_lstm/transpose_time_major/_713, str2code_conversion/StringSplit, str2code_conversion/hash_table_Lookup, Cast_3/_791)]]
[[Node: code2str_conversion/chars_conversion/Shape/_733 = _HostRecvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_1842_code2str_conversion/chars_conversion/Shape", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train.py", line 97, in
image_summaries=True))
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 241, in train
loss = self._train_model(input_fn=input_fn, hooks=hooks)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 686, in _train_model
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 518, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 862, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 818, in run
return self._sess.run(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 972, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 818, in run
return self._sess.run(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 895, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1124, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1321, in _do_run
options, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1340, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: All labels must be nonnegative integers, batch: 0 labels: -1,-1,-1,-1,-1,-1,-1
[[Node: CTCLoss = CTCLoss[ctc_merge_repeated=true, ignore_longer_outputs_than_inputs=true, preprocess_collapse_repeated=false, _device="/job:localhost/replica:0/task:0/cpu:0"](deep_bidirectional_lstm/transpose_time_major/_713, str2code_conversion/StringSplit, str2code_conversion/hash_table_Lookup, Cast_3/_791)]]
[[Node: code2str_conversion/chars_conversion/Shape/_733 = _HostRecvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_1842_code2str_conversion/chars_conversion/Shape", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"]]

Caused by op 'CTCLoss', defined at:
File "train.py", line 97, in
image_summaries=True))
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 241, in train
loss = self._train_model(input_fn=input_fn, hooks=hooks)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 630, in _train_model
model_fn_lib.ModeKeys.TRAIN)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 615, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "/data1/zj/tf-crnn/src/model.py", line 272, in crnn_fn
time_major=True)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/ctc_ops.py", line 152, in ctc_loss
ignore_longer_outputs_than_inputs=ignore_longer_outputs_than_inputs)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_ctc_ops.py", line 168, in _ctc_loss
name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1204, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): All labels must be nonnegative integers, batch: 0 labels: -1,-1,-1,-1,-1,-1,-1
[[Node: CTCLoss = CTCLoss[ctc_merge_repeated=true, ignore_longer_outputs_than_inputs=true, preprocess_collapse_repeated=false, _device="/job:localhost/replica:0/task:0/cpu:0"](deep_bidirectional_lstm/transpose_time_major/_713, str2code_conversion/StringSplit, str2code_conversion/hash_table_Lookup, Cast_3/_791)]]
[[Node: code2str_conversion/chars_conversion/Shape/_733 = _HostRecvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_1842_code2str_conversion/chars_conversion/Shape", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"]]

OutOfRange Problem

Hi,

mcq-2@mcq-2-linux:~/projects/tf-crnn$ python3 train.py -g 0 -ft /home/mcq-2/Documents/Dataset/DataSet_Text/Train.csv -fe /home/mcq-2/Documents/Dataset/DataSet_Text/Test.csv -o /home/mcq-2/Documents/Dataset/DataSet_Text/
/usr/local/lib/python3.5/dist-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
0%| | 0/6 [00:00<?, ?it/s]2018-02-27 16:00:07.305215: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-02-27 16:00:07.419715: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-02-27 16:00:07.419970: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:01:00.0
totalMemory: 10.91GiB freeMemory: 10.29GiB
2018-02-27 16:00:07.419982: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2018-02-27 16:00:09.022837: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: RandomShuffleQueue '_3_prepared_batch_queue/random_shuffle_queue' is closed and has insufficient elements (requested 128, current size 0)
[[Node: prepared_batch_queue = QueueDequeueManyV2[component_types=[DT_STRING, DT_FLOAT, DT_INT32, DT_STRING], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](prepared_batch_queue/random_shuffle_queue, prepared_batch_queue/n)]]
2018-02-27 16:00:09.022902: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: RandomShuffleQueue '_3_prepared_batch_queue/random_shuffle_queue' is closed and has insufficient elements (requested 128, current size 0)
[[Node: prepared_batch_queue = QueueDequeueManyV2[component_types=[DT_STRING, DT_FLOAT, DT_INT32, DT_STRING], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](prepared_batch_queue/random_shuffle_queue, prepared_batch_queue/n)]]
2018-02-27 16:00:09.023102: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: RandomShuffleQueue '_3_prepared_batch_queue/random_shuffle_queue' is closed and has insufficient elements (requested 128, current size 0)
[[Node: prepared_batch_queue = QueueDequeueManyV2[component_types=[DT_STRING, DT_FLOAT, DT_INT32, DT_STRING], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](prepared_batch_queue/random_shuffle_queue, prepared_batch_queue/n)]]

Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 783, in _train_model
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 521, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 892, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 965, in run
raise six.reraise(*original_exc_info)
File "/usr/local/lib/python3.5/dist-packages/six.py", line 693, in reraise
raise value
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 952, in run
return self._sess.run(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 1024, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 827, in run
return self._sess.run(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 889, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1120, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1317, in _do_run
options, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1336, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: RandomShuffleQueue '_3_prepared_batch_queue/random_shuffle_queue' is closed and has insufficient elements (requested 128, current size 0)
[[Node: prepared_batch_queue = QueueDequeueManyV2[component_types=[DT_STRING, DT_FLOAT, DT_INT32, DT_STRING], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](prepared_batch_queue/random_shuffle_queue, prepared_batch_queue/n)]]

Caused by op 'prepared_batch_queue', defined at:
File "train.py", line 97, in
image_summaries=False))
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 302, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 708, in _train_model
input_fn, model_fn_lib.ModeKeys.TRAIN)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 577, in _get_features_and_labels_from_input_fn
result = self._call_input_fn(input_fn, mode)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 663, in _call_input_fn
return input_fn(**kwargs)
File "/home/mcq-2/projects/tf-crnn/src/data_handler.py", line 37, in input_fn
name='prepared_batch_queue')
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/input.py", line 1225, in shuffle_batch
name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/input.py", line 796, in _shuffle_batch
dequeued = queue.dequeue_many(batch_size, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/data_flow_ops.py", line 464, in dequeue_many
self._queue_ref, n=n, component_types=self._dtypes, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 2418, in _queue_dequeue_many_v2
component_types=component_types, timeout_ms=timeout_ms, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1470, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

OutOfRangeError (see above for traceback): RandomShuffleQueue '_3_prepared_batch_queue/random_shuffle_queue' is closed and has insufficient elements (requested 128, current size 0)
[[Node: prepared_batch_queue = QueueDequeueManyV2[component_types=[DT_STRING, DT_FLOAT, DT_INT32, DT_STRING], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](prepared_batch_queue/random_shuffle_queue, prepared_batch_queue/n)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train.py", line 97, in
image_summaries=False))
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 302, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 783, in _train_model
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 537, in exit
self._close_internal(exception_type)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 574, in _close_internal
self._sess.close()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 820, in close
self._sess.close()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 941, in close
ignore_live_threads=True)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/usr/local/lib/python3.5/dist-packages/six.py", line 692, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/queue_runner_impl.py", line 238, in _run
enqueue_callable()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1231, in _single_operation_run
target_list_as_strings, status, None)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 473, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Expect 2 fields but have 1 in record 0
[[Node: csv_reading_op = DecodeCSV[OUT_TYPE=[DT_STRING, DT_STRING], field_delim=";", na_value="", use_quote_delim=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](file_reading_op:1, csv_reading_op/record_defaults_0, csv_reading_op/record_defaults_0)]]

can you help me please ?

Thank you

How can I use this model to train on other language such as Chinese?

Serving tf_serving

I had several problems at serving one trainned model with tensorflow_serving docker, I run into this error:
"Generic conv implementation does not support grouped convolutions for now.\\n\\t [[{{node deep_cnn/layer1/conv}}]]"

Do you have any tips about how to de-group this "grouped convolution" at deep_cnn or how to serve the model in a REST API?

how to print accuracy after print Loss?

I want print accuracy after 1 epoch. How to print accuracy after print Loss?
Tks

gpu memory

when I set the code like this

 os.environ['CUDA_VISIBLE_DEVICES'] = parameters.gpu
 config_sess = tf.ConfigProto()
 config_sess.gpu_options.per_process_gpu_memory_fraction = 0.6
 config_sess.gpu_options.allow_growth = True

or

 os.environ['CUDA_VISIBLE_DEVICES'] = parameters.gpu
 config_sess = tf.ConfigProto()
 config_sess.gpu_options.per_process_gpu_memory_fraction = 0.6
# config_sess.gpu_options.allow_growth = True

the tensorflow will occupied all gpu memory

Error while serving

Hi,

Thanks for this wonderful repository.
I trained the model on a custom dataset. I now have checkpoints stored. I get the following error while trying to load the model. Any idea what I might be doing wrong here?

OSError: Cannot parse file b'./out_dir/saved_model.pbtxt': 1:1 : Message type "tensorflow.SavedModel" has no field named "node"..

I can confirm that the files exist in the directory.

ctc_loss_calculator.cc no valid path found

When I train my own net, I got the following error:
2019-06-08 11:33:48.746469: W tensorflow/core/util/ctc/ctc_loss_calculator.cc:144] No valid path found.
Is it because that the label should be shorter than input length referring to below?
https://stackoverflow.com/questions/45130184/ctc-loss-error-no-valid-path-found
If so, how can I modify train.py to solve this problem?
Thanks in advance~

How to test and print the prediction results of my own trained model?

I trained my own model using my own images with their ground truth values and I saw evaluation accuracy at the end of training but I couldn't test different image to see the predicted values for my trained model

Error in generating sequential MNIST digits

I noticed an error in generation of MNIST digit sequences - if the first element of the label sequence is a zero, then the label does not take this into account, but the zero is included in the digit string.

e.g. if the label sequence is 02139, the digits '02139' will be saved as a image sequence but the label will be 2139, this also works for multiple zeros, e.g. 002139 is labelled as 2139 as well.

I couldn't find a way to fix this so I just checked if the first element of the label sequence was a zero. If so, continue on in the for loop.

Why do the new codes run so slowly?

I have trained the model on a data set containing 200, 000 images using codes of 0.5 version.
But the training speed is really slow ...
It have 0.07 global_step/s and elapsed 4 h to achieve 1k steps.
Meanwhile, the old version (0.3.0) codes, running on the same data set, have 1.6 global_step/s and only consumed 30 min achieving 1k steps.
The hyper-parameters are the same except batch size (128 for old one, 256 for the new one).
I can hardly figure out why.
Could anyone show me the cause?

Small script for converting ocropus data

Hi @solivr !
The idea of evaluating your tool is really interesting to us. We have worked with Ocropus with medieval litterary manuscripts ( https://graal.hypotheses.org/786 ) and I am really looking forward comparing both results with handscript that are close to typescript.
If you are in anyway interested, I'll post the small script I put together for converting our data for your input format ( https://github.com/PonteIneptique/ocropus-to-tf-crnn ) and keep you uptdated with the statistics :)

Get prediction results of all images with estimator

I managed to use tf.estimator.predict to get the prediction results of a csv file. The size of the images in the csv is not the multiple of batch_size.However, I found that the number of prediction results will always be the multiple of batch_size, which indicates that last batch of images of which the size less than batch size will not be preceeded by tf.estimator.predict and no results will be given for them.

Anyone has solution to it? Thanks in advance!

when I use tf 1.8 to train the code,the gpu util is 0 always

about mkdir for ".../export/..."

Hi @solivr :
it happens an error "can not make directory './output/export'sequencenumber''",then I interrupted this code and restarted it , it was running good,but couldn't save the model to ".../export/..." directory.What might be the problem? thanks.
I run code at windows10, using CPU not GPU.

sacred error

ERROR - CRNN_experiment - Failed after 0:00:00!
Traceback (most recent calls WITHOUT Sacred internals):
File "train.py", line 111, in run
config=est_config
File "/home/ramu_yarru/venv/lib/python3.5/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 210, in init
self._params = copy.deepcopy(params or {})
File "/home/ramu_yarru/venv/lib/python3.5/copy.py", line 155, in deepcopy
y = copier(x, memo)
File "/home/ramu_yarru/venv/lib/python3.5/copy.py", line 243, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/home/ramu_yarru/venv/lib/python3.5/copy.py", line 182, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/home/ramu_yarru/venv/lib/python3.5/copy.py", line 297, in _reconstruct
state = deepcopy(state, memo)
File "/home/ramu_yarru/venv/lib/python3.5/copy.py", line 155, in deepcopy
y = copier(x, memo)
File "/home/ramu_yarru/venv/lib/python3.5/copy.py", line 243, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/home/ramu_yarru/venv/lib/python3.5/copy.py", line 182, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/home/ramu_yarru/venv/lib/python3.5/copy.py", line 315, in _reconstruct
y.append(item)
sacred.utils.SacredError: The configuration is read-only in a captured function!

hi、is python2.7 or python 3？

pretrained model

Hi @solivr,

Is there pretrained model to test it?
Thanks

error when I run the code with Symbols

the Symbols is

Symbols = "'.,:;-_=()[]{}/°"
BLANK_SYMBOL = '$'

when I run the code I meet the error

Traceback (most recent call last):
  File "train.py", line 99, in <module>
    image_summaries=True))
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 241, in train
    loss = self._train_model(input_fn=input_fn, hooks=hooks)
           │                          │               └ []
           │                          └ <function data_loader.<locals>.input_fn at 0x7f20a01fbf28>
           └ <tensorflow.python.estimator.estimator.Estimator object at 0x7f20a01fc908>
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 686, in _train_model
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
       │      │             │                        └ EstimatorSpec(predictions={'raw_predictions': <tf.Tensor 'deep_bidirectional_lstm/raw_prediction:0' shape=(64, 75) dtype=int64>,...
       │      │             └ EstimatorSpec(predictions={'raw_predictions': <tf.Tensor 'deep_bidirectional_lstm/raw_prediction:0' shape=(64, 75) dtype=int64>,...
       │      └ <tensorflow.python.training.monitored_session.MonitoredSession object at 0x7f209b0322e8>
       └ None
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 518, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 862, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 818, in run
    return self._sess.run(*args, **kwargs)
           │               │       └ {'options': None, 'feed_dict': None, 'run_metadata': None}
           │               └ ([<tf.Operation 'group_deps' type=NoOp>, <tf.Tensor 'Print:0' shape=() dtype=float32>],)
           └ <tensorflow.python.training.monitored_session._CoordinatedSession object at 0x7f209acf6390>
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 972, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 818, in run
    return self._sess.run(*args, **kwargs)
           │               │       └ {'options': , 'fetches': {'caller': [<tf.Operation 'group_deps' type=NoOp>, <tf.Tensor 'Print:0' shape=() dtype=float32>], <tens...
           │               └ ()
           └ <tensorflow.python.training.monitored_session._HookedSession object at 0x7f209acf62e8>
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 895, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1124, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1321, in _do_run
    options, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1340, in _do_call
    raise type(e)(node_def, op, message)
                  │         │   └ 'All labels must be nonnegative integers, batch: 0 labels: 12,4,14,0,-1,-1,10\n\t [[Node: CTCLoss = CTCLoss[ctc_merge_repeated=t...
                  │         └ <tf.Operation 'CTCLoss' type=CTCLoss>
                  └ name: "CTCLoss"
op: "CTCLoss"
input: "deep_bidirectional_lstm/transpose_time_major"
input: "str2code_conversion/StringSplit"
inp...
tensorflow.python.framework.errors_impl.InvalidArgumentError: All labels must be nonnegative integers, batch: 0 labels: 12,4,14,0,-1,-1,10
         [[Node: CTCLoss = CTCLoss[ctc_merge_repeated=true, ignore_longer_outputs_than_inputs=true, preprocess_collapse_repeated=false, _device="/job:localhost/replica:0/task:0/cpu:0"](deep_bidirectional_lstm/transpose_time_major/_511, str2code_conversion/StringSplit, str2code_conversion/hash_table_Lookup, Cast_3/_589)]]
         [[Node: code2str_conversion/chars_conversion/Shape/_531 = _HostRecv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_1842_code2str_conversion/chars_conversion/Shape", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

Caused by op 'CTCLoss', defined at:
  File "train.py", line 99, in <module>
    image_summaries=True))
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 241, in train
    loss = self._train_model(input_fn=input_fn, hooks=hooks)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 630, in _train_model
    model_fn_lib.ModeKeys.TRAIN)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 615, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "/data/zhoujun/tf-crnn/src/model.py", line 272, in crnn_fn
    time_major=True)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/ctc_ops.py", line 152, in ctc_loss
    ignore_longer_outputs_than_inputs=ignore_longer_outputs_than_inputs)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_ctc_ops.py", line 168, in _ctc_loss
    name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1204, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): All labels must be nonnegative integers, batch: 0 labels: 12,4,14,0,-1,-1,10
         [[Node: CTCLoss = CTCLoss[ctc_merge_repeated=true, ignore_longer_outputs_than_inputs=true, preprocess_collapse_repeated=false, _device="/job:localhost/replica:0/task:0/cpu:0"](deep_bidirectional_lstm/transpose_time_major/_511, str2code_conversion/StringSplit, str2code_conversion/hash_table_Lookup, Cast_3/_589)]]
         [[Node: code2str_conversion/chars_conversion/Shape/_531 = _HostRecv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_1842_code2str_conversion/chars_conversion/Shape", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

and the image i used is like this

CTC_loss not work~

Hey,thks for your brilliant work~
I tried install warp_ctc and bind it with tensorflow,there are so many problems.Thus,i tried your ctc_loss in code

    loss_ctc = tf.nn.ctc_loss(labels=sparse_code_target,
                              inputs=predictions_dict['prob'],
                              sequence_length=tf.cast(features['images'].shape[0], tf.int32),
                              preprocess_collapse_repeated=False,
                              ctc_merge_repeated=True,
                              ignore_longer_outputs_than_inputs=False,
                              time_major=True)

---

however,tf throw that message:
ValueError: Shape must be rank 1 but is rank 0 for 'CTCLoss' (op: 'CTCLoss') with input shapes: [24,128,37], [?,2], [?], [].

due to i'm a noob in tensorflow,so can you get me out of trouble?

problem with training: less leaves in the beam search than requested

python train.py with config3.json
...
INFO:tensorflow:Saving checkpoints for 1101 into ./model3/model.ckpt.
INFO - tensorflow - Saving checkpoints for 1101 into ./model3/model.ckpt.
INFO:tensorflow:global_step/sec: 1.98484
INFO - tensorflow - global_step/sec: 1.98484
INFO:tensorflow:loss = 0.43665585, step = 1100 (50.381 sec)
INFO - tensorflow - loss = 0.43665585, step = 1100 (50.381 sec)

• Loss : [0.441212237]
  ...
• Loss : [0.68871814]
  2018-08-14 11:59:03.120680: W tensorflow/core/util/ctc/ctc_loss_calculator.cc:144] No valid path found.
• Loss : [inf]
• Loss : [0.642776]
• Loss : [0.515933156]
• Loss : [0.659167171]
  INFO:tensorflow:global_step/sec: 1.87844
  INFO - tensorflow - global_step/sec: 1.87844
  INFO:tensorflow:loss = 0.6591672, step = 1200 (53.237 sec)
  INFO - tensorflow - loss = 0.6591672, step = 1200 (53.237 sec)
• Loss : [1.09572434]
  ...
• Loss : [0.784255]
  2018-08-14 11:59:26.465221: W tensorflow/core/util/ctc/ctc_loss_calculator.cc:144] No valid path found.
• Loss : [inf]
• Loss : [0.728220403]
  ...
• Loss : [0.532087326]
  2018-08-14 12:00:05.965566: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at ctc_decoder_ops.cc:322 : Invalid argument: Less leaves in the beam search than requested.
  ERROR - CRNN_experiment - Failed after 0:11:56!
  Traceback (most recent calls WITHOUT Sacred internals):
  File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/tensorflow_gpu-1.9.0-py3.6-linux-x86_64.egg/tensorflow/python/client/session.py", line 1322, in _do_call
  return fn(*args)
  File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/tensorflow_gpu-1.9.0-py3.6-linux-x86_64.egg/tensorflow/python/client/session.py", line 1307, in _run_fn
  options, feed_dict, fetch_list, target_list, run_metadata)
  File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/tensorflow_gpu-1.9.0-py3.6-linux-x86_64.egg/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
  run_metadata)
  tensorflow.python.framework.errors_impl.InvalidArgumentError: Less leaves in the beam search than requested.
  [[Node: code2str_conversion/CTCBeamSearchDecoder = CTCBeamSearchDecoder[beam_width=100, merge_repeated=false, top_paths=2, _device="/job:localhost/replica:0/task:0/device:CPU:0"](deep_bidirectional_lstm/transpose_time_major/_667, code2str_conversion/Cast_1/_695)]]
  [[Node: code2str_conversion/chars_conversion/cond/map/TensorArray_1/_731 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_3701_code2str_conversion/chars_conversion/cond/map/TensorArray_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]

During handling of the above exception, another exception occurred:

Traceback (most recent calls WITHOUT Sacred internals):
File "train_mk.py", line 118, in run
image_summaries=True))
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/tensorflow_gpu-1.9.0-py3.6-linux-x86_64.egg/tensorflow/python/estimator/estimator.py", line 366, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/tensorflow_gpu-1.9.0-py3.6-linux-x86_64.egg/tensorflow/python/estimator/estimator.py", line 1119, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/tensorflow_gpu-1.9.0-py3.6-linux-x86_64.egg/tensorflow/python/estimator/estimator.py", line 1135, in _train_model_default
saving_listeners)
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/tensorflow_gpu-1.9.0-py3.6-linux-x86_64.egg/tensorflow/python/estimator/estimator.py", line 1336, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/tensorflow_gpu-1.9.0-py3.6-linux-x86_64.egg/tensorflow/python/training/monitored_session.py", line 577, in run
run_metadata=run_metadata)
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/tensorflow_gpu-1.9.0-py3.6-linux-x86_64.egg/tensorflow/python/training/monitored_session.py", line 1053, in run
run_metadata=run_metadata)
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/tensorflow_gpu-1.9.0-py3.6-linux-x86_64.egg/tensorflow/python/training/monitored_session.py", line 1144, in run
raise six.reraise(*original_exc_info)
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/tensorflow_gpu-1.9.0-py3.6-linux-x86_64.egg/tensorflow/python/training/monitored_session.py", line 1129, in run
return self._sess.run(*args, **kwargs)
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/tensorflow_gpu-1.9.0-py3.6-linux-x86_64.egg/tensorflow/python/training/monitored_session.py", line 1201, in run
run_metadata=run_metadata)
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/tensorflow_gpu-1.9.0-py3.6-linux-x86_64.egg/tensorflow/python/training/monitored_session.py", line 981, in run
return self._sess.run(*args, **kwargs)
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/tensorflow_gpu-1.9.0-py3.6-linux-x86_64.egg/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/tensorflow_gpu-1.9.0-py3.6-linux-x86_64.egg/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/tensorflow_gpu-1.9.0-py3.6-linux-x86_64.egg/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/tensorflow_gpu-1.9.0-py3.6-linux-x86_64.egg/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Less leaves in the beam search than requested.
[[Node: code2str_conversion/CTCBeamSearchDecoder = CTCBeamSearchDecoder[beam_width=100, merge_repeated=false, top_paths=2, _device="/job:localhost/replica:0/task:0/device:CPU:0"](deep_bidirectional_lstm/transpose_time_major/_667, code2str_conversion/Cast_1/_695)]]
[[Node: code2str_conversion/chars_conversion/cond/map/TensorArray_1/_731 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_3701_code2str_conversion/chars_conversion/cond/map/TensorArray_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]

Caused by op 'code2str_conversion/CTCBeamSearchDecoder', defined at:
File "train_mk.py", line 50, in
training_params: dict, _config):
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/sacred-0.7.4-py3.6.egg/sacred/experiment.py", line 137, in automain
self.run_commandline()
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/sacred-0.7.4-py3.6.egg/sacred/experiment.py", line 260, in run_commandline
return self.run(cmd_name, config_updates, named_configs, {}, args)
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/sacred-0.7.4-py3.6.egg/sacred/experiment.py", line 209, in run
run()
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/sacred-0.7.4-py3.6.egg/sacred/run.py", line 221, in call
self.result = self.main_function(*args)
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/sacred-0.7.4-py3.6.egg/sacred/config/captured_function.py", line 46, in captured_function
result = wrapped(*args, **kwargs)
File "train_mk.py", line 118, in run
image_summaries=True))
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/tensorflow_gpu-1.9.0-py3.6-linux-x86_64.egg/tensorflow/python/estimator/estimator.py", line 366, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/tensorflow_gpu-1.9.0-py3.6-linux-x86_64.egg/tensorflow/python/estimator/estimator.py", line 1119, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/tensorflow_gpu-1.9.0-py3.6-linux-x86_64.egg/tensorflow/python/estimator/estimator.py", line 1132, in _train_model_default
features, labels, model_fn_lib.ModeKeys.TRAIN, self.config)
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/tensorflow_gpu-1.9.0-py3.6-linux-x86_64.egg/tensorflow/python/estimator/estimator.py", line 1107, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "/data/progs/tf-crnn-master/tf_crnn/model.py", line 331, in crnn_fn
top_paths=parameters.num_beam_paths)
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/tensorflow_gpu-1.9.0-py3.6-linux-x86_64.egg/tensorflow/python/ops/ctc_ops.py", line 277, in ctc_beam_search_decoder
merge_repeated=merge_repeated))
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/tensorflow_gpu-1.9.0-py3.6-linux-x86_64.egg/tensorflow/python/ops/gen_ctc_ops.py", line 73, in ctc_beam_search_decoder
top_paths=top_paths, merge_repeated=merge_repeated, name=name)
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/tensorflow_gpu-1.9.0-py3.6-linux-x86_64.egg/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/tensorflow_gpu-1.9.0-py3.6-linux-x86_64.egg/tensorflow/python/framework/ops.py", line 3414, in create_op
op_def=op_def)
File "/data/Mika/anaconda3/envs/reOCR/lib/python3.6/site-packages/tensorflow_gpu-1.9.0-py3.6-linux-x86_64.egg/tensorflow/python/framework/ops.py", line 1740, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Less leaves in the beam search than requested.
[[Node: code2str_conversion/CTCBeamSearchDecoder = CTCBeamSearchDecoder[beam_width=100, merge_repeated=false, top_paths=2, _device="/job:localhost/replica:0/task:0/device:CPU:0"](deep_bidirectional_lstm/transpose_time_major/_667, code2str_conversion/Cast_1/_695)]]
[[Node: code2str_conversion/chars_conversion/cond/map/TensorArray_1/_731 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_3701_code2str_conversion/chars_conversion/cond/map/TensorArray_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]

estimator.evaluate is much slower than estimator.train

I find estimator.evaluate is much slower than estimator.train in train.py , and when estimator.evaluate is running, the gpu usage is always 0.

Change the input_shape=(, )

Hi,
i want to change the parameters input_shape in train.py to size (45, 100), but have an exception, could you please help to check?

InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [768,1024] rhs shape= [1280,1024]

Thanks

Traceback (most recent call last):
File "train.py", line 97, in
image_summaries=True))
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\estimator\estimator.py", line 314, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\estimator\estimator.py", line 815, in _train_model
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 539, in run
run_metadata=run_metadata)
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1013, in run
run_metadata=run_metadata)
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1089, in run
return self._sess.run(*args, **kwargs)
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1161, in run
run_metadata=run_metadata)
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 941, in run
return self._sess.run(*args, **kwargs)
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 895, in run
run_metadata_ptr)
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 1128, in _run
feed_dict_tensor, options, run_metadata)
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 1344, in _do_run
options, run_metadata)
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 1350, in _do_call
return fn(*args)
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 1329, in _run_fn
status, run_metadata)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 1350, in _do_call
return fn(*args)
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 1329, in _run_fn
status, run_metadata)
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 473, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [768,1024] rhs shape= [1280,1024]
[[Node: save/Assign_32 = Assign[T=DT_FLOAT, _class=["loc:@stack_bidirectional_rnn/cell_0/bidirectional_rnn/fw/basic_lstm_cell/kernel"], use_locking=true, valida
te_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](stack_bidirectional_rnn/cell_0/bidirectional_rnn/fw/basic_lstm_cell/kernel, save/RestoreV2_32)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train.py", line 108, in
preprocess_image_for_prediction(min_width=10))
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\estimator\estimator.py", line 550, in export_savedmodel
saver_for_restore.restore(session, checkpoint_path)
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\saver.py", line 1686, in restore
{self.saver_def.filename_tensor_name: save_path})
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 895, in run
run_metadata_ptr)
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 1128, in _run
feed_dict_tensor, options, run_metadata)
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 1344, in _do_run
options, run_metadata)
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 1363, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [768,1024] rhs shape= [1280,1024]
[[Node: save/Assign_32 = Assign[T=DT_FLOAT, _class=["loc:@stack_bidirectional_rnn/cell_0/bidirectional_rnn/fw/basic_lstm_cell/kernel"], use_locking=true, valida
te_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](stack_bidirectional_rnn/cell_0/bidirectional_rnn/fw/basic_lstm_cell/kernel, save/RestoreV2_32)]]

Caused by op 'save/Assign_32', defined at:
File "train.py", line 108, in
preprocess_image_for_prediction(min_width=10))
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\estimator\estimator.py", line 549, in export_savedmodel
sharded=True)
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\saver.py", line 1239, in init
self.build()
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\saver.py", line 1248, in build
self._build(self._filename, build_save=True, build_restore=True)
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\saver.py", line 1284, in _build
build_save=build_save, build_restore=build_restore)
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\saver.py", line 759, in _build_internal
restore_sequentially, reshape)
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\saver.py", line 471, in _AddShardedRestoreOps
name="restore_shard"))
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\saver.py", line 440, in _AddRestoreOps
assign_ops.append(saveable.restore(tensors, shapes))
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\saver.py", line 160, in restore
self.op.get_shape().is_fully_defined())
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\ops\state_ops.py", line 276, in assign
validate_shape=validate_shape)
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\ops\gen_state_ops.py", line 62, in assign
use_locking=use_locking, name=name)
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 3160, in create_op
op_def=op_def)
File "E:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 1625, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [768,1024] rhs shape= [1280,1024]
[[Node: save/Assign_32 = Assign[T=DT_FLOAT, _class=["loc:@stack_bidirectional_rnn/cell_0/bidirectional_rnn/fw/basic_lstm_cell/kernel"], use_locking=true, valida
te_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](stack_bidirectional_rnn/cell_0/bidirectional_rnn/fw/basic_lstm_cell/kernel, save/RestoreV2_32)]]

CSV files

I input the csv files according to your instruction but that is not working and generate following error
tensorflow.python.framework.errors_impl.InvalidArgumentError: Expect 2 fields but have 0 in record 0
could you please share with me csv files for training

using TFRecords

I'm trying to use TFRecord as the data input for the program, but I have encountered some problems。
the code used to convert img to TFRecord. I have test this code and work well

# -*- coding: utf-8 -*-
# @Time    : 2017/11/27 16:00
# @Author  : zhoujun
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import numpy as np
import tensorflow as tf
from PIL import Image
from tqdm import tqdm


def getFileName(path):
    return path.split('/')[-1]


def readLines(file_path):
    with open(file_path, 'r') as T:
        lines = T.readlines()
    return lines


def split_lines(src):
    lines = src
    label_record = {}
    for line in lines:
        name = line.split(' ')[0]
        label = line.split(' ')[1]
        label = label.split('\n')[0]
        label_record[name] = label
    return label_record


def _int64_feature(value):
    return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))


def _bytes_feature(value):
    return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))


def recordsCreater(label_file, dst_records):
    writer = tf.python_io.TFRecordWriter(dst_records)

    lines = readLines(label_file)
    label_record = split_lines(lines)
    index = 0

    pbar = tqdm(total=len(lines))
    for file_path, label in label_record.items():
        index = index + 1
        img = Image.open(file_path)
        image_raw = img.tobytes()

        cols = img.size[0]
        rows = img.size[1]
        depth = 3 if img.mode is 'RGB' else 1

        example = tf.train.Example(features=tf.train.Features(feature={
            'height': _int64_feature(rows),
            'width': _int64_feature(cols),
            'depth': _int64_feature(depth),
            'label': _bytes_feature(bytes(label, encoding = "utf8")  ),
            'image_raw': _bytes_feature(image_raw)}))
        writer.write(example.SerializeToString())
        writer.flush()
        pbar.update(1)
    print("done!")
    writer.close()
    pbar.close()

# 读取二进制数据
def recordsReader(filename):
    # 创建文件队列,不限读取的数量
    filename_queue = tf.train.string_input_producer([filename])
    # create a reader from file queue
    reader = tf.TFRecordReader()
    # reader从文件队列中读入一个序列化的样本
    _, serialized_example = reader.read(filename_queue)
    # get feature from serialized example
    # 解析符号化的样本
    features = tf.parse_single_example(
        serialized_example,
        features={
            'height': tf.FixedLenFeature([], tf.int64),
            'width': tf.FixedLenFeature([], tf.int64),
            'depth': tf.FixedLenFeature([], tf.int64),
            'label': tf.FixedLenFeature([], tf.string),
            'image_raw': tf.FixedLenFeature([], tf.string)
        }
    )
    image = tf.decode_raw(features['image_raw'], tf.uint8)
    height = tf.cast(features['height'], tf.int32)
    width = tf.cast(features['width'], tf.int32)
    depth = tf.cast(features['depth'], tf.int32)
    image = tf.reshape(image, [height,width,depth])
    label = tf.cast(features['label'], tf.string)
    return image, label

def test_reader(recordsFile):
    image, label = recordsReader(recordsFile)
    with tf.Session() as sess:  # 开始一个会话
        init_op = tf.initialize_all_variables()
        sess.run(init_op)
        coord = tf.train.Coordinator()
        threads = tf.train.start_queue_runners(coord=coord)
        for i in range(1):
            image, label = sess.run([image, label])  # 在会话中取出image和label
            print(label)
            # img = Image.fromarray(example, 'RGB')  # 如果img是RGB图像
            # img = Image.fromarray(example)
            #
            # img.save('./' + '_'+'Label_' + str(l) + '.jpg')  # 存下图片
            Image._show(Image.fromarray(image))
        coord.request_stop()
        coord.join(threads)

if __name__ == '__main__':
    test_label_file, test_dst_records = "E:\\val1.csv", "E:\\val1.tfrecords"
    # train_label_file, train_dst_records = "../MNIST_data/mnist_train/train.txt", "../MNIST_data/mnist_train.tfrecords"
    # recordsCreater(test_label_file, test_dst_records)
    # recordsCreater(test_label_file, test_dst_records)
    test_reader(test_dst_records)

the code used as imput_fn

# -*- coding: utf-8 -*-
# @Time    : 2017/11/27 19:12
# @Author  : zhoujun
import tensorflow as tf
from src.data_handler import padding_inputs_width, augment_data
from src.config import CONST,Params,Alphabet

def input_fn(filename, is_training, params, batch_size=1, num_epochs=1):
    """A simple input_fn using the tf.data input pipeline."""

    def example_parser(serialized_example):
        """Parses a single tf.Example into image and label tensors."""
        features = tf.parse_single_example(
            serialized_example,
            features={
                'height': tf.FixedLenFeature([], tf.int64),
                'width': tf.FixedLenFeature([], tf.int64),
                'depth': tf.FixedLenFeature([], tf.int64),
                'label': tf.FixedLenFeature([], tf.string),
                'image_raw': tf.FixedLenFeature([], tf.string)
            }
        )
        image = tf.decode_raw(features['image_raw'], tf.uint8)
        height = tf.cast(features['height'], tf.int32)
        width = tf.cast(features['width'], tf.int32)
        depth = tf.cast(features['depth'], tf.int32)
        image = tf.reshape(image, [height, width, depth])
        label = tf.cast(features['label'], tf.string)

        # Normalize the values of the image from the range [0, 255] to [-0.5, 0.5]
        # image = tf.cast(image, tf.float32) / 255 - 0.5
        # Data augmentation
        if is_training:
            image = augment_data(image)

        image, width = padding_inputs_width(image, params.input_shape, increment=CONST.DIMENSION_REDUCTION_W_POOLING)

        return {'images': image, 'images_widths': width, 'labels': label}, label

    dataset = tf.data.TFRecordDataset([filename])

    # Apply dataset transformations
    if is_training:
        # When choosing shuffle buffer sizes, larger sizes result in better
        # randomness, while smaller sizes have better performance. Because MNIST is
        # a small dataset, we can easily shuffle the full epoch.
        dataset = dataset.shuffle(buffer_size=1000)

    dataset = dataset.repeat(num_epochs)

    # Map example_parser over dataset, and batch results by up to batch_size
    dataset = dataset.map(example_parser).prefetch(batch_size)
    dataset = dataset.batch(batch_size)
    iterator = dataset.make_one_shot_iterator()
    image, label = iterator.get_next()

    return image, label

if __name__ == '__main__':
    parameters = Params(eval_batch_size=128,
                        input_shape=(32, 304),
                        digits_only=False,
                        alphabet=Alphabet.CHINESECHAR_LETTERS_DIGITS_EXTENDED,
                        alphabet_decoding='same',
                        )

    next_batch = input_fn(filename='E:\\val1.tfrecords.', is_training=False,params=parameters,batch_size=2)

    # Now let's try it out, retrieving and printing one batch of data.
    # Although this code looks strange, you don't need to understand
    # the details.
    with tf.Session() as sess:
        first_batch = sess.run(next_batch)
    print(first_batch)

when test this code the output is

({'images': array([[[[ 252.        ,  252.        ,  252.        ],
         [  72.777771  ,   72.777771  ,   72.777771  ],
         [ 247.74073792,  247.74073792,  247.74073792],
         ..., 
         [ 250.14813232,  250.14813232,  250.14813232],
         [ 254.29629517,  254.29629517,  254.29629517],
         [ 252.77780151,  252.77780151,  252.77780151]],

        [[ 251.25      ,  251.25      ,  251.25      ],
         [ 237.222229  ,  237.222229  ,  237.222229  ],
         [ 252.40740967,  252.40740967,  252.40740967],
         ..., 
         [ 254.66670227,  254.66670227,  254.66670227],
         [ 248.66665649,  248.66665649,  248.66665649],
         [ 253.75      ,  253.75      ,  253.75      ]],

        [[ 246.5       ,  246.5       ,  246.5       ],
         [ 253.05555725,  253.05555725,  253.05555725],
         [ 248.53703308,  248.53703308,  248.53703308],
         ..., 
         [ 253.07406616,  253.07406616,  253.07406616],
         [ 255.        ,  255.        ,  255.        ],
         [ 254.722229  ,  254.722229  ,  254.722229  ]],

        ..., 
        [[ 251.75      ,  251.75      ,  251.75      ],
         [ 251.        ,  251.        ,  251.        ],
         [ 250.66667175,  250.66667175,  250.66667175],
         ..., 
         [ 254.61114502,  254.61114502,  254.61114502],
         [ 245.16665649,  245.16665649,  245.16665649],
         [ 247.58332825,  247.58332825,  247.58332825]],

        [[ 251.5       ,  251.5       ,  251.5       ],
         [ 249.85185242,  249.85185242,  249.85185242],
         [ 248.777771  ,  248.777771  ,  248.777771  ],
         ..., 
         [ 254.79632568,  254.79632568,  254.79632568],
         [ 250.53703308,  250.53703308,  250.53703308],
         [ 251.88890076,  251.88890076,  251.88890076]],

        [[ 255.        ,  255.        ,  255.        ],
         [ 246.87037659,  246.87037659,  246.87037659],
         [ 251.8888855 ,  251.8888855 ,  251.8888855 ],
         ..., 
         [ 249.12036133,  249.12036133,  249.12036133],
         [ 251.43518066,  251.43518066,  251.43518066],
         [ 251.44442749,  251.44442749,  251.44442749]]],


       [[[ 255.        ,  255.        ,  255.        ],
         [ 255.        ,  255.        ,  255.        ],
         [ 255.        ,  255.        ,  255.        ],
         ..., 
         [ 255.        ,  255.        ,  255.        ],
         [ 255.        ,  255.        ,  255.        ],
         [ 255.        ,  255.        ,  255.        ]],

        [[ 255.        ,  255.        ,  255.        ],
         [ 255.        ,  255.        ,  255.        ],
         [ 255.        ,  255.        ,  255.        ],
         ..., 
         [ 255.        ,  255.        ,  255.        ],
         [ 255.        ,  255.        ,  255.        ],
         [ 255.        ,  255.        ,  255.        ]],

        [[ 255.        ,  255.        ,  255.        ],
         [ 255.        ,  255.        ,  255.        ],
         [ 255.        ,  255.        ,  255.        ],
         ..., 
         [ 255.        ,  255.        ,  255.        ],
         [ 255.        ,  255.        ,  255.        ],
         [ 255.        ,  255.        ,  255.        ]],

        ..., 
        [[ 250.        ,  250.        ,  250.        ],
         [ 254.88095093,  254.88095093,  254.88095093],
         [ 254.73809814,  254.73809814,  254.73809814],
         ..., 
         [ 255.        ,  255.        ,  255.        ],
         [ 253.86904907,  253.86904907,  253.86904907],
         [ 250.50003052,  250.50003052,  250.50003052]],

        [[ 254.        ,  254.        ,  254.        ],
         [ 254.16665649,  254.16665649,  254.16665649],
         [ 253.16665649,  253.16665649,  253.16665649],
         ..., 
         [ 252.        ,  252.        ,  252.        ],
         [ 250.23809814,  250.23809814,  250.23809814],
         [ 255.        ,  255.        ,  255.        ]],

        [[ 254.25      ,  254.25      ,  254.25      ],
         [ 253.45237732,  253.45237732,  253.45237732],
         [ 251.5952301 ,  251.5952301 ,  251.5952301 ],
         ..., 
         [ 253.5       ,  253.5       ,  253.5       ],
         [ 253.75      ,  253.75      ,  253.75      ],
         [ 249.42855835,  249.42855835,  249.42855835]]]], dtype=float32), 'images_widths': array([108,  84]), 'labels': array([b'2760$854$1429$1224', b'1105$1232$1560'], dtype=object)}, array([b'2760$854$1429$1224', b'1105$1232$1560'], dtype=object))

however when I use the input_fn as the input of estimator.evaluate, I meet some error and I don't know where is error.

the code is

#!/usr/bin/env python
__author__ = 'zj'

import argparse
import os
import sys
import numpy as np

try:
    import better_exceptions
except ImportError:
    pass
import tensorflow as tf
from src.model2 import crnn_fn
from src.data_handler import data_loader
from src.config import Params, Alphabet
from src.input_utils import input_fn


def main(unused_argv):
    models_path = FLAGS.input_model_dir
    if not os.path.exists(models_path):
        assert FileNotFoundError

    models_list = [os.path.join(models_path, x[:-5]) for x in os.listdir(models_path) if x.endswith('.meta')]

    # 输出路径不存在就创建
    if not os.path.exists(FLAGS.output_model_dir):
        os.makedirs(FLAGS.output_model_dir)

    parameters = Params(eval_batch_size=128,
                        input_shape=(32, 304),
                        digits_only=False,
                        alphabet=Alphabet.CHINESECHAR_LETTERS_DIGITS_EXTENDED,
                        alphabet_decoding='same',
                        csv_delimiter=' ',
                        csv_files_eval=FLAGS.csv_files_eval,
                        output_model_dir=FLAGS.output_model_dir,
                        gpu=FLAGS.gpu
                        )

    model_params = {
        'Params': parameters,
    }

    os.environ['CUDA_VISIBLE_DEVICES'] = parameters.gpu
    config_sess = tf.ConfigProto()
    config_sess.gpu_options.per_process_gpu_memory_fraction = 0.6

    # Config estimator
    est_config = tf.estimator.RunConfig()
    est_config = est_config.replace(session_config=config_sess,
                                    save_summary_steps=100,
                                    model_dir=parameters.output_model_dir)

    estimator = tf.estimator.Estimator(model_fn=crnn_fn,
                                       params=model_params,
                                       config=est_config,
                                       model_dir=parameters.output_model_dir,
                                       )
    # Count number of image filenames in csv
    n_samples = 0
    for file in parameters.csv_files_eval:
        with open(file, mode='r', encoding='utf8') as csvfile:
            n_samples += len(csvfile.readlines())
    print(n_samples, np.floor(n_samples / parameters.eval_batch_size), parameters.eval_batch_size)
    try:
        with open(FLAGS.output_file, encoding='utf-8', mode='w') as save_file:
            for model in models_list:
                # eval_results = estimator.evaluate(input_fn=data_loader(csv_filename=parameters.csv_files_eval,
                #                                                        params=parameters,
                #                                                        batch_size=parameters.eval_batch_size,
                #                                                        num_epochs=1),
                #                                   steps=np.floor(n_samples / parameters.eval_batch_size),
                #                                   checkpoint_path=model)
                eval_results = estimator.evaluate(input_fn=input_fn(filename=parameters.csv_files_eval,
                                                                    is_training=False,
                                                                    params=parameters,
                                                                    batch_size=parameters.eval_batch_size,
                                                                    num_epochs=1),
                                                  steps=np.floor(n_samples / parameters.eval_batch_size),
                                                  checkpoint_path=model)
                print('model: %s Evaluation results: %s' % (model, str(eval_results)))
                save_file.write(model + ' ' + str(eval_results) + '\n')

    except KeyboardInterrupt:
        print('Interrupted')


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('-fe', '--csv_files_eval', required=True, type=str, help='CSV filename for evaluation',
                        nargs='*', default=None)
    parser.add_argument('-o', '--output_model_dir', required=True, type=str,
                        help='Directory for output', default='./estimator')
    parser.add_argument('-m', '--input_model_dir', required=True, type=str,
                        help='Directory for output', default='./estimator')
    parser.add_argument('-g', '--gpu', type=str, help="GPU 0,1 or '' ", default='0')
    parser.add_argument('-of', '--output_file', required=True, type=str, help="the log output file")

    tf.logging.set_verbosity(tf.logging.DEBUG)
    FLAGS, unparsed = parser.parse_known_args()
    tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)

run script is

python3 test_model.py -fe /data/zj/data/val1.tfrecords -o result/model_test1 -m models_vgg_100K_no_eval/1 -g 3 -of test_model1.txt

the error is

INFO:tensorflow:Using config: {'_task_id': 0, '_num_worker_replicas': 1, '_num_ps_replicas': 0, '_session_config': gpu_options {
  per_process_gpu_memory_fraction: 0.6
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_master': '', '_tf_random_seed': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f5a50a2f898>, '_save_checkpoints_secs': 600, '_is_chief': True, '_save_checkpoints_steps': None, '_log_step_count_steps': 100, '_service': None, '_model_dir': 'result/model_test1', '_task_type': 'worker', '_save_summary_steps': 100}
10000 78.0 128
Traceback (most recent call last):
  File "test_model.py", line 103, in <module>
    tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
    │               │           │              └ []
    │               │           └ <module 'sys' (built-in)>
    │               └ <function main at 0x7f5a94cd5f28>
    └ <module 'tensorflow' from '/usr/local/lib/python3.5/dist-packages/tensorflow/__init__.py'>
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
    │         │    │               └ []
    │         │    └ <module 'sys' (built-in)>
    │         └ <function main at 0x7f5a94cd5f28>
    └ <module 'sys' (built-in)>
  File "test_model.py", line 82, in main
    checkpoint_path=model)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 355, in evaluate
    name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 808, in _evaluate_model
    input_fn, model_fn_lib.ModeKeys.EVAL)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 577, in _get_features_and_labels_from_input_fn
    result = self._call_input_fn(input_fn, mode)
             │                   │         └ 'eval'
             │                   └ ({'images_widths': <tf.Tensor 'IteratorGetNext:1' shape=(?,) dtype=int32>, 'images': <tf.Tensor 'IteratorGetNext:0' shape=(?, 32...
             └ <tensorflow.python.estimator.estimator.Estimator object at 0x7f5a50a2f6d8>
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 656, in _call_input_fn
    input_fn_args = util.fn_args(input_fn)
                    │            └ ({'images_widths': <tf.Tensor 'IteratorGetNext:1' shape=(?,) dtype=int32>, 'images': <tf.Tensor 'IteratorGetNext:0' shape=(?, 32...
                    └ <module 'tensorflow.python.estimator.util' from '/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/util.py'>
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/util.py", line 57, in fn_args
    return tuple(tf_inspect.getargspec(fn).args)
                 │                     └ ({'images_widths': <tf.Tensor 'IteratorGetNext:1' shape=(?,) dtype=int32>, 'images': <tf.Tensor 'IteratorGetNext:0' shape=(?, 32...
                 └ <module 'tensorflow.python.util.tf_inspect' from '/usr/local/lib/python3.5/dist-packages/tensorflow/python/util/tf_inspect.py'>
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/util/tf_inspect.py", line 45, in getargspec
    if d.decorator_argspec is not None), _inspect.getargspec(target))
  File "/usr/lib/python3.5/inspect.py", line 1043, in getargspec
    getfullargspec(func)
    │              └ ({'images_widths': <tf.Tensor 'IteratorGetNext:1' shape=(?,) dtype=int32>, 'images': <tf.Tensor 'IteratorGetNext:0' shape=(?, 32...
    └ <function getfullargspec at 0x7f5a1855bf28>
  File "/usr/lib/python3.5/inspect.py", line 1095, in getfullargspec
    raise TypeError('unsupported callable') from ex
TypeError: unsupported callable

should space character take label?

In ASR task, silence is labeled for CTC loss. Isn't need for OCR?

How to get word from chars without tensorflow

hi，
when I get the raw_predictions,how to get word from chars without tensorlfow.
Thx

why calculate the seq_len_inputs by use img_width / 2*2

the seq_len_inputs is the length of string in the input image or the pixels of single character

What is the role of fully-connection layer after rnn layers?

I see a full connection layer after rnn, I do not know what is the role of this layer, after removing what will be the impact?

Usage of Lexicon

First of all thank you for the code.
Does your implementation contain a lexicon functionality as described in the original paper?

One question of scared

When I ran this program，I got this : “sacred.utils.SacredError: The configuration is read-only in a captured function!". So how can I solve this problem and run the program correctly?

Is there a script to test the model with one images

now, I have trained a model with mnist dataset, how can I test the model?, or Is there a script ?

Longer outputs than inputs when training on word datasets

When training on word-recognition datasets, the sequence outputs are often longer than the image inputs, and Tensorflow will say something like Not enough time for target transition sequence (required: x, available: y). I suspect this is a problem with the dataset itself (i.e. the target sequence is longer than the CTC input sequence collected from the image for some reason - maybe some items in the dataset are mislabelled), does anyone else have a similar issue?

I can get around this issue by setting ignore_longer_outputs_than_inputs=False in tf.nn.ctc_loss but then I get a cryptic error message: check failed: start <= limit (x vs. y), which makes me think they are related.

Using the regular conjoined MNIST examples or datasets containing only sequences of digits trains fine, but using the synthetic dataset and other word-recognition datasets is plagued with this problem.

Why use symmetric padding ?

Hi,

I noticed that in the case of input images of different sizes, the padding is with SYMMETRIC values and only at the right side of the image (code). Why not use the CONSTANT option, with constant_values = 255 (or 1 if the image is normalised) ?
(edit: or constant_values = avg_color of the image)

Also, in the other case, when the image is less than half the target_width, it is replicated horizontally. However, the ground-truth data is not replicated. Does this not hurt the algorithm ?

Thank you !

Frozen a model issues

Hi,
thanks for your repo.
I want to export a frozen model after trained, but how to set the input & output node name?
i tried output_node_names = "code2str_conversion/predicted_words" but seems input "images" is not in graph:
ValueError: Attempted to map inputs that were not found in graph_def: [images:0]

Could you please advice how can i export a frozen model?

Thanks.

train Synth90K error

Error Information:tensorflow.python.framework.errors_impl.InvalidArgumentError: Less leaves in the beam search than requested.

I think that the problem is CTC beam search error, however, I don't know how to solve.

Batch inference is not working with exported model

Hi All,

Need help on how to inference images in batches.
I get this error when I try with current exported model

ValueError: Cannot feed value of shape (1, 26, 218, 1) for Tensor 'Placeholder:0', which has shape '(?, ?, 1)'

Tried changing preprocess_image_for_prediction() in src/data_handler.py but getting one or the other error.
Current code is

def serving_input_fn():
        # define placeholder for input image
        image = tf.placeholder(dtype=tf.float32, shape=[None, None, None, 1])
        print("image shape ", tf.shape(image))
        shape = tf.shape(image)
        print("Shape variable ", shape)
        # Assert shape is h x w x c with c = 1
        ratio = tf.divide(shape[2], shape[1])
        print("ratio ", ratio)
        increment = CONST.DIMENSION_REDUCTION_W_POOLING
        new_width = tf.cast(tf.round((ratio * fixed_height) / increment) * increment, tf.int32)
        print("new_width ", new_width)
        resized_image = tf.cond(new_width < tf.constant(min_width, dtype=tf.int32),
                                true_fn=lambda: tf.image.resize_images(image, size=(fixed_height, min_width)),
                                false_fn=lambda: tf.image.resize_images(image, size=(fixed_height, new_width))
                                )
        print("resized image shape ", tf.shape(resized_image))
        print("new_width[None] ", new_width[None])
        print("type of resized_input ", type(resized_image[None]))
        # Features to serve
        features = {'images': resized_image,  # cast to 1 x h x w x c
                    'images_widths': new_width[None]  # cast to tensor
                    }
        # Inputs received
        receiver_inputs = {'images': image}
        return tf.estimator.export.ServingInputReceiver(features, receiver_inputs)

and I get bellow error with this code.

Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/framework/tensor_util.py", line 460, in make_tensor_proto
str_values = [compat.as_bytes(x) for x in proto_values]
File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/framework/tensor_util.py", line 460, in
str_values = [compat.as_bytes(x) for x in proto_values]
File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/util/compat.py", line 65, in as_bytes
(bytes_or_text,))
TypeError: Expected binary or unicode string, got None
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "train.py", line 114, in
preprocess_image_for_prediction(min_width=10))
File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 436, in export_savedmodel
mode=model_fn_lib.ModeKeys.PREDICT)
File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 615, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "/home/ubuntu/ctnn/tf-crnn/src/model.py", line 236, in crnn_fn
conv = deep_cnn(features['images'], (mode == tf.estimator.ModeKeys.TRAIN), summaries=False)
File "/home/ubuntu/ctnn/tf-crnn/src/model.py", line 160, in deep_cnn
name='reshaped') # [batch, width, height x features]
File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/ops/gen_array_ops.py", line 2619, in reshape
name=name)
File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 493, in apply_op
raise err
File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 490, in apply_op
preferred_dtype=default_dtype)
File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 676, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/framework/constant_op.py", line 121, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/framework/constant_op.py", line 102, in constant
tensor_util.make_tensor_proto(value, dtype=dtype, shape=shape, verify_shape=verify_shape))
File "/home/ubuntu/.local/lib/python3.5/site-packages/tensorflow/python/framework/tensor_util.py", line 464, in make_tensor_proto
"supported type." % (type(values), values))
TypeError: Failed to convert object of type <class 'list'> to Tensor. Contents: [None, -1, 512]. Consider casting elements to a supported type.

Let me know if anyone has solved this.

Error at the end of traning

I have provided .csv file according to instructions at end of 100% training i received following errror
tensorflow.python.framework.errors_impl.NotFoundError: Failed to create a directory: ./export_model_dir\export\temp-b'1514633130'; No such file or directory

csv_delimiter bug

in file train.py
parameters = Params(train_batch_size=128, eval_batch_size=128, learning_rate=1e-3, # 1e-3 recommended learning_decay_rate=0.95, learning_decay_steps=5000, evaluate_every_epoch=5, save_interval=5e3, input_shape=(32, 304), optimizer='adam', digits_only=False, alphabet=Alphabet.LETTERS_DIGITS_EXTENDED, alphabet_decoding='same', csv_delimiter=' ', csv_files_eval=args.get('csv_files_eval'), csv_files_train=args.get('csv_files_train'), output_model_dir=args.get('output_model_dir'), n_epochs=args.get('nb_epochs'), gpu=args.get('gpu') )
csv_delimiter should be assigned ' '

How should I build a character dictionary config file

I am gonna run the first test on the data I have but there is no documentation about how I should recreate a similar file as https://github.com/solivr/tf-crnn/blob/master/tf_crnn/data/lookup_letters_digits_symbols.json and of course I am already running into an error :) :

Traceback (most recent calls WITHOUT Sacred internals):
  File "train.py", line 79, in run
    discarded_chars=discarded_chars)
  File "/script/tf_crnn/config.py", line 69, in check_input_file_alphabet
    filename, extra_chars)
AssertionError: There are 30 unknown chars in /sources/output/train/groundtruth.csv :
 {'\u0367', '\ua76f', '\u036c', '\u014d', '\u2022', '\u1e8f', '\u1e6b', '\u1de4',
 '\u0364', '\ua759', '\xf1', '\u0129', '\u1d49', '\u0291', '\u204a', '\ue5dc', 
'\u01ba', '\u0363', '\ua751', '\u0365', '\u0113', '\u1d48', '\u0167', '\ufeff', 
'\xe9', '\u016b', '\u0305', '\ue681', '\u0153', '\u0101'}

Any hint how it should be built would be greatly appreciated. I'll probably add it to my ocropus script :)