Issue with training LM WordLevelOneBwdsSimpleSampledSoftmax Model about lingvo

[jonathanasdf]

The error is here

2019-04-17 09:53:49.916769: E tensorflow/core/common_runtime/executor.cc:636] Executor failed to create kernel. Invalid argument: Cannot parse tensor from tensor_proto.
[[{{node 1bwds_word_level_lm/lm/softmax/weight_0/var/Adagrad/Initializer/Const}}]]
2019-04-17 09:53:50.831555: W tensorflow/core/framework/op_kernel.cc:1408] OP_REQUIRES failed at constant_op.cc:76 : Invalid argument: Cannot parse tensor from proto: dtype: DT_FLOAT
tensor_shape {
dim {
size: 99184
}
dim {
size: 512
}
}
float_val: 1

This is likely some kind of version mismatch in either tensorflow or protobuf. Are you running this inside docker or with your own configurations?

[ashu5644]

I am running it inside docker only with image at lingvo github repo.
tf version: tf-nightly 1.14.1-dev20190307

[jonathanasdf]

Ok, I will look into this next monday.

[ashu5644]

@jonathanasdf, Is this error is due to running on single machine and large tensor size? I have found a similar issue on stackeoverflow, https://stackoverflow.com/questions/46941747/tensorflow-error-cannot-parse-tensor-from-proto

[jonathanasdf]

It should be different -- in your case the tensor is only 99184*512*4=200MB... but maybe you had less than 200MB free for some reason.

I just did a fresh clone of the repo, started the docker, ran download_lm1b.py and the command you posted and everything ran fine, so sorry I cannot reproduce your problem.

[ashu5644]

I am trying to run model on single machine with single gpu, are you running with same configurations?

[jonathanasdf]

Yes, I ran on my local machine with a single GPU with 12GB memory.

Maybe your GPU only has 8GB memory which is probably not enough? If changing rnn_state_dim to something smaller makes it stop crashing then that might be the problem.