load_model failing when running validate_on_lfw about facenet

[GangqiangZhao]

The similar problem encountered when I am try to import the "model.ckpt-500000" mode:

Not found: Tensor name "incept3c/in3_conv5x5_19/weights/ExponentialMovingAverage_1" not found in checkpoint files

[nbenhaim]

I reverted the validate_on_lfw.py to the version which loads the model without needing the meta file, and it worked. It appears as if the provided meta file is not compatible with the trained ckpt model file. If you load the model by creating variables using the call to inference and then restoring all the variables, it works

[GangqiangZhao]

Great, thanks. But how to change the validate_on_lfw.py file?

[nbenhaim]

i looked at this revision and changed my file based on this - load the model as he was doing it before this change

d17aa06

[davidsandberg]

This is a bit strange. Just to make sure I understand correctly... On the wiki there is a link to a zip file that contains both the model parameters and the meta-file (that contains the model definition). Are you using this meta-file?
Here is what it looks like when extracted:

david@deeplab1:~/datasets$ ls -al ~/Downloads/20160514-234418/
total 88256
drwxrwxr-x 2 david david     4096 jul 19 08:27 .
drwxr-xr-x 6 david david    12288 jul 19 08:27 ..
-rw------- 1 david david 83731015 jul  8 00:13 model.ckpt-500000
-rw------- 1 david david  6616113 jul  8 00:13 model.ckpt-500000.meta

And when I run validate_on_lfw.py i set the parameter
--model_file ~/Downloads/20160514-234418/model.ckpt-500000
And when I run this it loads the model without any problems.

[GangqiangZhao]

I am sure the same model and meta data is used.

[nbenhaim]

@davidsandberg , I downloaded the exact zip file you reference, and I get an error related to a variable ExponentialMovingAverage_1 not being found just as @GangqiangZhao described as well. I can try again later to double check

[davidsandberg]

Which Tensorflow release are you using? I'm on TF r0.9. I'm not sure that this problem is related to tensorflow release but I have difficulties seeing what else it could be.

[GangqiangZhao]

I am using the master version in Github.

[GangqiangZhao]

After updating the numpy to 1.11 version, I am finally able to complete the lfw validation on Tensorflow version r0.8. It looks like the problem is caused by the different version of tensorflow and some unknown tensorflow install setup.

[davidsandberg]

I reverted to TF r0.8 and it's still working fine. I'm on numpy 1.10.2.

[Jinwoo-Jeon]

I was on gtx1080, cuda8.0, TF r0.9 and numpy 1.11 version. I was also experiencing same error. But after downgrading TF to r0.8 it works fine with the last commit.

77b66a2

[stgzr]

Maybe it is caused by different version of tensorflow?
For my case, when I execute validate_on_lfw.py using tensorflow r0.9, I met serveral errors as follows:

......
W tensorflow/core/framework/op_kernel.cc:936] Not found: Tensor name "incept4e/in2_conv3x3_45/batch_norm/batch_norm/gamma/ExponentialMovingAverage_1" not found in checkpoint files models/model.ckpt-500000
[[Node: ema_restore/restore_slice_447 = RestoreSlice[dt=DT_FLOAT, preferred_shard=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_ema_restore/Const_0, ema_restore/restore_slice_447/tensor_name, ema_restore/restore_slice_447/shape_and_slice)]]
E tensorflow/core/client/tensor_c_api.cc:485] Tensor name "incept4e/in2_conv3x3_45/batch_norm/batch_norm/gamma/ExponentialMovingAverage_1" not found in checkpoint files models/model.ckpt-500000
[[Node: ema_restore/restore_slice_447 = RestoreSlice[dt=DT_FLOAT, preferred_shard=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_ema_restore/Const_0, ema_restore/restore_slice_447/tensor_name, ema_restore/restore_slice_447/shape_and_slice)]]
[[Node: ema_restore/restore_slice_462/_84 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:2", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_3252_ema_restore/restore_slice_462", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:2"]]
......

While after I uninstalled r0.9 and rollback to tensorflow r0.8, it is working fine now.

[LiuzcEECS]

I think the issue might have been solved since some problem of function load_model() in facenet.py has been fixed.

[lbg940131]

I don't know why "images_placeholder.get_shape()[1]" returns (?,?)
Is anyone getting the same error?

[LiuzcEECS]

@lbg940131 I thought the "?" might mean the size can be changed according to the number of the images you feed