Training loss remains nan about fcn.tensorflow

[shekkizh]

Each value in the annotation image corresponds to an object in the image. NUM_OF_CLASSESS corresponds to the maximum number of objects to be annotated i.e the max. value in annotation image provided the values are continuous.

Nan can result due to several reasons. Assuming your inputs are all fine, try reducing learning rate.

[jiao0805]

Thank you very much
will give a try

[jiao0805]

I tried to reduce my learning rate but the loss still remained nan.
I turned on the debug mode and i got the following error.

InvalidArgumentError (see above for traceback): Nan in summary histogram for: inference/conv2_1_w_1
[[Node: inference/conv2_1_w_1 = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](inference/conv2_1_w_1/tag, inference/conv2_1_w/read/_107)]]

CONV2_1_W_1 is loaded from the vgg19.mat, isn't it？
Then what makes the node value nan？

[shekkizh]

Although the weights are loaded from saved file while training you are changing weights which at some point becomes nan. Try learning rates in the order of 10-7 or even smaller to verify if the problem is actually due to large loss terms.

[shekkizh]

@jiao0805 Maybe you are facing the same problem discussed in Issue #17

[jiao0805]

Yes I solved this problem by change the num of classes。
Thanks a lot and I will close this problem

[myusernamesxy]

Hi, I face the same problem with you, but I haven't solved it, may I discuss with you?