Flowers retraining example on inception-v3 diverges (with loss = NaN) about models

[sguada]

can you try with
.../inception/flowers_train
--train_dir=/tmp/flowers
--batch_size=16
--data_dir=/tmp/flowers-data
--max_steps=2000
--input_queue_memory_factor=2
--pretrained_model_checkpoint_path=model.ckpt-157585
--fine_tune=True
--initial_learning_rate=0.001
--num_epochs_per_decay=10

[robwell]

Yes, thank you! I am currently running cifar10 with multiple GPUs just to make sure I'm not running into something funny not related to the flowers example. I will follow your suggestion as soon as that finishes.

[robwell]

I ran your suggested setup 10 times and it diverged to NaN after: [10, 490, 110, 100, 510, 250, 280, 230, 90, 100] steps (with num_gpus=1)

Might this be a floating point precision issue?

In contrast, cifar10 with multiple GPUs (cifar10_multi_gpu_train.py) ran 20,000 steps before I stopped it.

[sguada]

My guess is that since CIFAR10 is only 32x32 pixels, when it gets cropped and expanded to 299x299 the activations of the network can become too flat and gradients unstable.
Try changing the --image_size=150 although that could be still too big for cifar10.
Another option is keep decreasing the --initial_learning_rate until it stops diverging.

[robwell]

I'm sorry, I did not point inception-v3 at CIFAR10 - I ran the simple tutorial demo (cifar10_multi_gpu_train.py) just as a way of making sure that the newly build machine I have is not failing for some reason unrelated to the inception-v3 codes.

I did try to decrease the initial_learning_rate and stopped when I got to 10e-9 as that was a millionth the rate suggested in the example. I also messed around with the batch_size. I am rather baffled.

[sguada]

At least there is an error in the code I would like to close this issue. As mentioned inception_v3 maybe is not best suited for CIFAR10 due to its low resolution

[robwell]

I've given up on inception-v3 for my work so I cannot report any more insights other then those I've mentioned above.

I do not understand "At least there is an error in the code"

To reiterate the CIFAR10 comment above - I did not use inception-v3 for that, I was simply reporting that the GPUs I am using were able to handle that example. I was trying to be helpful and eliminate the possibility that my GPU configuration was causing more general problems then the issues I was having with inception-v3

[RT-TL]

@robwell This issue has come of age but did you re-address this & come across a solution? I have the same issue (training with CPU instead of GPU). The issue also appears with mobilnet for most configuurations.