Error in weights file I guess...I am at a loss about keras-transfer-learning-for-oxford102

[shirishr]

I ran the train.py with these theano flags

optimizer=fast_compile
exception_verbosity=high

to get some more information:

Model is created
Fine tuning...
Freezing 80 layers
Found 6149 images belonging to 102 classes.
Found 1020 images belonging to 102 classes.
./models\base_model.py:43: UserWarning: Update your `fit_generator` call to the Keras 2 API: `fit_generator(<keras.pre..., epochs=1000, class_weight={0: 2.4765..., validation_data=<keras.pre..., validation_steps=1020, callbacks=[<keras.ca..., steps_per_epoch=192)`
  class_weight=self.class_weight)
Epoch 1/1000

*GpuDnnConv images and kernel must have the same stack size*

Apply node that caused the error: GpuDnnConv{algo='small', inplace=False}(GpuContiguous.0, GpuContiguous.0, GpuAllocEmpty.0, GpuDnnConvDesc{border_mode='valid', subsample=(2, 2), conv_mode='conv', precision='float32'}.0, Constant{1.0}, Constant{0.0})
Toposort index: 2266
Inputs types: [CudaNdarrayType(float32, 4D), CudaNdarrayType(float32, 4D), CudaNdarrayType(float32, 4D), <theano.gof.type.CDataType object at 0x000001E9F08BC3C8>, Scalar(float32), Scalar(float32)]
Inputs shapes: [(32, 3, 230, 230), (7, 7, 3, 64), (32, 7, 114, 84), 'No shapes', (), ()]
Inputs strides: [(158700, 52900, 230, 1), (1344, 192, 64, 1), (67032, 9576, 84, 1), 'No strides', (), ()]
Inputs values: ['not shown', 'not shown', 'not shown', <capsule object NULL at 0x000001EA0BD67540>, 1.0, 0.0]
Inputs type_num: ['', '', '', '', 11, 11]
Inputs name: ('image', 'kernel', 'output', 'descriptor', 'alpha', 'beta')

[Arsey]

@shirishr what Python and TensorFlow version do you use?

[shirishr]

Sasha, I am using Python version 3.5 and Tensorflow 1.2 (Windows 10 with CUDA GPU) Thanks Shirish

…

________________________________ From: Alexander Lazarev <[email protected]> Sent: Thursday, July 20, 2017 3:34 PM To: Arsey/keras-transfer-learning-for-oxford102 Cc: Shirish (Sam) Ranade; Mention Subject: Re: [Arsey/keras-transfer-learning-for-oxford102] Error in weights file I guess...I am at a loss (#21) @shirishr<https://github.com/shirishr> what Python and TensorFlow version do you use? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#21 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AHaPnabk2eVt9OicjkERyAXLPkYPxp2dks5sP3OMgaJpZM4OdVVh>.

[shirishr]

Sasha,
On the same machine I have a virtual machine with Ubuntu on it. In this I have installed Python 2.7 and Theano (No GPU) where everything seems to work but training is so slow ...it like watching grass grow slowly.

Do you have a trained model with Python 3.5? Should we ask @fcollet?

[Arsey]

@shirishr and what's the Kerase's version?
BTW don't think we should mention fcollet here, also weights issue is backend specific

[shirishr]

I see. My Keras version is 2.0.2

[shirishr]

Hello Sasha,

I upgraded my keras version 2.0.6 by using

pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps

That eliminated the error. So it is safe to close the issue

[shirishr]

I suggest a minor code change at lines 65~66 as:
the current lines are:

def get_input_tensor(self):
    return Input(shape=(3,) + self.img_size)

which should be changed to:

def get_input_tensor(self):
	if keras.backend.backend()=='tensorflow':
		return Input(self.img_size + (3,))
	else:
		return Input(shape=(3,) + self.img_size)

I am assuming theano and CNTK use identical image_data_format

[Arsey]

@shirishr thanks for your patient. I'm just in process of adopting the code for
(Keras 1 or Keras 2) + (Theano or Tensorflow) + (Python 2 or Python 3)

[shirishr]

Sasha it's a very nice piece of software engineering. It has worked for me (with the code change that I suggested) for the following combination: Windows 10 + GPU (that too with 1050Ti which NVIDIA says is not supported for computations) + Keras 2.0.6 + Tensorflow 1.2.1 + Python 3.5 For some reason when I use Theano....it uses CPU i/o GPU (I have to figure out why it is so) The training code has been running overnight with 40 epochs completed and accuracy is 95% already. I'll stop at 100 epochs and see how the client and server part of your code works. Thank you for a good code on Github, Shirish

…

________________________________ From: Alexander Lazarev <[email protected]> Sent: Monday, July 24, 2017 5:48 AM To: Arsey/keras-transfer-learning-for-oxford102 Cc: Shirish (Sam) Ranade; Mention Subject: Re: [Arsey/keras-transfer-learning-for-oxford102] Error in weights file I guess...I am at a loss (#21) @shirishr<https://github.com/shirishr> thanks for your patient. I'm just in process of adopting the code for (Keras 1 or Keras 2) + (Theano or Tensorflow) + (Python 2 or Python 3) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#21 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AHaPnXsVHe0YlgGFVby71t6x25agM1pbks5sRDAjgaJpZM4OdVVh>.

[Arsey]

@shirishr I'm glad you like it.
Speaking about Theano, try to set device using env variable THEANO_FLAGS='device=gpu0' or THEANO_FLAGS='device=cuda', also see the warnings on running the train process

[shirishr]

I have tried setting backend to "theano" in keras.json
and set device = "gpu" in .theanorc
and seen some compiler errors (gcc++ not found or minGW or cl.exe not found) I don't exactly remember. I have abandoned theano for the time being. I am afraid I may break something else if I change current setup. Maybe one of these days I will set a target to myself and fix that issue as well.
Thanks

[Arsey]

if TensorFlow works for you just use it as it's even faster and more perspective