Kernel crashing with an unknow reason about tcav

[jameswex]

@francescobodria Would you be able to point us to the model files as well so we can test this ourselves and debug? Also point us to the data directories for the images you are using. Thanks!

@a-brain-known-as-ralph FYI

[francescobodria]

Sure,
I've zipped all in this file. Since I am using CIFAR dataset the images are very small. The model is called cnn_simple_cifar, the target folder is deer, the concepts folders are soil, grass and garden.
The model specs are:
Model: "sequential_1"

Layer (type) Output Shape Param

conv2d_1 (Conv2D) (None, 32, 32, 32) 896
conv2d_2 (Conv2D) (None, 32, 32, 32) 9248
max_pooling2d_1 (MaxPooling2 (None, 16, 16, 32) 0
dropout_1 (Dropout) (None, 16, 16, 32) 0
conv2d_3 (Conv2D) (None, 16, 16, 64) 18496
conv2d_4 (Conv2D) (None, 16, 16, 64) 36928
max_pooling2d_2 (MaxPooling2 (None, 8, 8, 64) 0
dropout_2 (Dropout) (None, 8, 8, 64) 0
conv2d_5 (Conv2D) (None, 8, 8, 128) 73856
conv2d_6 (Conv2D) (None, 8, 8, 128) 147584
max_pooling2d_3 (MaxPooling2 (None, 4, 4, 128) 0
dropout_3 (Dropout) (None, 4, 4, 128) 0
flatten_1 (Flatten) (None, 2048) 0
dense_1 (Dense) (None, 128) 262272
dropout_4 (Dropout) (None, 128) 0
dense_2 (Dense) (None, 10) 1290

Total params: 550,570
Trainable params: 550,570
Non-trainable params: 0

As bottlenecks, I was trying to use 'conv2d_6'
I can point you to the same google drive folder if you prefer.
tcav-cifar.zip

[yingji425]

Hi! @francescobodria I am not sure if you have already solved this problem.
I also met a similar problem a month ago, and I found my problem was caused by the loss part.
I used categorical_crossentropy but the label input for gradient calculation is not one-hot representation (it is just a index). So it failed as Check failed: NDIMS == new_sizes.size() (2 vs. 1). It cannot calculate the gradient.
Then I retrained my model with SparseCategoricalCrossentropy loss then it solved the problem.
I don't know if your problem is the same as mine. Maybe you can also check the input in tcav calcualtion part.
Just a little hint. Hope this can help you.

[BeenKim]

@francescobodria Let us know if your issue hasn't resolved and if so, please feel free to reopen this issue.