titu1994 / keras-non-local-nets

Keras implementation of Non-local Neural Networks

Keras Non-Local Neural Networks

Keras implementation of Non-local blocks from [1].

• Support for "Gaussian", "Embedded Gaussian" and "Dot" instantiations of the Non-Local block.
• Support for variable shielded computation mode (reduces computation by N**2 x, where N is default to 2)
• Support for "Concatenation" instantiation will be supported when authors release their code.

Usage Templates

The script non_local.py contains the NonLocalBlock instance which takes in an input tensor and wraps a non-local block around it.

from non_local import NonLocalBlock
from tensorflow.keras.layers import Input, Conv1D, Conv2D, Conv3D

ip = Input(shape=(...))  # input tensor with an "N" rank order of 3, 4 or 5
x = ConvND(...)         # convolution operation with aforementioned rank 
...
non_local_block = NonLocalBlock(intermediate_dim=None, compression=2, mode='embedded', add_residual=True)
x = non_local_block(x)
...

The script non_local_layerstyle.py contains the NonLocalBlock layer which takes in an input tensor and wraps a non-local block around it. Made to facilitate the neural network builder using the Sequential method.

from non_local_layerstyle import NonLocalBlock
from tensorflow.keras.layers import Input, Conv1D, Conv2D, Conv3D

# Define the input shape
input_shape = (...)  # shape of input tensor

model = Sequential()
model.add(ConvND(...))         # convolution operation with an "N" rank order of 3, 4 or 5
...
model.add(NonLocalBlock(intermediate_dim=None, compression=2, mode='embedded', add_residual=True))
...

Basic block

From [1], a basic Non-Local block with the Embedded Gaussian instantiation has the below logic:

1. Xiaolong Wang, Ross Girshick, Abhinav Gupta, Kaiming He. "Non-local Neural Networks." arXiv:1711.07971 [cs.CV], 21 Nov 2017. Link

keras-non-local-nets's Issues

Arbitrary Style Transfer with Style-Attentional Networks

How about implementing this neural network: https://arxiv.org/abs/1812.02342v3? It is very similar to the one presented here. I've contacted the authors - they don't plan to post the code any time soon.

Are there any matrix multiplication?

In the published paper and your code, I saw dot product in many place,
But as far as I understand from Figure 2 in paper, it is matrix multiplication rather than a dot product,
there should be such a procedure for part of "embedded gaussian"

1. reshape HW512 to HW*512
2. transpose one output into 512*HW
3. apply matrix multiplication
   (HW512) * (512HW) = HW*HW
   Are there anything wrong with this?
   Thank you for your reply!

Is training and inference scripts available?

How extensively detailed this implementation is! However, it is better to be shipped with training and inference scripts. Could you please share your training and inference experience? Thank you!

Can I use this model for video classification?

@titu1994 Thank you for your code! I want to use the non-local resnet model for video classification, as in the paper. The author emphasizes that the convolution operation should be 3-d convolution, but I can't see it in your nonlocal_resnet.py file. So I think this model doesn't fit video classification task, am I right?

Running Error: ValueError: Tried to convert 'shape' to a tensor and failed.

When i import the non_local_block in my own keras code, it happens
ValueError: Tried to convert 'shape' to a tensor and failed. Error: None values not supported.

Here is my code review:
def init_model():
im_n = Input(shape=(None, None, 1))
x = Conv2D(64, 3, activation='relu', padding='same')(im_n)
x = **non_local_block**(x)
resd = Conv2D(1, 3, padding='same')(x)
model = Model(inputs=im_n, outputs=resd)
model.compile(optimizer=Adam(lr=0.001, beta_1=0.9,
beta_2=0.999, epsilon=1e-8),
loss='mean_squared_error')
return model

Here is the ERROR INFO:
Traceback (most recent call last):
File "/Users/code/train.py", line 49, in <module>
model = init_model()
File "/Users/code/train.py", line 35, in init_model
x = non_local_block(x)
File "/Users/code/non_local.py", line 136, in non_local_block
y = Reshape((dim1, dim2, intermediate_dim))(y)
File "/Users/anaconda3/envs/tf-keras/lib/python3.6/site-packages/keras/engine/base_layer.py", line 457, in __call__
output = self.call(inputs, **kwargs)
File "/Users/anaconda3/envs/tf-keras/lib/python3.6/site-packages/keras/layers/core.py", line 401, in call
return K.reshape(inputs, (K.shape(inputs)[0],) + self.target_shape)
File "/Users/anaconda3/envs/tf-keras/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 1969, in reshape
return tf.reshape(x, shape)
File "/Users/anaconda3/envs/tf-keras/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 6482, in reshape
"Reshape", tensor=tensor, shape=shape, name=name)
File "/Users/anaconda3/envs/tf-keras/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 528, in _apply_op_helper (input_name, err))
ValueError: Tried to convert 'shape' to a tensor and failed. Error: None values not supported.

Thx!!!

Using a Non-Local Block significantly reduces accuracy.

Below is my code:

ResNet50NLBase = NonLocalResNet(
      input_shape=(IMAGE_SIZE, IMAGE_SIZE, 3),
      classes=6,
      include_top=False,
      repetitions=[3, 4, 6, 3]
  )
  ResNet50NL = Model(inputs=ResNet50NLBase.inputs, outputs=ResNet50NLBase.layers[-2].output)

  #model.add(RN50)
  model.add(ResNet50NL)
  model.add(GlobalAveragePooling2D())
  model.add(Dense(6, activation='softmax'))

  model.compile(
      loss='categorical_crossentropy',
      optimizer=Adam(),
      metrics=['categorical_accuracy',
               tfa.metrics.CohenKappa(num_classes=6, sparse_labels=False, weightage="quadratic")]
  )

This has significantly reduced accuracy compared to:

RN50 = tf.keras.applications.resnet50.ResNet50(
    input_shape=(IMAGE_SIZE, IMAGE_SIZE, 3),
    weights='imagenet',
    include_top=False,
    pooling='avg'
)
RN50 = Model(inputs=RN50.inputs, outputs=RN50.layers[-2].output)```
Can you explain this?

ValueError: None values not supported

ValueError: None values not supported.
I missed the problem in rows 136,which is "y = Reshape((dim1, dim2, intermediate_dim))(y)",it means we can't reshape the tensor to [None,None,None,128] from the original tensor with shape [None,None,128].I have been in the trouble for a few days,is there any one can help me solve this problem?Thanks for your help!My keras is 2.1.5 and tensorflow is 1.8.

Reproducing the paper result

I wonder if you have successfully reproduced the numbers (accuracy) reported in the paper

Error in YOLOv3

HI,
Thanks for your code because your code can improve my work in YOLOv2 but I have error in YOLOv3. Could anyone help me ?
I put non local block in this part

Layer 80 => 82

pred_yolo_1 = _conv_block(x, [{'filter': 512, 'kernel': 3, 'stride': 1, 'bnorm': True, 'dilation_rate': 2,  'leaky': True,  'layer_idx': 80},
                         {'filter': (3*(5+nb_class)), 'kernel': 1, 'stride': 1, 'bnorm': False, 'leaky': False, 'layer_idx': 81}], do_skip=False)
loss_yolo_1 = YoloLayer(anchors[12:], 
                        [1*num for num in max_grid], 
                        batch_size, 
                        warmup_batches, 
                        ignore_thresh, 
                        grid_scales[0],
                        obj_scale,
                        noobj_scale,
                        xywh_scale,
                        class_scale)([input_image, pred_yolo_1, true_yolo_1, true_boxes])

# Layer 83 => 86
x = _conv_block(x, [{'filter': 512, 'kernel': 1, 'stride': 1, 'bnorm': True, 'leaky': True, 'layer_idx': 84}], do_skip=False)

x = UpSampling2D(2)(x)
# x = concatenate([x, skip_61])

# 1*1 conv for addition
skip_61 = Conv2D(512, (1, 1), strides=(1,1), padding='same', name='add_conv_21', use_bias=False)(skip_61)
skip_61 = BatchNormalization(name='add_norm_21')(skip_61)
skip_61 = LeakyReLU(alpha=0.1)(skip_61)

#non-local block
skip_61 = non_local_block(skip_61, mode='embedded', compression=2) 

and the result is

Question

Hello,thanks for your code. When I use the non local module in my model, I am always reminded that my graphics card is out of memory. In fact, when the module is removed, the model can be trained normally. Does this module need to occupy a lot of memory?

`IndexError: tuple index out of range` at line 152 of nonlocal_resnet.py

I tried to use NonLocalResNet50:

model = NonLocalResNet50(input_shape=(224, 224, 3), classes=340)

But got the following error:

IndexError: tuple index out of range

It's raised from line 152 of nonlocal_resnet.py:

    150     x = block_function(filters=filters, stage=stage, block=i,
    151                        transition_strides=transition_strides[i],
--> 152                        dilation_rate=dilation_rates[i],
    153                        is_first_block_of_first_layer=(is_first_layer and i == 0),
    154                        dropout=dropout,

Tensorflow

Can you update this to be compatible with the latest version of Tensorflow?

compilation error

there is a compilation error when i tried to use NonLocalResNet50((224, 224, 3), classes=10).
need to make changes as below

diff --git a/nonlocal_resnet.py b/nonlocal_resnet.py
index 0e7d43e..c9eca02 100644
--- a/nonlocal_resnet.py
+++ b/nonlocal_resnet.py
@@ -149,7 +149,7 @@ def _residual_block(block_function, filters, blocks, stage,
         for i in range(blocks):
             x = block_function(filters=filters, stage=stage, block=i,
                                transition_strides=transition_strides[i],
-                               dilation_rate=dilation_rates[i],
+                               dilation_rate=transition_dilation_rates[i],
                                is_first_block_of_first_layer=(is_first_layer and i == 0),
                                dropout=dropout,
                                residual_unit=residual_unit)(x)
@@ -457,5 +457,5 @@ def NonLocalResNet152(input_shape, classes):


 if __name__ == '__main__':
-    model = NonLocalResNet18((32, 32, 3), classes=10)
-    model.summary()
\ No newline at end of file
+    model = NonLocalResNet50((224, 224, 3), classes=10)
+    model.summary()

Integrate a non local block with Mask R-CNN?

Hi, the original paper tested a Mask R-CNN with a non-local block (by adding one non-local block
right before the last residual block of res4 ).

I want to use your non-local block for object detection along with Mask R-CNN ( say https://github.com/matterport/Mask_RCNN )

How do I go about this?

Hello, how to use this program? Where should I put it in my code?

Small bug about NonLocalResNet34/50/.... ？

when use depth>18, will have a bug.
In _redidual_block:
def _residual_block(block_function, filters, blocks, stage,
transition_strides=None, transition_dilation_rates=None,
dilation_rates=None, is_first_layer=False, dropout=None,
residual_unit=_bn_relu_conv):
if transition_dilation_rates is None:
transition_dilation_rates = [(1, 1)] * blocks
if transition_strides is None:
transition_strides = [(1, 1)] * blocks
if dilation_rates is None:
dilation_rates = [(1, 1)] * blocks
Adding the last two lines of code above is necessary.

Weight sharing

In the paper when adding multiple nonlocal blocks, I think the weight W_g in the affinity function f is shared for each nonlocal block. Is it true? If it's true, how to you implement this in your codes? Thanks