Input about triq

[junyongyou]

Hi, you didn't describe your problem clearly. The two scenarios are quite different.

1. If you input one image, and then use TRIQ and then do segmentation. I am not sure the segmentation will be done on what? TRIQ does not change the input image, it only calculates the quality value. Please clarify your question.

2. If you want to input two images. You need to modify the create_triq_model method. You need to define the other operations (e.g., backbone) on the two input images, and how to combine them ...

`def create_triq_model(n_quality_levels,
input_shape=(None, None, 3),
backbone='resnet50',
transformer_params=(2, 32, 8, 64),
maximum_position_encoding=193,
vis=False):
"""
Creates the hybrid TRIQ model
:param n_quality_levels: number of quality levels, use 5 to predict quality distribution
:param input_shape: input shape
:param backbone: bakbone nets, supports ResNet50 and VGG16 now
:param transformer_params: Transformer parameters
:param maximum_position_encoding: the maximal number of positional embeddings
:param vis: flag to visualize attention weight maps
:return: TRIQ model
"""
input_1 = Input(shape=input_shape)
#input_2 = your another image
if backbone == 'resnet50':
backbone_model = ResNet50(inputs,
return_feature_maps=False, return_last_map=True)
elif backbone == 'vgg16':
backbone_model = VGG16(inputs, return_last_map=True)
else:
raise NotImplementedError

C5 = backbone_model.output

dropout_rate = 0.1

transformer = TriQImageQualityTransformer(
    num_layers=transformer_params[0],
    d_model=transformer_params[1],
    num_heads=transformer_params[2],
    mlp_dim=transformer_params[3],
    dropout=dropout_rate,
    n_quality_levels=n_quality_levels,
    maximum_position_encoding=maximum_position_encoding,
    vis=vis
)
outputs = transformer(C5)

model = Model(inputs=[input_1, input_2], outputs=outputs)
model.summary()
return model` 

[Alen334]

`def create_triq_model(n_quality_levels,
input_shape=(None, None, 3),
backbone='resnet50',
transformer_params=(2, 32, 8, 64),
maximum_position_encoding=193,
vis=False):
"""
Creates the hybrid TRIQ model
:param n_quality_levels: number of quality levels, use 5 to predict quality distribution
:param input_shape: input shape
:param backbone: bakbone nets, supports ResNet50 and VGG16 now
:param transformer_params: Transformer parameters
:param maximum_position_encoding: the maximal number of positional embeddings
:param vis: flag to visualize attention weight maps
:return: TRIQ model
"""

inputs = Input(shape=input_shape)
m = inputs
x = tf.keras.layers.Lambda(tf.split, arguments={'axis': 2, 'num_or_size_splits': 2})(m)
input1 = x[0]
input2 = x[1]
print(input1.shape)
inputs1 = Input(shape=(input1.shape[1], input1.shape[2], input1.shape[3]))
inputs2 = Input(shape=(input2.shape[1], input2.shape[2], input2.shape[3]))
if backbone == 'resnet50':
    backbone_model1 = ResNet50(inputs1, return_feature_maps=False, return_last_map=True)
    backbone_model2 = ResNet50(inputs2, return_feature_maps=False, return_last_map=True)
elif backbone == 'vgg16':
    backbone_model1 = VGG16(inputs1, return_last_map=True)
    backbone_model2 = VGG16(inputs2, return_last_map=True)
else:
    raise NotImplementedError

C51 = backbone_model1.output
C52 = backbone_model2.output

dropout_rate = 0.1

transformer = TriQImageQualityTransformer(
    num_layers=transformer_params[0],
    d_model=transformer_params[1],
    num_heads=transformer_params[2],
    mlp_dim=transformer_params[3],
    dropout=dropout_rate,
    n_quality_levels=n_quality_levels,
    maximum_position_encoding=maximum_position_encoding,
    vis=vis
)

outputs1 = transformer(C51)
outputs2 = transformer(C52)
outputs = tf.concat([outputs1, outputs2], 0)
model = Model(inputs=inputs, outputs=outputs)
#model.summary()  # 输出模型参数
return model

`

oh, i'm sorry. it is just like this. The input data is [768, 2048, 3], and then divided into two sets of data, the size is [768, 1024, 3], and then put into the transformer respectively to get the output.

[junyongyou]

If you are using the fixed input size, you do not need to specify the input size as (None, None, 3). I don't see any problems in your script.

[Alen334]

Thank you very much for your patient reply and guidance, I think the problem may be in the model call. It probably works now.

[junyongyou]

Thank you very much for your patient reply and guidance, I think the problem may be in the model call. It probably works now.

Great to hear it works.