OREPA_LargeConvBase result is wrong about orepa_cvpr2022

[JUGGHM]

Hi, When I was reproducing OREPA_LargeConvBase, I found some results that were different from what I expected. Could you tell me the reason for this result?

import torch
from torch import nn as nn
import torch.nn.functional as F
import torch.nn.init as init
import math

weight = nn.Parameter(torch.Tensor(128, 64, 3, 3))
weight1 = nn.Parameter(torch.Tensor(128, 128, 3, 3))
weight2 = nn.Parameter(torch.Tensor(64, 128, 3, 3))

init.kaiming_uniform_(weight, a=math.sqrt(5))
init.kaiming_uniform_(weight1, a=math.sqrt(5))
init.kaiming_uniform_(weight2, a=math.sqrt(5))

rep_weight = weight.transpose(0, 1)
rep_weight = F.conv2d(rep_weight, weight1, groups=1, padding=2)
rep_weight = F.conv2d(rep_weight, weight2, groups=1, padding=2)
rep_weight = rep_weight.transpose(0, 1)

data = torch.randn((1, 64, 1080, 1920)) * 255
conv_result = F.conv2d(F.conv2d(F.conv2d(data, weight=weight, padding=1), weight=weight1, padding=1), weight=weight2, padding=1)
rep_result = F.conv2d(input=data, weight=rep_weight, bias=None, stride=1, padding=3)

diff = torch.abs(rep_result - conv_result)
print(f"max diff: {diff.max()}")
print(f"median diff: {diff.median()}")
print(f"mean diff: {diff.mean()}")


# max diff: 365.46533203125
# median diff: 44.0756950378418
# mean diff: 52.17301559448242

Thanks for your interest jhl13! There are mainly two reasons but both are inside our expectation.

1. I guess that the 180 degree flip in the Figure 4 of the paper is missing in your code. Therefore we need to modify the code:

rep_weight = F.conv2d(rep_weight, torch.flip(weight1, [2,3]), groups=1, padding=2)
rep_weight = F.conv2d(rep_weight, torch.flip(weight2, [2,3]), groups=1, padding=2)

2. There is a boundary effect which we didn't discuss in the paper. Let's consider a simpler case:

import torch
from torch import nn as nn
import torch.nn.functional as F
import torch.nn.init as init
import math

weight = nn.Parameter(torch.Tensor(2, 1, 3, 3))
weight1 = nn.Parameter(torch.Tensor(3, 2, 3, 3))

init.kaiming_uniform_(weight, a=math.sqrt(5))
init.kaiming_uniform_(weight1, a=math.sqrt(5))

rep_weight = F.conv2d(weight.transpose(0, 1), torch.flip(weight1, [2,3]), groups=1, padding=2).transpose(0, 1)

data = torch.ones((1, 1, 5, 5)) * 255
conv_result = F.conv2d(F.conv2d(data, weight=weight, padding=1), weight=weight1, padding=1)
rep_result = F.conv2d(input=data, weight=rep_weight, bias=None, stride=1, padding=2)

And there is difference between the boundaries (! but not the central regions) of the conv_result and rep_result:
The first channel of conv result:

[ -19.1563,  -16.8371,  -23.4708,  -17.2442,  -26.5005],
[  11.5157,   -0.2203,   19.0002,   26.5831,  -18.4257],
[  18.2313,   -9.1425,    2.4204,   23.9011,  -51.2190],
[   6.9783,  -17.4167,  -16.3604,    9.9677,  -51.4312],
[  16.0287,    8.8003,   16.2718,   40.8220,  -19.1323]

The first channel of rep result:

[-27.1100,    2.0168,    3.8379,    6.3456,  -18.9813],
[  -2.2159,   -0.2203,   19.0002,   26.5831,   -1.8732],
[ -27.9867,   -9.1425,    2.4205,   23.9011,  -51.1907],
[ -34.7375,  -17.4167,  -16.3604,    9.9677,  -62.1802],
[ -34.7501,   -3.3232,   -6.3288,   23.1935,  -49.2377]

This is because when executing the standard two-step convolution, during the second convolution the boundary of the first-conv result tensor is involved. However, when conducting a one-pass re-parameterized convolution, the involve boundaries is actually zero. So the conclusion is that a stacked three 3x3 convolutional structural is not exactly the same as a 7x7 one. For details about the role zero padding plays in neural networks, you could refer to this paper.

[jhl13]

Hi, When I was reproducing OREPA_LargeConvBase, I found some results that were different from what I expected. Could you tell me the reason for this result?

import torch
from torch import nn as nn
import torch.nn.functional as F
import torch.nn.init as init
import math

weight = nn.Parameter(torch.Tensor(128, 64, 3, 3))
weight1 = nn.Parameter(torch.Tensor(128, 128, 3, 3))
weight2 = nn.Parameter(torch.Tensor(64, 128, 3, 3))

init.kaiming_uniform_(weight, a=math.sqrt(5))
init.kaiming_uniform_(weight1, a=math.sqrt(5))
init.kaiming_uniform_(weight2, a=math.sqrt(5))

rep_weight = weight.transpose(0, 1)
rep_weight = F.conv2d(rep_weight, weight1, groups=1, padding=2)
rep_weight = F.conv2d(rep_weight, weight2, groups=1, padding=2)
rep_weight = rep_weight.transpose(0, 1)

data = torch.randn((1, 64, 1080, 1920)) * 255
conv_result = F.conv2d(F.conv2d(F.conv2d(data, weight=weight, padding=1), weight=weight1, padding=1), weight=weight2, padding=1)
rep_result = F.conv2d(input=data, weight=rep_weight, bias=None, stride=1, padding=3)

diff = torch.abs(rep_result - conv_result)
print(f"max diff: {diff.max()}")
print(f"median diff: {diff.median()}")
print(f"mean diff: {diff.mean()}")


# max diff: 365.46533203125
# median diff: 44.0756950378418
# mean diff: 52.17301559448242

Thanks for your interest jhl13! There are mainly two reasons but both are inside our expectation.

1. I guess that the 180 degree flip in the Figure 4 of the paper is missing in your code. Therefore we need to modify the code:

rep_weight = F.conv2d(rep_weight, torch.flip(weight1, [2,3]), groups=1, padding=2)
rep_weight = F.conv2d(rep_weight, torch.flip(weight2, [2,3]), groups=1, padding=2)

2. There is a boundary effect which we didn't discuss in the paper. Let's consider a simpler case:

import torch
from torch import nn as nn
import torch.nn.functional as F
import torch.nn.init as init
import math

weight = nn.Parameter(torch.Tensor(2, 1, 3, 3))
weight1 = nn.Parameter(torch.Tensor(3, 2, 3, 3))

init.kaiming_uniform_(weight, a=math.sqrt(5))
init.kaiming_uniform_(weight1, a=math.sqrt(5))

rep_weight = F.conv2d(weight.transpose(0, 1), torch.flip(weight1, [2,3]), groups=1, padding=2).transpose(0, 1)

data = torch.ones((1, 1, 5, 5)) * 255
conv_result = F.conv2d(F.conv2d(data, weight=weight, padding=1), weight=weight1, padding=1)
rep_result = F.conv2d(input=data, weight=rep_weight, bias=None, stride=1, padding=2)

And there is difference between the boundaries (! but not the central regions) of the conv_result and rep_result: The first channel of conv result:

[ -19.1563,  -16.8371,  -23.4708,  -17.2442,  -26.5005],
[  11.5157,   -0.2203,   19.0002,   26.5831,  -18.4257],
[  18.2313,   -9.1425,    2.4204,   23.9011,  -51.2190],
[   6.9783,  -17.4167,  -16.3604,    9.9677,  -51.4312],
[  16.0287,    8.8003,   16.2718,   40.8220,  -19.1323]

The first channel of rep result:

[-27.1100,    2.0168,    3.8379,    6.3456,  -18.9813],
[  -2.2159,   -0.2203,   19.0002,   26.5831,   -1.8732],
[ -27.9867,   -9.1425,    2.4205,   23.9011,  -51.1907],
[ -34.7375,  -17.4167,  -16.3604,    9.9677,  -62.1802],
[ -34.7501,   -3.3232,   -6.3288,   23.1935,  -49.2377]

This is because when executing the standard two-step convolution, during the second convolution the boundary of the first-conv result tensor is involved. However, when conducting a one-pass re-parameterized convolution, the involve boundaries is actually zero. So the conclusion is that a stacked three 3x3 convolutional structural is not exactly the same as a 7x7 one. For details about the role zero padding plays in neural networks, you could refer to this paper.

Thank you very much for such a detailed answer！
After adding the 180 degree flip, the result is correct.

But in your project, does the OREPA_LargeConvBase implementation not add the 180 degree flip?

[JUGGHM]

This does not matter because all elements in the parameter tensor are initialized without spatial bias, so it will have no impact on final performance. If you are not feeling good, you could regard the latter kernels as the flipped ones.

[jhl13]

This does not matter because all elements in the parameter tensor are initialized without spatial bias, so it will have no impact on final performance. If you are not feeling good, you could regard the latter kernels as the flipped ones.

Got it, thank you!