ssarcandy / deepcoral

🧠 A PyTorch implementation of 'Deep CORAL: Correlation Alignment for Deep Domain Adaptation.', ECCV 2016

I am Andy Hsu, graduated student of CMLab, National Taiwan University.
Interested in programming and actively share various projects on GitHub, trying to make the world better (or just make my life easier).
Learn more about me on my blog 👉 https://ssarcandy.tw/

deepcoral's Issues

Why it's loss = loss/(4*d*4) ?

I noticed in the Deep CORAL paper that the CORAL loss is calculated by
loss = loss / (4 * d * d)
But line 26 of models.py defines the CORAL loss as
loss = loss/(4*d*4)
I do not understand. Is there a mistake?

I think you have a error.

DEEP_CORAL_LOSS:
def CORAL(source, target):
d = source.data.shape[1]
ns = source.data.shape[0]
nt = target.data.shape[0]

# source covariance
xm = torch.mean(source, 0, keepdim=True) - source
xc = (xm.t() @ xm) / (ns-1)

# target covariance
xmt = torch.mean(target, 0, keepdim=True) - target
xct = xmt.t() @ xmt / (nt-1)

print(xc, xct)

# frobenius norm between source and target
loss = torch.sum(torch.mul((xc - xct), (xc - xct)))
loss = loss/(4*d*d)
return loss

Graph

Can you provide the script to get the graph please?

self.target_fc never used

Hello, in models DeepCoral you declare a target_fc but it seems to me it's never used:

class DeepCORAL(nn.Module):
    def __init__(self, num_classes=1000):
        super(DeepCORAL, self).__init__()
        self.sharedNet = AlexNet()
        self.source_fc = nn.Linear(4096, num_classes)
        self.target_fc = nn.Linear(4096, num_classes)

        # initialize according to CORAL paper experiment
        self.source_fc.weight.data.normal_(0, 0.005)
        self.target_fc.weight.data.normal_(0, 0.005)

    def forward(self, source, target):
        source = self.sharedNet(source)
        source = self.source_fc(source)

        target = self.sharedNet(target)
        target = self.source_fc(target)
        return source, target

Am I missing something?

TypeError: unsupported operand type(s) for /: 'tuple' and 'int'

I'm sorry i met the following error using the office31 dataset

[INFO] Loading datasets: amazon
[INFO] Loading datasets: webcam
Traceback (most recent call last):
File "main.py", line 151, in
res = train(model, optimizer, e+1, _lambda)
File "main.py", line 28, in train
source, target = list(enumerate(source_loader)), list(enumerate(target_loader))
File "/home/zjp/anaconda3/envs/pytorch0.2/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 201, in next
return self._process_next_batch(batch)
File "/home/zjp/anaconda3/envs/pytorch0.2/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 221, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
TypeError: Traceback (most recent call last):
File "/home/zjp/anaconda3/envs/pytorch0.2/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 40, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/zjp/anaconda3/envs/pytorch0.2/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 40, in
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/zjp/anaconda3/envs/pytorch0.2/lib/python3.6/site-packages/torchvision/datasets/folder.py", line 67, in getitem
img = self.transform(img)
File "/home/zjp/anaconda3/envs/pytorch0.2/lib/python3.6/site-packages/torchvision/transforms.py", line 29, in call
img = t(img)
File "/home/zjp/anaconda3/envs/pytorch0.2/lib/python3.6/site-packages/torchvision/transforms.py", line 139, in call
ow = int(self.size * w / h)
TypeError: unsupported operand type(s) for /: 'tuple' and 'int'

Error in calculation of covariance matrix

I think that there's an error in the calculation of covariance matrices. In the line xm = torch.mean(source, 1, keepdim=True) - source, imo 1 must be replaced by 0, as I would like to calculate the mean of each feature along the entire batch. Please correct me if I am wrong.
Thanks, wonderful work by the way.

Error !!!

I run successfully python3 main.py but when I test, then got error. Can you fix it please?

~$ python -m DeepCORAL.tests.test -v

(pytorch_python3) user@PA00074589:~$ python -m DeepCORAL.tests.test -v
test_CORAL_backward (main.TestCORAL) ... ERROR
test_CORAL_forward (main.TestCORAL) ... ERROR
test_feature_covariance_mat (main.TestCORAL) ... ERROR
test_forbenius_norm (main.TestCORAL) ... ok

======================================================================
ERROR: test_CORAL_backward (main.TestCORAL)

Traceback (most recent call last):
File "/home/user/DeepCORAL/tests/test.py", line 37, in test_CORAL_backward
valid = torch.autograd.gradcheck(coral, test[x], eps=1e-3)
File "/home/user/pytorch_python3/lib/python3.5/site-packages/torch/autograd/gradcheck.py", line 154, in gradcheck
output = func(*inputs)
File "/home/user/DeepCORAL/models.py", line 31, in forward
cs = feature_covariance_mat(ns, source)
File "/home/user/DeepCORAL/models.py", line 13, in feature_covariance_mat
tmp = ones_t.matmul(d)
File "/home/user/pytorch_python3/lib/python3.5/site-packages/torch/tensor.py", line 180, in matmul
return torch.matmul(self, other)
File "/home/user/pytorch_python3/lib/python3.5/site-packages/torch/functional.py", line 173, in matmul
return torch.mm(tensor1, tensor2)
TypeError: torch.mm received an invalid combination of arguments - got (torch.cuda.FloatTensor, torch.FloatTensor), but expected one of:

• (torch.cuda.FloatTensor source, torch.cuda.FloatTensor mat2)
  didn't match because some of the arguments have invalid types: (torch.cuda.FloatTensor, torch.FloatTensor)
• (torch.cuda.sparse.FloatTensor source, torch.cuda.FloatTensor mat2)
  didn't match because some of the arguments have invalid types: (torch.cuda.FloatTensor, torch.FloatTensor)

======================================================================
ERROR: test_CORAL_forward (main.TestCORAL)

Traceback (most recent call last):
File "/home/user/DeepCORAL/tests/test.py", line 29, in test_CORAL_forward
res = coral.forward(*test[x])
File "/home/user/DeepCORAL/models.py", line 31, in forward
cs = feature_covariance_mat(ns, source)
File "/home/user/DeepCORAL/models.py", line 13, in feature_covariance_mat
tmp = ones_t.matmul(d)
File "/home/user/pytorch_python3/lib/python3.5/site-packages/torch/tensor.py", line 180, in matmul
return torch.matmul(self, other)
File "/home/user/pytorch_python3/lib/python3.5/site-packages/torch/functional.py", line 173, in matmul
return torch.mm(tensor1, tensor2)
TypeError: torch.mm received an invalid combination of arguments - got (torch.cuda.FloatTensor, torch.FloatTensor), but expected one of:

• (torch.cuda.FloatTensor source, torch.cuda.FloatTensor mat2)
  didn't match because some of the arguments have invalid types: (torch.cuda.FloatTensor, torch.FloatTensor)
• (torch.cuda.sparse.FloatTensor source, torch.cuda.FloatTensor mat2)
  didn't match because some of the arguments have invalid types: (torch.cuda.FloatTensor, torch.FloatTensor)

======================================================================
ERROR: test_feature_covariance_mat (main.TestCORAL)

Traceback (most recent call last):
File "/home/user/DeepCORAL/tests/test.py", line 20, in test_feature_covariance_mat
res = feature_covariance_mat(*test[x])
File "/home/user/DeepCORAL/models.py", line 13, in feature_covariance_mat
tmp = ones_t.matmul(d)
File "/home/user/pytorch_python3/lib/python3.5/site-packages/torch/tensor.py", line 180, in matmul
return torch.matmul(self, other)
File "/home/user/pytorch_python3/lib/python3.5/site-packages/torch/functional.py", line 173, in matmul
return torch.mm(tensor1, tensor2)
TypeError: torch.mm received an invalid combination of arguments - got (torch.cuda.FloatTensor, torch.FloatTensor), but expected one of:

• (torch.cuda.FloatTensor source, torch.cuda.FloatTensor mat2)
  didn't match because some of the arguments have invalid types: (torch.cuda.FloatTensor, torch.FloatTensor)
• (torch.cuda.sparse.FloatTensor source, torch.cuda.FloatTensor mat2)
  didn't match because some of the arguments have invalid types: (torch.cuda.FloatTensor, torch.FloatTensor)

---

Ran 4 tests in 1.179s

FAILED (errors=3)

Test accuracy is low than original test accuracy in the paper.

Test accuracy is low than original test accuracy in the paper. Can you fix it please?

Inputs to the CORAL loss

    out1, out2 = model(source_data, target_data)

    classification_loss = torch.nn.functional.cross_entropy(out1, source_label)
    coral = models.CORAL()
    coral_loss = coral(out1, out2)

    sum_loss = _lambda*coral_loss + classification_loss

From your codes, the input to the CORAL loss function is the prediction. Although it seems the case in the original DeepCORAL paper, it's not really reasonable to me. To my understanding, the main goal of domain adaptation is to match the feature distribution between source and target domain, so I think it makes more sense to calculate the loss function using the feature vectors from previous layers (e.g. fc6, fc7).
In addition, in some famous DA works:
https://arxiv.org/pdf/1412.3474.pdf
http://proceedings.mlr.press/v37/ganin15.pdf
https://arxiv.org/pdf/1502.02791.pdf
They all have loss functions for those layers. Maybe you can try different positions of the CORAL loss function and see whether it improve the performance or not.
Thanks.

The reason for setting _lambda ascending ?

I just run the code using default setting of weight of _lambda , and i observe very tiny difference between testing accuracy of source and target. In the other hand, fixed value of 1 shows better result.

result of ascending _lambda ( _lambda = (e+1)/EPOCHS)

result of descending _lambda ( _lambda = (EPOCHS-e)/EPOCHS)

result of fixed _lambda ( _lambda = 1)

Correlation Loss

From my understanding, we know that tensorflow and pytorch can calculate gradients automatically. In original coral implementation, they used caffe which cannot calculate gradients automatically and thats why they used back-propagation. So can we use simply the following correlation loss in pytorch as well? (https://github.com/tensorflow/models/blob/master/research/domain_adaptation/domain_separation/losses.py)

def correlation_loss(source_samples, target_samples):
    source_samples -= tf.reduce_mean(source_samples, 0)
    target_samples -= tf.reduce_mean(target_samples, 0)
    source_samples = tf.nn.l2_normalize(source_samples, 1)
    target_samples = tf.nn.l2_normalize(target_samples, 1)
    source_cov = tf.matmul(tf.transpose(source_samples), source_samples)
    target_cov = tf.matmul(tf.transpose(target_samples), target_samples)
    corr_loss = tf.reduce_mean(tf.square(source_cov - target_cov))
    return corr_loss

Any update results like Deep CORAL paper ?

Pretrained model

In your code you are using pretrained model. For an example, I will not use pretrained model, then what can I do?
I am writing a data loader code. Hopefully, I will get good accuracy as paper and then I will send pull request.

Please help me so that I can use your code without pretrained model.

Thanks in advanced.

code about testing

Hello,
I have some questions about your testing codes in main.py:
1.

# sum up batch loss
test_loss += torch.nn.functional.cross_entropy(out, target, size_average=False).data[0]
# get the index of the max log-probability
pred = out.data.max(1, keepdim=True)[1]
correct += pred.eq(target.data.view_as(pred)).cpu().sum()

why don't you directly calculate the loss as you do for training
like this: classification_loss = torch.nn.functional.cross_entropy(out1, source_label)
Instead, you sum up the loss then divide by the times you sum up.
Is there any reason for that?

out1, out2 = model(data, data)
out = out1 if mode == 'source' else out2

This part looks a little bit weird to me. Please correct me if I am wrong.
out1 is the output from forwarding data to the source branch, and out2 is the output from forwarding data to the target branch.
I thought we still use source branch for testing since the parameters learned in the source branch will be affected by the target branch through the CORAL loss. My concern is that the target branch is not trained using the source labels, so I am not sure whether it will produce the correct labels for the testing data.
It would be great if you can clarify my concern, or you can show the result forwarding the testing data through the source branch.
Thank you!!!

The point of self.target_fc?

Hi,

I was using the code from models.py to try and improve transfer learning results between two datasets. It made me wonder what the point of self.target_fc is in the DeepCORAL class since it is never actually used in the forward pass. Instead, `self.source_fc' is applied to both the source and target data.

Also, in main.py the test function has a mode argument which selects the output to return. Given the fact that it calls model(data, data) and that there is no difference between how these two inputs will be processed, is such indexing redundant?

The CORAL Loss and Forbenius Norm is defined differently and the results dont match.

There is a difference between the CORAL Loss used in the paper and the one described in this code. Is it because they both are giving the same result ? The Forbenius Norm is also different.

The results were also different from the ones in the paper. Is there any idea why the results have changed?

Any response would be helpful.

A line in main.py().

What is the use of model.train() in line 25 of main.py ? Is it not redundant ?

Low Accuracy for SHVN ---> MNIST

I got this accuracy for SHVN -----> MNIST

###Test Source: Epoch: 2460, avg_loss: 0.0003, Accuracy: 73255/73257 (100.00%)
###Test Target: Epoch: 2460, avg_loss: 21.5493, Accuracy: 35242/60000 (58.74%)

That means data loading does not matter to get good accuracy. Because for SHVN---->MNIST, I just use default data processing.

CORAL loss is defined differently from the original paper

I noticed that both the covariance and the Frobenius norm are computed differently in your implementation.

You compute the Frobenius norm as below:
# frobenius norm between source and target
loss = torch.mean(torch.mul((xc - xct), (xc - xct)))
However as stated here http://mathworld.wolfram.com/FrobeniusNorm.html , after squaring each element and summing them, should be computed the square root of the sum not the mean of the squared elements.

In the original paper the covariances are computed as below :
https://arxiv.org/abs/1607.01719

While in your implementation:

# source covariance 
xm = torch.mean(source, 0, keepdim=True) - source
xc = xm.t() @ xm

# target covariance
xmt = torch.mean(target, 0, keepdim=True) - target
xct = xmt.t() @ xmt