raminnakhli / decoupled-contrastive-learning Goto Github PK

Hi,

I saw your implementation of the nominator of DCLW weight_fn, which uses element multiplication for the z1 and z2 calculation.
But in the paper, the nominator of DCLW weight_fn formula is: exp(<z1, z2> / sigma). Can you tell me why you use element multiplication for the z1 and z2 calculation instead of torch.mm(z1, z2) or dot multiplication?

Thanks.

Hello, thanks for your excellent work. I noticed that your learning rate is fixed during training. Whey you do not apply a learning rate schedule?

Hi there,

Great work! I am trying to learn from the formula and walk through the proof of propositions. I was trying to find the derivatives of the loss function and encountered finding the partial derivative of the cosine similarity term. By comparing my answer and steps in the paper I kind of get this result (as the image shows).

I am not quite sure if that is correct and how that works. Would you mind help me a bit with this?

Can you tell me why exist loss with negative values? Is that a potential bug in your code?

May I ask whether DCL can be applied to SupCon？

Thank you for your work!
How to modify the loss to MoCo style.

I think the total loss of InfoNCE as well as DCL should be averaged.
$L=\sum_{k \in { 1,2}, i \in [1,N]}L_{i}^{(k)}$
should be
$L=\frac{1}{N}\sum_{k \in { 1,2}, i \in [1,N]}L_{i}^{(k)}$

and
$L_{DC}=\sum_{k \in { 1,2}, i \in [1,N]}L_{DC,i}^{(k)}$
should be
$L_{DC}=\frac{1}{N}\sum_{k \in { 1,2}, i \in [1,N]}L_{DC,i}^{(k)}$

the code does this

return (positive_loss + negative_loss).mean()

Hello, Thanks for your good work! I'd like to know if the cosine annealing schedule is also applied to the small-scale dataset experiments of CIFAR and STL10?

Hi,

I'm trying to find an official source for the Imagenet-100 dataset used in this paper.

Regards

In DCLW, your code is like:

weight_fn = lambda z1, z2: 2 - z1.size(0) * torch.nn.functional.softmax((z1 * z2).sum(dim=1) / sigma, dim=0).squeeze()

I think the right way shall be like:

weight_fn = lambda z1, z2: 2 - torch.nn.functional.softmax((z1 * z2).sum(dim=1) / sigma, dim=0).squeeze()

z1.size(0) is not a variable introduced in the origin paper.

What do you think of it?

I wonder in the proof of Proposition 1. If there should include a (2)?

    Hi @raminnakhli Thanks for the reply. 

As you can see from the two pictures, the formulas are similar which all include: exp(<z_1, z_2>). But in your code, for formula(5) (6) you use matrix multiplication, and for the formula of w(z_1, z_2), you use element multiplication.

Could you please explain to me why? Thanks!

Originally posted by @wqtwjt1996 in #10 (comment)