dequanwang / tent Goto Github PK

Hi, thanks for sharing your great work. And the current repository only contains example code to illustrate how tent works. I am wondering if you will share the code to exactly reproduce results on ImageNet-C or some implementation details if the code is not available. Because it seems that tent is very sensitive to choices of some hyperparameters.

According to the code (and assuming that STEPS = 1, i dont understand how the outputs change after the adaptation:

def forward(self, x):
    if self.episodic:
        self.reset()

    for _ in range(self.steps):
        outputs = forward_and_adapt(x, self.model, self.optimizer)
    return outputs

@torch.enable_grad()  # ensure grads in possible no grad context for testing
def forward_and_adapt(x, model, optimizer):
    """Forward and adapt model on batch of data.

    Measure entropy of the model prediction, take gradients, and update params.
    """
    # forward
    outputs = model(x)
    # adapt
    loss = softmax_entropy(outputs).mean(0)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    return outputs

judging by the code, you return the original outputs however they do change somehow, how?

In table 2 for the TENT paper, there's RG, UDA-SS, TTT and PL, but they are not implemented in the current repo. Which implementations were used for the above methods?

Hi, thanks for sharing your great work. Recently, I try to reproduce your experiments on digit 5 with a simple network. However, it doesn't work. Could you please also share the code used for the source-free UDA experiments?

Hello! First of all, thank you for sharing your great work

When I run example--WRN-28-10, I find that: url_base ,which is = "https://docs.google.com/uc?export=download", can't be accessed. Is the link invalid？

If it is because the url is invalid, could you upload it again?

thank you very much!

In the paper, the backbone network for the experiments on cifar10 and cifar100 is R-26, but the implementation is WRN-28-10, could the authors provide the R-26 experiment code? I hope the authors provide more details about how to implement the R-26 and the training setup. Thanks very much!

Hi @DequanWang ,

Thanks for you open source the code firstly, I try to use the model as my paper baseline, but the training loss value is always 0.

My training script code following,

self.model.train()
self.model = tent.configure_model(self.model)
params, _ = tent.collect_params(self.model)
self.optimizer = optim.Adam(params, lr=3e-4)
tent_model = tent.Tent(self.model, self.optimizer)

for idx, data in enumerate(self.train_loader):
        image = data['img'].type(torch.FloatTensor).cuda(non_blocking=True)

        output, loss = tent_model(image)

It's worth mentioning that this happens whether I load the model or not.

Looking Forward to Hearing from You.

Hello!
I have a problem understanding the code when it is online (episodic) and step=1.
The output of the old model(before adaptation) is used for both backprop and accuracy calculation.
So the online scenario doesn't actually output the results after the test adaptation, instead it outputs the results before the test adaptation.

Thank you in advance!

Hello! First of all, thank you for sharing your great work :)
I've tested your code on CIFAR-10-C (severity 5) using CIFAR-10 pretrained ResNet 18/34/50, following the configuration in /master/cfgs/tent.yaml. While the tent indeed increases the classification performance at the very first epoch, it deteriorates with more epochs (even below the baseline model). This is my result:

I expected the performance to gradually increase, but it didn't. Is this result reasonable, or did I do something wrong?
Thank you :)

Hello,
Thanks for a very nice paper and open code. Would you have an example code for running tent for a segmentation network ?
thanks
Mathilde

I tested semantic segmentation from the cityscape dataset to the Zurich dataset. The miou for the former is 73%, and Zurich is 32% without tent, but becomes 16% with tent. I was hoping you could give me your semantic segmentation code for GTA to cityscapes so I can see what went wrong, please

Hi, thanks for your great work.
I re-implement Tent based on Transfer-Learning-Library to evaluate its performance on DA datasets.
And I only update BN layers as recommented in the paper.
However, I find the accuracy is really low. Could you please release the code to help reproducing the results on Digits and VisDA. Thanks a lot.

as title

As the paper said, TNET needs 2× the inference time plus 1× the gradient time per test point, but I found there is only one forward and one gradient update in the code

Is the right order: forward, backward, and forward?

I've trained my model and have some tests statistics X. Afterward, I an untented, but configured version of the model and I am unable to reach the accuracy X. I've changed the batch size to match the training batch size, but still the issue persist. It probably has to do with using the "model.train()" function. Note, I am using Efficientnet_v2_s as my model.

Hi,

Thank you for a very nice paper. I have tried to reproduce (or get close) to the results mentioned in the paper. I cannot reproduce the results for Cifar 10c and Cifar 100c by using the resnet-26 as also used by TTT (Sun et. al) paper. I am pretty sure, that I am doing everything right because my trained (resnet-26) gives me an error of 8.2 percent on the clean test set.

When I try to use it with tent, the performance degrades. I am attaching the log file for Resnet 26 here with this issue just so you know what is happening. I must also mention that the results mentioned in the paper (for Cifar10c and 100c) were reproduced by using a Wide resnet 28. But there is NO mention of using a wide resnet 28 in the paper.

Did you update the final version of the paper or I am doing something wrong? I am also attaching the log file with this issue for the results obtained by using wide resnet 28.

Also, can you please tell me what architecture did you use for the imagenet results? Basically when I take an off the shelf resnet 50 (pretrained on imagenet) and try to reproduce the results. It doesn't. In fact the system totally blows up! Is it some other model used for imagenet results as well, instead of resnet 50? Also, adding the log file for Imagenet results.

I look forward to your answer.

Thanks and BR,

Mirza.

tent_resnet_26.txt
tent_wide_resnet_28.txt
tent_imagenet.txt

Model is set to train mode in TENT so that BatchNorm modules use batch statistics at test time. However, this also sets other modules to train mode, for instance Dropout.

It is also possible to set submodules to train mode only, and I believe this would achieve the desired behaviour for BatchNorm without affecting other modules. Is my understanding correct?

Hi!Thanks for sharing your code here!
Here's a question,what is threat model for?I couldn't find it in the paper.looking forward to your reply!

all i know is minimizing the shannon entropy can lower the uncertainty of the model, but how can i assure the model is becoming more accurate instead of becoming less accurate after minimizing the entropy?looking forward to an answer

I have a simple question related to the offline tent experiment.
What should I change if I want to get results of the offline tent (ex, Table 2)?
Should I change _C.MODEL.EPISODIC = True

Termination For online adaptation, no termination is necessary, and iteration continues as long as
there is test data. For offline adaptation, the model is first updated and then inference is repeated.
Adaptation may of course continue by updating for multiple epochs.

Thanks for sharing this example code. Any plan to release code on ImageNet or segmentation datasets ?