Validation data is same as test data? about sdat

[rangwani-harsh]

Yes, in domain adaptation, the standard is to report validation accuracy. The same is followed for reporting results on all baselines.

[toshi2k2]

And the same validation data can be used to tune the model/hyperparameters or choose the best model dependent on validation loss?

[rangwani-harsh]

In domain adaptation, one adaptation setting like Ar --> Pr for Office-Home can be used to tune the model/hyperparameters and used for all other adaptation settings like (Ar --> Cl, Ar--> Rw, etc.). Hence, the evaluation is fair overall.

[toshi2k2]

Correct me if I'm wrong, but it appears that your best model for a specific source-target data pair is chosen on the training model's performance on the same target domain test data (examples/cdan_mcc_sdat.py#L159). This doesn't seem to be a fair evaluation, or am I misunderstanding something here?

        # evaluate on validation set
        acc1 = utils.validate(val_loader, classifier, args, device)
        if args.log_results:
            wandb.log({'epoch': epoch, 'val_acc': acc1})

        # remember best acc@1 and save checkpoint
        torch.save(classifier.state_dict(),
                   logger.get_checkpoint_path('latest'))
        if acc1 > best_acc1:
            shutil.copy(logger.get_checkpoint_path('latest'),
                        logger.get_checkpoint

[rangwani-harsh]

Yes, the best model is chosen based on target domain accuracy. But all the methods follow this; hence the comparison is fair (refer: https://github.com/thuml/Transfer-Learning-Library for details). This is the case for Office-Home and VisDA datasets.

For DomainNet, there are separate sets of data for validation and testing; hence test set is used for final accuracy evaluation.

Hope this helps :)

[toshi2k2]

In my opinion, I think that is a form of evaluation data contamination and given the random nature of neural network loss optimization as well as the fact that many methods may not able to minimize target domain loss without this kind of model selection - this setup may not be a fair comparison.
i also didn't notice this setup in older relevant papers like MCD(unless I missed this). Is there any major work which started this evaluation process?

[rangwani-harsh]

I do agree that there are some concerns regarding this evaluation procedure. In this work, we followed and reported results using the standard protocol at: https://github.com/thuml/Transfer-Learning-Library. I am unaware of the work that started, but most new works follow this protocol (https://github.com/thuml/Transfer-Learning-Library).

I will close this issue for now as it's unrelated to our work. In case you have any questions related to our implementation, please feel free to re-open it.