confusion regarding additional datasets about wsolevaluation

[SanghyukChun]

No. We only use held-out sets for finding the hyperparameters. They are not used after the hyperparameter search. This is the correct procedure proposed in our paper:

1. Train a number of models with the train-weaksup split (e.g., original CUB train set).
2. Find the best model from (1) with the train-fullsup split (e.g., additional CUB V2).
3. The best model chosen from (2) is tested with the test split (e.g., original CUB test set).

You can find the details in Section 5.2. "Comparison of WSOL methods".

(..) we have randomly searched the optimal hyperparam- eters over the train-fullsup with 30 trials
(..) The checkpoints that achieves the best localization performance on train-fullsup are used for evaluation.

You can also find the details in our code.

wsolevaluation/main.py

Lines 367 to 386 in 38d4aef

	for epoch in range(trainer.args.epochs):
	print("===========================================================")
	print("Start epoch {} ...".format(epoch + 1))
	trainer.adjust_learning_rate(epoch + 1)
	train_performance = trainer.train(split='train')
	trainer.report_train(train_performance, epoch + 1, split='train')
	trainer.evaluate(epoch + 1, split='val')
	trainer.print_performances()
	trainer.report(epoch + 1, split='val')
	trainer.save_checkpoint(epoch + 1, split='val')
	print("Epoch {} done.".format(epoch + 1))
	print("===========================================================")
	print("Final epoch evaluation on test set ...")
	trainer.load_checkpoint(checkpoint_type=trainer.args.eval_checkpoint_type)
	trainer.evaluate(trainer.args.epochs, split='test')
	trainer.print_performances()
	trainer.report(trainer.args.epochs, split='test')
	trainer.save_performances()