Why use drop_last=True in test (and val) dataloader? about patchtst

[namctin]

Thank you for confirming the result @oguiza! Since ILI is a fairly small dataset, so the setting of drop_last can affect the result. I would think this will be diminished for larger datasets. But in any case, we should set the drop_last=False and we will fix that in the code.

[ts-kim]

I discovered that a number of the performance gain in the supervised patchTST comes from using 'drop_last=True' for the test dataset.

With 'drop_last=False', for the ETTh1, Traffic, and Illness datasets, there was a drop in performance, while there was no drop for the Weather dataset.
I have not yet tested on the other datasets.

I think this can be a significant problem, as it would require the most of the tables in the paper to be rewritten and this could be considered as a cheating.

[namctin]

Hi @oguiza, that is a great catch. Thank you very much! Our thought was to use the same setting in the dataloader they had for a fair comparison. But you are correct that setting drop_last=True will lead to the number of test samples not to be consistent with different choices of the batch size, although this difference may not cause a significant performance drop. We will change that setting in the code.

For self-supervised learning, we accidentally use the default drop_last=False which turns out to be the right choice, and the reported self-supervised results in the paper is for this setting.

[oguiza]

Thanks for your quick response @namctin.
I just run the script on the ILI dataset and the difference was significant in my opinion. If the performance with self-supervised is measured with all test samples as you say, this might indicate that the difference between supervised and self-supervised would be bigger than what's reflected in your paper. I'm sharing this just in case you want to investigate it further.