Unable to reproduce models and Increasing validation loss about inspyrenet

[plemeri]

Did you solve the problem? For your information, we trained our model for high resolution configuration with 8 Titan RTX GPUs with 1 sample for each device which is total 8 samples for the batch size. Although we also tried using single GPU but we did not have any trouble like above.

Also, I keep noticing that some people have training issue which I cannot reproduce the problem myself.
I cannot be sure but the problem might come from the CUDA version, pytorch version, accidentally using half precision, dataset problem, or others.

I would like to ask you to not modify a single script of our repository and try to reproduce the basic model first to make sure that our code is working fine on your machine. The configuration that you may find is InSPyReNet_SwinB.yaml. Please do not change any code and just use DUTS-TR for training. Then, evaluate on other benchmarks including UHRSD-TE. If you can reproduce our results, then you might changed something causing the problem.

I also would like to mention that I did not trained our model many times to produce the best result for the paper. I just trained once and tested on various GPU servers and verified that our method consistently produced almost identical results, so if you solve the problem above, I can guarantee that you will get the results that you've expected, so don't give up on your project and I'll be your help as much as I can.