unexptected behavior of AdamP on CPU? about adamp

[bhheo]

Oh, then, it is clear that AdamP causes the problem.

I'm sorry but, it is really hard to figure out the reason for cpu issue.
I have read the code multiple times. Everything looks just fine.

I'm suspicious of the built-in cosine functionF.cosine_similarity.
You can try an old version of AdamP(https://github.com/clovaai/AdamP/blob/v0.2.0/adamp/adamp.py) to verify the built-in cosine function.
If the old version has the same problem, I have no idea what is causing the problem.

[bhheo]

Thank you for your valuable comments 👍
I will fix it when I find a clue for the problem.

[bhheo]

Hi

I haven't tried AdamP on cpu. Most of the experiments are conducted on GPU environments.
So, I'm not 100% sure, but, I believe our implementation can work on cpu.
BecauseAdamP consists of simple operations which has no dependency on cuda.

I think other parts of the training code might cause the problem.
I recommend you to test other optimizers such as Adam or AdamW.
If other optimizers work normally on the CPU, I will try to figure out the cause of the problem.

[Junyoungpark]

Hi, Thanks for the reply.

I already checked the code on cpu with Adam and other optimizers in torch.optim. All of them work properly on both devices; cuda and cpu. FYI, the machine that I used for the experiments equips a Ryzen 2990wx and two Nvidia RTX titans.

BTW, AdamP results in better training outcomes than Adam on cuda. Cheers Adamp.

[Junyoungpark]

oh actually I'm fine w/ the current version of AdamP because I still can run the codes on cuda. Thanks for the comments :)
I raise an issue because it might be helpful to the contributors/maintainers to notice this 'maybe-issue' problem.