Multi-GPU training. about joeynmt

[may-]

Hi @tongye98

Thank you for asking!
The short answer is no, simply because I cannot make it work. I tried to integrate DistributedDataParallel, but for some reason I keep getting errors in my Single-Node Multi-GPU environment. Moreover, it's difficult to test multi-node distributed learning on my gpu cluster.

I totally agree, it's really nice to have. We are waiting for a contributor who can work on it. Your help is needed!!

[may-]

Reminder for myself:

• https://pytorch.org/docs/stable/notes/ddp.html
• https://pytorch.org/tutorials/intermediate/ddp_tutorial.html
• https://pytorch.org/tutorials/intermediate/dist_tuto.html

[Darwin99-debug]

Hi @may- , I used this tutorial(https://www.youtube.com/watch?v=-K3bZYHYHEA&list=PL_lsbAsL_o2CSuhUhJIiW0IkdT5C2wGWj) to integrate DDP and torch run in a Pytorch project and it worked well for me!

[may-]

Hi @Darwin99-debug,
Thank you for your comment!
Actually, we have been working on this (DDP) for a while. You can see our progress in the ddp branch (#225)

Although it works, we are concerned that our current DDP implementation makes JoeyNMT less readable, less novice-friendly.
So we hesitate to integrate DDP to the main branch. Any comments and suggestions will be appreciated!

FYI @juliakreutzer