Warnings end error using TuckerTRL with multiple GPUs about torch

[JeanKossaifi]

Thanks for reporting @segalinc!

So the first issue seems to be related to this pytorch/pytorch#46983, I don't think we ever add anything to a ParameterList that's not an nn.Parameter, but will double check. That issue should be fixed in PyTorch 1.7.1 pytorch/pytorch#49285

The other issue is new to me, I'll have a look and see why this happens.

As a side note, you may want to use nn.parallel.DistributedDataParallel instead of just DataParallel : https://pytorch.org/docs/stable/notes/cuda.html#cuda-nn-ddp-instead

[segalinc]

I actually have PyTorch 1.7.1. I also tried to update it but still...

[JeanKossaifi]

Ok, so the error you got was coming from the way we handled thread-safe tenalg backend setting in TensorLy and is fixed by tensorly/tensorly@f0b701e

However, there seems to still be an issue, seemingly related to pytorch/pytorch#36035. It seems the parameters in the factors ParameterList are not copied to the devices -- let me know if you also experience this.

[segalinc]

For the first issue I will try to update the package and hopefully is fixed... Thank you! For the second issue that's exactly what happens, they are empty and get the warning Cristina

…

Sent from my OnePlus

On Fri, Jan 29, 2021, 16:24 Jean Kossaifi ***@***.***> wrote: Ok, so the error you got was coming from the way we handled thread-safe tenalg backend setting in TensorLy and is fixed by tensorly/tensorly@ f0b701e <tensorly/tensorly@f0b701e> However, there seems to still be an issue, seemingly related to pytorch/pytorch#36035 <pytorch/pytorch#36035>. It seems the parameters in the factors ParameterList are not copied to the devices -- let me know if you also experience this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACHLPESFIQ43HZA7JJWCNCTS4NGUPANCNFSM4WWFGBXA> .

[JeanKossaifi]

Thanks, I've commented on the PyTorch issue at pytorch/pytorch#36035 (comment), but it seems they are not actively working on this.

[JeanKossaifi]

I pushed a temporary fix in 38d2614
Let me know if this doesn't fix your problem @segalinc

[HuyTu7]

Hi I also just encountered the second issue when trying multiple GPU with torch.nn.DataParallel in both Pytorch 1.7 and 1.8. Any recommendations?

[JeanKossaifi]

If your issue is with PyTorch, I recommending commenting in the corresponding issue: pytorch/pytorch#36035 (comment)

In TensorLy-Torch we use a custom ParameterList, feel free to try for your application!