Training problem about multi-tacotron-voice-cloning

[ramanova]

Hello, getting the same error, torch==1.5.0
I see that we have

    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    # FIXME: currently, the gradient is None if loss_device is cuda
    loss_device = torch.device("cpu")

after that if we are using clip_grad_norm_ from torch, it performs operation on all of the parameters, two of which are on cpu, and the rest on cuda:0

total_norm = torch.norm(torch.stack([torch.norm(p.grad.detach(), norm_type) for p in parameters]), norm_type)

which throws the error.
Could it be that the torch version is incorrect? I'm using 1.5.0

[UPDATE]
Reinstalled torch and it started training!
pip uninstall torch # you might need to ran it twice, check with
pip list | grep torch # that you don't have torch left
pip install torch # or pip install torch==1.5.0 to ensure the version

[vlomme]

Hello. use
python encoder_train.py my_run --clean_data_root D:\Datasets\SV2TTS\encoder

[vlomme]

not enough video memory. Reduce the Batch size

[JokeCorleone]

Hello @vlomme
Thank for your support,
When I used python encoder_train.py my_run --clean_data_root D:\Datasets\SV2TTS\encoder,
The result is

File "encoder_train.py", line 46, in
train(**vars(args))
File "D:\SV2TTS\encoder\train.py", line 87, in train
model.do_gradient_ops()
File "D:\SV2TTS\encoder\model.py", line 39, in do_gradient_ops
clip_grad_norm_(self.parameters(), 3, norm_type=2)
File "C:\Users\Admin\anaconda3\envs[Test_Voice]\lib\site-packages\torch\nn\utils\clip_grad.py", line 30, in clip_grad_norm_
total_norm = torch.norm(torch.stack([torch.norm(p.grad.detach(), norm_type) for p in parameters]), norm_type)
RuntimeError: All input tensors must be on the same device. Received cpu and cuda:0

[JokeCorleone]

Hello,
When I trained vocoder (run python vocoder_train.py my_run D:\Datasets), I encountered an error:

+------------+--------+--------------+
| Batch size | LR | Sequence Len |
+------------+--------+--------------+
| 60 | 0.0001 | 1000 |
+------------+--------+--------------+

RuntimeError: CUDA out of memory. Tried to allocate 118.00 MiB (GPU 0; 4.00 GiB total capacity; 2.87 GiB already allocated; 10.61 MiB free; 32.29 MiB cached)

how can i solve this error ?

[JokeCorleone]

Thank @vlomme