OSError: [Errno 12] Cannot allocate memory about jukebox

[sdtblck]

I'm also getting OOM errors when running it on colab

[kylemcdonald]

Same, with 2080Ti (11GB). Made sure to select the GPU that isn't being used for the GUI.

(jukebox) $ nvidia-smi
Thu Apr 30 14:23:24 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 435.21       Driver Version: 435.21       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 208...  Off  | 00000000:05:00.0  On |                  N/A |
| 30%   38C    P8     4W / 260W |    198MiB / 11016MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 208...  Off  | 00000000:09:00.0 Off |                  N/A |
| 30%   39C    P8    20W / 260W |      1MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1129      G   /usr/lib/xorg/Xorg                            78MiB |
|    0      1305      G   /usr/bin/gnome-shell                         117MiB |
+-----------------------------------------------------------------------------+
(jukebox) $ CUDA_VISIBLE_DEVICES=1 python jukebox/sample.py --model=5b_lyrics --name=sample_5b --levels=3 --sample_length_in_seconds=20 --total_sample_length_in_seconds=180 --sr=44100 --n_samples=6 --hop_fraction=0.5,0.5,0.125
Using cuda True
{'name': 'sample_5b', 'levels': 3, 'sample_length_in_seconds': 20, 'total_sample_length_in_seconds': 180, 'sr': 44100, 'n_samples': 6, 'hop_fraction': (0.5, 0.5, 0.125)}
Setting sample length to 881920 (i.e. 19.998185941043083 seconds) to be multiple of 128
Downloading from gce
Restored from /home/kyle/.cache/jukebox-assets/models/5b/vqvae.pth.tar
0: Loading vqvae in eval mode
Conditioning on 1 above level(s)
Checkpointing convs
Checkpointing convs
Loading artist IDs from /home/kyle/Documents/jukebox/jukebox/jukebox/data/ids/v2_artist_ids.txt
Loading artist IDs from /home/kyle/Documents/jukebox/jukebox/jukebox/data/ids/v2_genre_ids.txt
Level:0, Cond downsample:4, Raw to tokens:8, Sample length:65536
Downloading from gce
Restored from /home/kyle/.cache/jukebox-assets/models/5b/prior_level_0.pth.tar
0: Loading prior in eval mode
Conditioning on 1 above level(s)
Checkpointing convs
Checkpointing convs
Loading artist IDs from /home/kyle/Documents/jukebox/jukebox/jukebox/data/ids/v2_artist_ids.txt
Loading artist IDs from /home/kyle/Documents/jukebox/jukebox/jukebox/data/ids/v2_genre_ids.txt
Level:1, Cond downsample:4, Raw to tokens:32, Sample length:262144
Downloading from gce
Restored from /home/kyle/.cache/jukebox-assets/models/5b/prior_level_1.pth.tar
0: Loading prior in eval mode
Loading artist IDs from /home/kyle/Documents/jukebox/jukebox/jukebox/data/ids/v2_artist_ids.txt
Loading artist IDs from /home/kyle/Documents/jukebox/jukebox/jukebox/data/ids/v2_genre_ids.txt
Level:2, Cond downsample:None, Raw to tokens:128, Sample length:1048576
0: Converting to fp16 params
Downloading from gce
Restored from /home/kyle/.cache/jukebox-assets/models/5b_lyrics/prior_level_2.pth.tar
0: Loading prior in eval mode
Traceback (most recent call last):
  File "jukebox/sample.py", line 237, in <module>
    fire.Fire(run)
  File "/home/kyle/anaconda3/envs/jukebox/lib/python3.7/site-packages/fire/core.py", line 127, in Fire
    component_trace = _Fire(component, args, context, name)
  File "/home/kyle/anaconda3/envs/jukebox/lib/python3.7/site-packages/fire/core.py", line 366, in _Fire
    component, remaining_args)
  File "/home/kyle/anaconda3/envs/jukebox/lib/python3.7/site-packages/fire/core.py", line 542, in _CallCallable
    result = fn(*varargs, **kwargs)
  File "jukebox/sample.py", line 234, in run
    save_samples(model, device, hps, sample_hps)
  File "jukebox/sample.py", line 215, in save_samples
    ancestral_sample(labels, sampling_kwargs, priors, hps)
  File "jukebox/sample.py", line 123, in ancestral_sample
    zs = _sample(zs, labels, sampling_kwargs, priors, sample_levels, hps)
  File "jukebox/sample.py", line 94, in _sample
    prior.cuda()
  File "/home/kyle/anaconda3/envs/jukebox/lib/python3.7/site-packages/torch/nn/modules/module.py", line 304, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/home/kyle/anaconda3/envs/jukebox/lib/python3.7/site-packages/torch/nn/modules/module.py", line 201, in _apply
    module._apply(fn)
  File "/home/kyle/anaconda3/envs/jukebox/lib/python3.7/site-packages/torch/nn/modules/module.py", line 201, in _apply
    module._apply(fn)
  File "/home/kyle/anaconda3/envs/jukebox/lib/python3.7/site-packages/torch/nn/modules/module.py", line 201, in _apply
    module._apply(fn)
  [Previous line repeated 3 more times]
  File "/home/kyle/anaconda3/envs/jukebox/lib/python3.7/site-packages/torch/nn/modules/module.py", line 223, in _apply
    param_applied = fn(param)
  File "/home/kyle/anaconda3/envs/jukebox/lib/python3.7/site-packages/torch/nn/modules/module.py", line 304, in <lambda>
    return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA out of memory. Tried to allocate 34.00 MiB (GPU 0; 10.76 GiB total capacity; 9.87 GiB already allocated; 2.62 MiB free; 10.03 GiB reserved in total by PyTorch)

[axiezai]

@prafullasd thank you for the response, seems like my swap is fine, the problem is low GPU memory, changing the model to 1b_lyrics worked fine, seems like I need to find some balance between n_samples and model size. Cheers!

[prafullasd]

For low GPU memory environments, try passing a lower max_batch_size / n_samples to sample.py, or the 1b_lyrics model instead of the 5b_lyrics.

[prafullasd]

The swap I'm not sure hmm it looks like it fails to download the model. You'll need 2Gb each for upsamplers and 1B_lyric model, and 11Gb for 5b_lyric model. Maybe try changing the path it downloads to? In gcs_utils.py, we currently download to .cache, but try another location?