server.py doesn't seem to use DP/DDP for multi-GPU about text-generation-webui

[quarterturn]

using 6.7B:

Tue Jan 10 05:19:36 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.60.13    Driver Version: 525.60.13    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla P100-PCIE...  On   | 00000000:05:00.0 Off |                    0 |
| N/A   31C    P0    32W / 250W |  13062MiB / 16384MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla P100-PCIE...  On   | 00000000:42:00.0 Off |                    0 |
| N/A   32C    P0    26W / 250W |      2MiB / 16384MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A     17510      C   python                          13060MiB |
+-----------------------------------------------------------------------------+

It's only bound to GPU 0.

[oobabooga]

I don't have a multi-GPU machine to test, so I need your help:

1. Is server.py loading the model using line 36 or line 46? Can you add print statements before each of those lines to check?
2. If it is using line 46, can you try changing it to this?

model = AutoModelForCausalLM.from_pretrained(Path(f"models/{model_name}"), device_map='auto')

I am not sure if the transformers library supports splitting models across multiple GPUs out of the box, but this might work.

[quarterturn]

It 'works' but not the way you want:
No more exception:

Loading KoboldAI_OPT-13B-Erebus...
loaded on line 46
Loaded the model in 65.22 seconds.
Running on local URL:  http://0.0.0.0:7860

To create a public link, set `share=True` in `launch()`.

but it's not allocating memory across the GPUs correctly:

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A     21306      C   python                           8068MiB |
|    1   N/A  N/A     21306      C   python                           8504MiB |
+-----------------------------------------------------------------------------+

As well as responses take forever to return.

[oobabooga]

Unfortunately, it seems that device_map="auto" is the only default solution provided by transformers at the moment: huggingface/transformers#15799 (comment)

[oobabooga]

Can you try this?

model = AutoModelForCausalLM.from_pretrained(Path(f"models/{model_name}"), torch_dtype=torch.float16, device_map='auto')

Maybe it is trying to load the model in 32-bit mode and sending part of the layers to the CPU.

[quarterturn]

Much better:

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A     33094      C   python                          13266MiB |
|    1   N/A  N/A     33094      C   python                          12730MiB |
+-----------------------------------------------------------------------------+

It's able to answer questions, albeit not fast on my hardware:
Input:
Question: Who is Asuka?
Output:
I think it's a character from Evangelion. I'm not really into anime so that was just what my friend told me once when we were talking about this stuff and she seemed to know her name, but then again he might have been pulling our legs... But if you want the real answer here goes nothing! She's an angel of some sort who can take any form she wants at will as long as there are no humans around because angels don't like them or something along those lines. So basically in other words she could be anyone she wanted with whatever powers they had on Earth without having to worry about being recognized by someone else. That makes sense right?! Anyway back to why I brought up all these things.... Because now for reasons unknown to us both (and probably most people) she has taken over his body which explains how everything happened today."<|endoftext|>

[oobabooga]

It should now be possible to reproduce this ok-ish behavior across multiple GPUs with

python server.py --auto-devices

I will close the issue. In case of errors, feel free to reopen it.