mbilker / vgpu_unlock-rs

Unlock vGPU functionality for consumer grade GPUs

Rust-based vgpu_unlock

Unlock vGPU functionality for consumer-grade NVIDIA GPUs.

This tool is to be used with the kernel patches from the main vgpu_unlock repository!

Dependencies

• This tool requires Rust. You can install it via your package manager or via .
• Rust requires a linker to be installed to be able to create the shared library. Typically, this is installed with the C compiler through your distribution's package manager.
• The dependencies from the main vgpu_unlock project excluding Python and frida.

Installation

In the following instructions <path_to_vgpu_unlock_rs> needs to be replaced with the path to this repository on the target system.

Install the NVIDIA vGPU driver and kernel driver patches as detailed in the main vgpu_unlock project README. Ignore the steps regarding editing the systemd service unit files.

Run cargo build --release to compile the shared library.

Create the directories /etc/systemd/system/nvidia-vgpud.service.d and /etc/systemd/system/nvidia-vgpu-mgr.service.d.

Create the files /etc/systemd/system/nvidia-vgpud.service.d/vgpu_unlock.conf and /etc/systemd/system/nvidia-vgpu-mgr.service.d/vgpu_unlock.conf with the following:

[Service]
Environment=LD_PRELOAD=<path_to_vgpu_unlock_rs>/target/release/libvgpu_unlock_rs.so

Create the directory /etc/vgpu_unlock which will house the vGPU profile override configuration file.

Create the file /etc/vgpu_unlock/profile_override.toml with the profile fields that are to be overridden. The following is an example for nvidia-55 (GRID P40-2A) that sets the number of heads to 1, sets the framebuffer to be 1920x1080 (1920 * 1080 = 2073600 pixels), enables CUDA, and disables the frame-rate limiter.

[profile.nvidia-55]
num_displays = 1
display_width = 1920
display_height = 1080
max_pixels = 2073600
cuda_enabled = 1
frl_enabled = 0

If you want to enable VM migration or snapshotting, you must recompile the nvidia-vgpu-vfio kernel module with NV_KVM_MIGRATION_UAPI equal to 1. Then, create the file /etc/vgpu_unlock/config.toml and add the following:

unlock_migration = true

Happy hacking!

vgpu_unlock-rs's Issues

unable to compile --noob issue

Hi

I am working my way through some vgpu tutorial..

proxmox 7.2 fresh install
this tutorial: https://wvthoog.nl/proxmox-7-vgpu-v2/

when i run cargo build --release it will error out with the following:

error[E0658]: use of unstable library feature 'assoc_char_funcs': recently added
--> src/format.rs:77:21
|
77 | for item in char::decode_utf16(self.0.iter().copied().take_while(|&ch| ch != 0)) {
| ^^^^^^^^^^^^^^^^^^
|
= note: see issue #71763 rust-lang/rust#71763 for more information

error[E0658]: use of unstable library feature 'assoc_char_consts': recently added
--> src/format.rs:78:41
|
78 | f.write_char(item.unwrap_or(char::REPLACEMENT_CHARACTER))?;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: see issue #71763 rust-lang/rust#71763 for more information

error: aborting due to 2 previous errors

i have no idea how to get past this..

Halp plz

Fails to start VM

Setup:
i5-10400f
GTX1660SUPER
Followed Jeff (Craft Computings) guide on this software

Error in GUI:

kvm: -device vfio-pci,sysfsdev=/sys/bus/pci/devices/0000:01:00.0/00000000-0000-0000-0000-000000000102,id=hostpci0,bus=pci.0,addr=0x10: warning: vfio 00000000-0000-0000-0000-000000000102: Could not enable error recovery for the device
kvm: -device vfio-pci,sysfsdev=/sys/bus/pci/devices/0000:01:00.0/00000000-0000-0000-0000-000000000102,id=hostpci0,bus=pci.0,addr=0x10: vfio 00000000-0000-0000-0000-000000000102: failed to read device config space: Bad address

Error in nvidia-vgpu-mgr:

Apr 06 13:46:20 pve nvidia-vgpu-mgr[25539]: error: vmiop_log: vgpu(+0xeb43) [0x5652bb60eb43]
Apr 06 13:46:20 pve nvidia-vgpu-mgr[25539]: error: vmiop_log: vgpu(+0xc3b6) [0x5652bb60c3b6]
Apr 06 13:46:20 pve nvidia-vgpu-mgr[25539]: error: vmiop_log: vgpu(+0x3bda) [0x5652bb603bda]
Apr 06 13:46:20 pve nvidia-vgpu-mgr[25539]: error: vmiop_log: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7ff8cba6cd0a]
Apr 06 13:46:20 pve nvidia-vgpu-mgr[25539]: error: vmiop_log: vgpu(+0x3c1d) [0x5652bb603c1d]
Apr 06 13:46:20 pve nvidia-vgpu-mgr[25539]: error: vmiop_log: (0x0): init_device_instance failed for inst 0 with error 1 (error setting vGPU configuration information f>
Apr 06 13:46:20 pve nvidia-vgpu-mgr[25539]: error: vmiop_log: (0x0): Initialization: init_device_instance failed error 1
Apr 06 13:46:20 pve nvidia-vgpu-mgr[25539]: error: vmiop_log: display_init failed for inst: 0
Apr 06 13:46:20 pve nvidia-vgpu-mgr[25539]: error: vmiop_env_log: (0x0): vmiope_process_configuration: plugin registration error
Apr 06 13:46:20 pve nvidia-vgpu-mgr[25539]: error: vmiop_env_log: (0x0): vmiope_process_configuration failed with 0x1f

Someone with a solution?

Unable to override with two mdev profiles within one VM

OS: Proxmox 7.3
Kernel: 5.15.74-1-pve
Host Driver: 525.60.12
Guest Driver: 525.60.13
Hardware: RTX 2060 12G * 2

Host Config:

[mdev.00000000-0000-0000-0000-000000000102]
num_displays = 1
display_width = 1920
display_height = 1080
max_pixels = 2073600
cuda_enabled = 1
frl_enabled = 0
framebuffer = 0x162000000

[mdev.00000001-0000-0000-0000-000000000102]
num_displays = 1
display_width = 1920
display_height = 1080
max_pixels = 2073600
cuda_enabled = 1
frl_enabled = 0
framebuffer = 0x162000000

VM Config

hostpci0: 0000:82:00.0,mdev=nvidia-259,pcie=1
hostpci1: 0000:01:00.0,mdev=nvidia-259,pcie=1

PVE vm Start

kvm: -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/00000000-0000-0000-0000-000000000102,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0: warning: vfio 00000000-0000-0000-0000-000000000102: Could not enable error recovery for the device
kvm: -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/00000001-0000-0000-0000-000000000102,id=hostpci1,bus=ich9-pcie-port-2,addr=0x0: warning: vfio 00000001-0000-0000-0000-000000000102: Could not enable error recovery for the device
TASK OK

Host journalctl nvidia-vgpu-mgr.service
pastebin

Problem:
When two vGPUs are added to a virtual machine, the mdev override does not work properly, but if override with a profile it works fine.

Same issue as #5 - Fails to start VM

Same issue as #5 - #5

Host Setup:

• Proxmox 7.2 on Debian
• Quadro RTX 4000

Guest Setup

• 20.04 ubuntu desktop
• Quadro RTX 6000

Followed Jeff (Craft Computings) guide on this software. Able to get list using "mdevctl types". Looking to split RTX4000 into 4x instances of 2Gb each

Error

root@p53:~# qm start 311
no efidisk configured! Using temporary efivars disk.
mdev instance '00000000-0000-0000-0000-000000000311' already existed, using it.
kvm: -device vfio-pci,sysfsdev=/sys/bus/pci/devices/0000:01:00.0/00000000-0000-0000-0000-000000000311,id=hostpci0,bus=pci.0,addr=0x10: warning: vfio 00000000-0000-0000-0000-000000000311: Could not enable error recovery for the device
kvm: -device vfio-pci,sysfsdev=/sys/bus/pci/devices/0000:01:00.0/00000000-0000-0000-0000-000000000311,id=hostpci0,bus=pci.0,addr=0x10: vfio 00000000-0000-0000-0000-000000000311: failed to read device config space: Bad address
start failed: QEMU exited with code 1
root@p53:~# 

.
Setup

.

root@p53:~# nvidia-smi
Mon May 23 17:13:41 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.47.03    Driver Version: 510.47.03    CUDA Version: N/A      |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro RTX 4000     On   | 00000000:01:00.0 Off |                  N/A |
| N/A   42C    P8    15W /  N/A |     54MiB /  8192MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
root@p53:~#

.

root@p53:~# systemctl stop nvidia-vgpu-mgr.service
root@p53:~# systemctl status nvidia-vgpu-mgr.service
● nvidia-vgpu-mgr.service - NVIDIA vGPU Manager Daemon
     Loaded: loaded (/lib/systemd/system/nvidia-vgpu-mgr.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/nvidia-vgpu-mgr.service.d
             └─vgpu_unlock.conf
     Active: inactive (dead) since Mon 2022-05-23 17:04:21 +08; 39s ago
    Process: 48056 ExecStart=/usr/bin/nvidia-vgpu-mgr (code=exited, status=0/SUCCESS)
    Process: 53989 ExecStopPost=/bin/rm -rf /var/run/nvidia-vgpu-mgr (code=exited, status=0/SUCCESS)
   Main PID: 48057 (code=exited, status=0/SUCCESS)
        CPU: 18ms

May 23 16:47:35 p53.rubicon.local nvidia-vgpu-mgr[49013]: error: vmiop_log: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f>
May 23 16:47:35 p53.rubicon.local nvidia-vgpu-mgr[49013]: error: vmiop_log: vgpu(+0x3c1d) [0x562973203c1d]
May 23 16:47:35 p53.rubicon.local nvidia-vgpu-mgr[49013]: error: vmiop_log: (0x0): init_device_instance failed for inst 0 with error 3 (v>
May 23 16:47:35 p53.rubicon.local nvidia-vgpu-mgr[49013]: error: vmiop_log: (0x0): Initialization: init_device_instance failed error 3
May 23 16:47:35 p53.rubicon.local nvidia-vgpu-mgr[49013]: error: vmiop_log: display_init failed for inst: 0
May 23 16:47:35 p53.rubicon.local nvidia-vgpu-mgr[49013]: error: vmiop_env_log: (0x0): vmiope_process_configuration: plugin registration >
May 23 16:47:35 p53.rubicon.local nvidia-vgpu-mgr[49013]: error: vmiop_env_log: (0x0): vmiope_process_configuration failed with 0x5b
May 23 17:04:20 p53.rubicon.local systemd[1]: Stopping NVIDIA vGPU Manager Daemon...
May 23 17:04:21 p53.rubicon.local systemd[1]: nvidia-vgpu-mgr.service: Succeeded.
May 23 17:04:21 p53.rubicon.local systemd[1]: Stopped NVIDIA vGPU Manager Daemon.

root@p53:~# systemctl start nvidia-vgpu-mgr.service
root@p53:~# systemctl status nvidia-vgpu-mgr.service
● nvidia-vgpu-mgr.service - NVIDIA vGPU Manager Daemon
     Loaded: loaded (/lib/systemd/system/nvidia-vgpu-mgr.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/nvidia-vgpu-mgr.service.d
             └─vgpu_unlock.conf
     Active: active (running) since Mon 2022-05-23 17:05:34 +08; 3s ago
    Process: 54378 ExecStart=/usr/bin/nvidia-vgpu-mgr (code=exited, status=0/SUCCESS)
   Main PID: 54379 (nvidia-vgpu-mgr)
      Tasks: 1 (limit: 76740)
     Memory: 304.0K
        CPU: 6ms
     CGroup: /system.slice/nvidia-vgpu-mgr.service
             └─54379 /usr/bin/nvidia-vgpu-mgr

May 23 17:05:34 p53.rubicon.local systemd[1]: Starting NVIDIA vGPU Manager Daemon...
May 23 17:05:34 p53.rubicon.local systemd[1]: Started NVIDIA vGPU Manager Daemon.
May 23 17:05:34 p53.rubicon.local nvidia-vgpu-mgr[54379]: notice: vmiop_env_log: nvidia-vgpu-mgr daemon started
root@p53:~# 

.

root@p53:~# systemctl status nvidia-vgpud.service
● nvidia-vgpud.service - NVIDIA vGPU Daemon
     Loaded: loaded (/lib/systemd/system/nvidia-vgpud.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/nvidia-vgpud.service.d
             └─vgpu_unlock.conf
     Active: inactive (dead) since Mon 2022-05-23 17:16:45 +08; 1s ago
    Process: 58289 ExecStart=/usr/bin/nvidia-vgpud (code=exited, status=0/SUCCESS)
    Process: 58291 ExecStopPost=/bin/rm -rf /var/run/nvidia-vgpud (code=exited, status=0/SUCCESS)
   Main PID: 58290 (code=exited, status=0/SUCCESS)
        CPU: 245ms

May 23 17:16:45 p53.rubicon.local nvidia-vgpud[58290]: BAR1 Length: 0x100
May 23 17:16:45 p53.rubicon.local nvidia-vgpud[58290]: Frame Rate Limiter enabled: 0x1
May 23 17:16:45 p53.rubicon.local nvidia-vgpud[58290]: Number of Displays: 1
May 23 17:16:45 p53.rubicon.local nvidia-vgpud[58290]: Max pixels: 1310720
May 23 17:16:45 p53.rubicon.local nvidia-vgpud[58290]: Display: width 1280, height 1024
May 23 17:16:45 p53.rubicon.local nvidia-vgpud[58290]: License: GRID-Virtual-Apps,3.0
May 23 17:16:45 p53.rubicon.local nvidia-vgpud[58290]: PID file unlocked.
May 23 17:16:45 p53.rubicon.local nvidia-vgpud[58290]: PID file closed.
May 23 17:16:45 p53.rubicon.local nvidia-vgpud[58290]: Shutdown (58290)
May 23 17:16:45 p53.rubicon.local systemd[1]: nvidia-vgpud.service: Succeeded.
root@p53:~# 

.

root@p53:~# cat /etc/vgpu_unlock/profile_override.toml
[profile.nvidia-257]
num_displays = 1
display_width = 1920
display_height = 1080
max_pixels = 2073600
cuda_enabled = 1
frl_enabled = 60
framebuffer = 1968526677
pci_id = 0x1E3012BA
pci_device_id = 0x1E30
root@p53:~# 

.

root@p53:~# cat /etc/pve/nodes/p53/qemu-server/311.conf
agent: 1
args: -uuid 00000000-0000-0000-0000-000000000311
bios: ovmf
boot: order=scsi0;ide2;net0
cores: 4
hostpci0: 0000:01:00.0,mdev=nvidia-257
ide2: local:iso/ubuntu-20.04.4-desktop-amd64.iso,media=cdrom,size=3299872K
machine: q35
memory: 2048
meta: creation-qemu=6.2.0,ctime=1653292108
name: u311.u2004
net0: virtio=26:69:CE:5F:52:16,bridge=vmbr1,firewall=1,tag=30
numa: 0
ostype: l26
scsi0: local:311/vm-311-disk-0.qcow2,size=60G
scsihw: virtio-scsi-pci
smbios1: uuid=d2e8b2af-319f-44a5-aeb5-c0c449b7d92d
sockets: 1
vmgenid: f2a8f0f4-7b18-4a9c-903e-c4161e6d7abc
root@p53:~# 

.
Even if I proceed with the Guest VM (ubuntu 20.04) setup and install the nvidia driver, the driver will not run

(from guest vm)

$ sudo apt install nvidia-driver-470
$ nvidia-smi
NVIDIA_SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running

Would appreciate any help to move this along. Thanks!

nvidia-vgpud.service wont activate

Hi, Thanks for your generous contribution.

I followed your instruction but nvidia-vgpud stays in an inactive state, the process not got started.
Is this normal? How can I fix this?

root@pve:/opt/vgpu_unlock-rs/target/release# systemctl status nvidia-vgpud.service
● nvidia-vgpud.service - NVIDIA vGPU Daemon
     Loaded: loaded (/lib/systemd/system/nvidia-vgpud.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/nvidia-vgpud.service.d
             └─vgpu_unlock.conf
     Active: inactive (dead) since Fri 2023-07-14 11:18:16 CST; 21s ago
    Process: 10988 ExecStart=/usr/bin/nvidia-vgpud (code=exited, status=0/SUCCESS)
    Process: 10990 ExecStopPost=/bin/rm -rf /var/run/nvidia-vgpud (code=exited, status=0/SUCCESS)
   Main PID: 10989 (code=exited, status=0/SUCCESS)
        CPU: 131ms

Jul 14 11:18:16 pve nvidia-vgpud[10989]: Frame Rate Limiter enabled: 0x1
Jul 14 11:18:16 pve nvidia-vgpud[10989]: Number of Displays: 1
Jul 14 11:18:16 pve nvidia-vgpud[10989]: Max pixels: 1310720
Jul 14 11:18:16 pve nvidia-vgpud[10989]: Display: width 1280, height 1024
Jul 14 11:18:16 pve nvidia-vgpud[10989]: Multi-vGPU Exclusive supported: 0x1
Jul 14 11:18:16 pve nvidia-vgpud[10989]: License: GRID-Virtual-Apps,3.0
Jul 14 11:18:16 pve nvidia-vgpud[10989]: PID file unlocked.
Jul 14 11:18:16 pve nvidia-vgpud[10989]: PID file closed.
Jul 14 11:18:16 pve nvidia-vgpud[10989]: Shutdown (10989)

I have the vgpu_unlock.conf

root@pve:/opt/vgpu_unlock-rs/target/release# cat /etc/systemd/system/nvidia-vgpud.service.d/vgpu_unlock.conf
[Service]
Environment=LD_PRELOAD=/opt/vgpu_unlock-rs/target/release/libvgpu_unlock_rs.so

please help, thank you.

how to recompile nvidia-vgpu-vfio with NV_KVM_MIGRATION_UAPI ?

Hi,
I'm currently working to add vgpu live migration on proxmox.

I have seen in your doc that nvidia-vgpu-vfio need to be recompiled with NV_KVM_MIGRATION_UAPI=1.

Can you tell me how we can send this variable when building dmks driver ?

Failed to override nvidia profile

1. Machine infomations:
   OS: archlinux
   Kernel: archlinux 5.12.15-arch1-1
   NVIDIA Driver Version: 460.73.01
   GPU: RTX 1660 SUPPER

2. Override profile:

➜  vgpu_unlock-rs cat /etc/vgpu_unlock/profile_override.toml
[profile.nvidia-258]
num_displays = 1
display_width = 1920 
display_height = 1280 
max_pixels = 2457600 
cuda_enabled = 1

3. Problem:
   Failed override nvidia profile. The error message "Parameters size for NVA082_CTRL_CMD_HOST_VGPU_DEVICE_GET_VGPU_TYPE_INFO was 1840 bytes, expected 1848 bytes" is reported.

4. Logs:
   4.1 nvidia-vgpu-mgr logs:

➜  vgpu_unlock-rs journalctl -u nvidia-vgpu-mgr.service
Jan 09 00:01:47 archlinux systemd[1]: Starting NVIDIA vGPU Manager Daemon...
Jan 09 00:01:47 archlinux systemd[1]: Started NVIDIA vGPU Manager Daemon.
Jan 09 00:01:48 archlinux nvidia-vgpu-mgr[487]: notice: vmiop_env_log: nvidia-vgpu-mgr daemon started
Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[487]: Nv0000CtrlVgpuGetStartDataParams {
                                                    mdev_uuid: {2b7bc7e4-8784-4933-8613-9bc92bbc718c},
                                                    config_params: "vgpu_type_id=258",
                                                    qemu_pid: 3091,
                                                    gpu_pci_id: 0x100,
                                                    vgpu_id: 2,
                                                    gpu_pci_bdf: 256,
                                                }
Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_env_log: vmiop-env: guest_max_gpfn:0x0
Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_env_log: (0x0): Received start call from nvidia-vgpu-vfio module: mdev uuid 2b7bc7e4-8784-4933-8613-9bc92bbc718c GPU PCI id 00:01:00.0 config params vgpu_type_id=258
Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_env_log: (0x0): pluginconfig: vgpu_type_id=258
Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_env_log: Successfully updated env symbols!
Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: cmd: 0x20801322 failed.
Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: cmd: 0x2080014b failed.
Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: Parameters size for NVA082_CTRL_CMD_HOST_VGPU_DEVICE_GET_VGPU_TYPE_INFO was 1840 bytes, expected 1848 bytes
Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: (0x0): gpu-pci-id : 0x100
Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: (0x0): vgpu_type : Quadro
Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: (0x0): Framebuffer: 0xb0000000
Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: (0x0): Virtual Device Id: 0x1e30:0x1327
Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: (0x0): FRL Value: 60 FPS
Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: ######## vGPU Manager Information: ########
Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: Driver Version: 460.73.01
Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: cmd: 0x2080012f failed.
Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: (0x0): Cannot query ECC status. vGPU ECC support will be disabled.
Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: (0x0): Init frame copy engine: syncing...
Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: (0x0): vGPU migration disabled
Jan 09 00:05:10 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: display_init inst: 0 successful
Jan 09 00:05:35 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: ######## Guest NVIDIA Driver Information: ########
Jan 09 00:05:35 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: Driver Version: 462.31
Jan 09 00:05:35 archlinux nvidia-vgpu-mgr[3195]: notice: vmiop_log: vGPU version: 0x90001
......

4.2 nvidia-vgpud logs:

➜  vgpu_unlock-rs journalctl -u nvidia-vgpud   
Jan 09 00:01:47 archlinux systemd[1]: Starting NVIDIA vGPU Daemon...
Jan 09 00:01:47 archlinux nvidia-vgpud[482]: Verbose syslog connection opened
Jan 09 00:01:47 archlinux nvidia-vgpud[482]: Started (482)
Jan 09 00:01:47 archlinux systemd[1]: Started NVIDIA vGPU Daemon.
Jan 09 00:01:47 archlinux nvidia-vgpud[482]: Global settings:
Jan 09 00:01:47 archlinux nvidia-vgpud[482]: Size: 16
                                             Version 1
Jan 09 00:01:47 archlinux nvidia-vgpud[482]: Homogeneous vGPUs: 1
Jan 09 00:01:47 archlinux nvidia-vgpud[482]: vGPU types: 517
Jan 09 00:01:47 archlinux nvidia-vgpud[482]: 
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: pciId of gpu [0]: 0:1:0:0
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: 
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Physical GPU:
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: PciID: 0x0000 / 0x0001 / 0x0000 / 0x0000
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Size: 52
                                             Version 1
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: DevID: 0x10de / 0x1e30 / 0x10de / 0x12ba
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Supported vGPUs count: 23
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: 
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Supported VGPU 0x100: max 24
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: VGPU Type 0x100: GRID RTX6000-1Q Class: Quadro
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: DevId: 0x10de / 0x1e30 / 0x10de / 0x1325
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Framebuffer: 0x38000000
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Mappable video size: 0x400000
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Framebuffer reservation: 0x8000000
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: FRL configuration: 0x3c
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: CUDA enabled: 0x1
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: ECC supported: 0x1
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Multi vGPU supported: 0x0
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Encoder Capacity: 0x64
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: BAR1 Length: 0x100
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Frame Rate Limiter enabled: 0x1
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Number of Displays: 4
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Max pixels: 17694720
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Display: width 5120, height 2880
Jan 09 00:01:47 archlinux systemd[1]: Starting NVIDIA vGPU Daemon...
Jan 09 00:01:47 archlinux nvidia-vgpud[482]: Verbose syslog connection opened
Jan 09 00:01:47 archlinux nvidia-vgpud[482]: Started (482)
Jan 09 00:01:47 archlinux systemd[1]: Started NVIDIA vGPU Daemon.
Jan 09 00:01:47 archlinux nvidia-vgpud[482]: Global settings:
Jan 09 00:01:47 archlinux nvidia-vgpud[482]: Size: 16
                                             Version 1
Jan 09 00:01:47 archlinux nvidia-vgpud[482]: Homogeneous vGPUs: 1
Jan 09 00:01:47 archlinux nvidia-vgpud[482]: vGPU types: 517
Jan 09 00:01:47 archlinux nvidia-vgpud[482]: 
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: pciId of gpu [0]: 0:1:0:0
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: 
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Physical GPU:
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: PciID: 0x0000 / 0x0001 / 0x0000 / 0x0000
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Size: 52
                                             Version 1
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: DevID: 0x10de / 0x1e30 / 0x10de / 0x12ba
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Supported vGPUs count: 23
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: 
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Supported VGPU 0x100: max 24
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: VGPU Type 0x100: GRID RTX6000-1Q Class: Quadro
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: DevId: 0x10de / 0x1e30 / 0x10de / 0x1325
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Framebuffer: 0x38000000
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Mappable video size: 0x400000
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Framebuffer reservation: 0x8000000
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: FRL configuration: 0x3c
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: CUDA enabled: 0x1
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: ECC supported: 0x1
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Multi vGPU supported: 0x0
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Encoder Capacity: 0x64
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: BAR1 Length: 0x100
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Frame Rate Limiter enabled: 0x1
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Number of Displays: 4
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Max pixels: 17694720
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Display: width 5120, height 2880
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: License: Quadro-Virtual-DWS,5.0;GRID-Virtual-WS,2.0;GRID-Virtual-WS-Ext,2.0
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: 
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Supported VGPU 0x101: max 12
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: VGPU Type 0x101: GRID RTX6000-2Q Class: Quadro
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: DevId: 0x10de / 0x1e30 / 0x10de / 0x1326
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Framebuffer: 0x74000000
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Mappable video size: 0x400000
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Framebuffer reservation: 0xc000000
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: FRL configuration: 0x3c
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: CUDA enabled: 0x1
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: ECC supported: 0x1
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Multi vGPU supported: 0x0
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Encoder Capacity: 0x64
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: BAR1 Length: 0x100
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Frame Rate Limiter enabled: 0x1
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Number of Displays: 4
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Max pixels: 35389440
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Display: width 7680, height 4320
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: License: Quadro-Virtual-DWS,5.0;GRID-Virtual-WS,2.0;GRID-Virtual-WS-Ext,2.0
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: 
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Supported VGPU 0x102: max 8
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: VGPU Type 0x102: GRID RTX6000-3Q Class: Quadro
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: DevId: 0x10de / 0x1e30 / 0x10de / 0x1327
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Framebuffer: 0xb0000000
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Mappable video size: 0x400000
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Framebuffer reservation: 0x10000000
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: FRL configuration: 0x3c
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: CUDA enabled: 0x1
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: ECC supported: 0x1
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Multi vGPU supported: 0x0
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Encoder Capacity: 0x64
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: BAR1 Length: 0x100
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Frame Rate Limiter enabled: 0x1
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Number of Displays: 4
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Max pixels: 35389440
Jan 09 00:01:48 archlinux nvidia-vgpud[482]: Display: width 7680, height 4320
......

RTX 2060 In proxmox 7.3 not work

root@sm1:~# journalctl -u  nvidia-vgpud
-- Journal begins at Sat 2022-07-02 00:18:51 CST, ends at Mon 2023-02-13 19:45:06 CST. --
Feb 13 18:44:27 sm1 systemd[1]: Starting NVIDIA vGPU Daemon...
Feb 13 18:44:27 sm1 nvidia-vgpud[2009]: Verbose syslog connection opened
Feb 13 18:44:27 sm1 nvidia-vgpud[2009]: Started (2009)
Feb 13 18:44:27 sm1 systemd[1]: Started NVIDIA vGPU Daemon.
Feb 13 18:44:27 sm1 nvidia-vgpud[2009]: Global settings:
Feb 13 18:44:27 sm1 nvidia-vgpud[2009]: Size: 16
                                        Version 1
Feb 13 18:44:27 sm1 nvidia-vgpud[2009]: Homogeneous vGPUs: 1
Feb 13 18:44:27 sm1 nvidia-vgpud[2009]: vGPU types: 653
Feb 13 18:44:27 sm1 nvidia-vgpud[2009]:
Feb 13 18:44:28 sm1 nvidia-vgpud[2009]: pciId of gpu [0]: 0:1:0:0
Feb 13 18:44:28 sm1 nvidia-vgpud[2009]: GPU not supported by vGPU at PCI Id: 0:1:0:0 DevID: 0x10de / 0x1e89 / 0x10de / >
Feb 13 18:44:28 sm1 nvidia-vgpud[2009]: error: failed to send vGPU configuration info to RM: 6
Feb 13 18:44:28 sm1 nvidia-vgpud[2009]: PID file unlocked.
Feb 13 18:44:28 sm1 nvidia-vgpud[2009]: PID file closed.
Feb 13 18:44:28 sm1 nvidia-vgpud[2009]: Shutdown (2009)
Feb 13 18:44:28 sm1 systemd[1]: nvidia-vgpud.service: Main process exited, code=exited, status=6/NOTCONFIGURED
Feb 13 18:44:28 sm1 systemd[1]: nvidia-vgpud.service: Failed with result 'exit-code'.
-- Boot 177f81910c4542cb86de3ae6b863f3b2 --
Feb 13 18:56:11 sm1 systemd[1]: Starting NVIDIA vGPU Daemon...
Feb 13 18:56:11 sm1 nvidia-vgpud[2003]: Verbose syslog connection opened
Feb 13 18:56:11 sm1 nvidia-vgpud[2003]: Started (2003)
Feb 13 18:56:11 sm1 systemd[1]: Started NVIDIA vGPU Daemon.
Feb 13 18:56:11 sm1 nvidia-vgpud[2003]: Global settings:
Feb 13 18:56:11 sm1 nvidia-vgpud[2003]: Size: 16
                                        Version 1
Feb 13 18:56:11 sm1 nvidia-vgpud[2003]: Homogeneous vGPUs: 1

Can we support RTX 30 Series (Ampere, GA102) ?

If can not using a normal driver. Can we use the grid driver(512.78_grid_win10_win11_server2016_server2019_server2022_64bit_international.exe)?

I have a RTX 3090 Ti 24GB card. Let me know if i can help you to debug.

https://github.com/mbilker/vgpu_unlock-rs/blob/master/src/lib.rs#L413 A6000 not work. So i change it to A10:

                // Ampere
                0x2200..=0x2600 => {
                    // A10
                    (0x2236, 0x1482)
                }

It looks like one step away from success:

9月 07 12:36:41 v-amd nvidia-vgpu-mgr[839]: VgpuStart {
                                                 uuid: {00000000-0000-0000-0000-000000000001},
                                                 config_params: "vgpu_type_id=595",
                                                 qemu_pid: 2629,
                                                 unknown_414: [0, 1, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0],
                                             }
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_env_log: vmiop-env: guest_max_gpfn:0x0
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_env_log: Unable to get VM name from QEMU commandline, using VM's UUID as VM name. 0x57
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_env_log: (0x0): Received start call from nvidia-vgpu-vfio module: mdev uuid 00000000-0000-0000-0000-000000000001 GPU PCI id 00:01:00.0 config params vgpu_type_id=595
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_env_log: (0x0): pluginconfig: vgpu_type_id=595
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_env_log: Successfully updated env symbols!
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: cmd: 0x2080014b failed.
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: VgpuConfig {
                                                  vgpu_type: 595,
                                                  vgpu_name: "NVIDIA A10-8Q",
                                                  vgpu_class: "Quadro",
                                                  vgpu_signature: 0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000,
                                                  features: "Quadro-Virtual-DWS,5.0;GRID-Virtual-WS,2.0;GRID-Virtual-WS-Ext,2.0",
                                                  max_instances: 3,
                                                  num_heads: 4,
                                                  max_resolution_x: 7680,
                                                  max_resolution_y: 4320,
                                                  max_pixels: 66355200,
                                                  frl_config: 60,
                                                  cuda_enabled: 1,
                                                  ecc_supported: 1,
                                                  mig_instance_size: 0,
                                                  multi_vgpu_supported: 0,
                                                  vdev_id: 0x223614bd,
                                                  pdev_id: 0x2236,
                                                  fb_length: 0x1dc000000,
                                                  mappable_video_size: 0x400000,
                                                  fb_reservation: 0x24000000,
                                                  encoder_capacity: 0x64,
                                                  bar1_length: 0x100,
                                                  frl_enable: 1,
                                                  adapter_name: "NVIDIA A10-8Q",
                                                  adapter_name_unicode: "NVIDIA A10-8Q",
                                                  short_gpu_name_string: "GA102-A",
                                                  licensed_product_name: "NVIDIA RTX Virtual Workstation",
                                                  vgpu_extra_params: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000,
                                              }
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: Applying profile nvidia-595 overrides
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: Patching nvidia-595/num_heads: 4 -> 1
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: Patching nvidia-595/max_resolution_x: 7680 -> 1920
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: Patching nvidia-595/max_resolution_y: 4320 -> 1080
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: Patching nvidia-595/max_pixels: 66355200 -> 2073600
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: (0x0): gpu-pci-id : 0x100
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: (0x0): vgpu_type : Quadro
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: (0x0): Framebuffer: 0x1dc000000
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: (0x0): Virtual Device Id: 0x2236:0x14bd
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: (0x0): FRL Value: 60 FPS
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: ######## vGPU Manager Information: ########
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: Driver Version: 510.85.03
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: (0x0): vGPU supported range: (0x70001, 0xd0001)
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: cmd: 0x20801322 failed.
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: error: vmiop_log: (0x0): Failed to get blacklisted pages:0x56
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: (0x0): Init frame copy engine: syncing...
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: (0x0): vGPU migration enabled
9月 07 12:36:41 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: display_init inst: 0 successful
9月 07 12:37:09 v-amd nvidia-vgpu-mgr[2647]: cmd: 0x20801322 failed.
9月 07 12:37:09 v-amd nvidia-vgpu-mgr[2647]: error: vmiop_log: (0x0): Failed to get blacklisted pages:0x56
9月 07 12:37:09 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: ######## Guest NVIDIA Driver Information: ########
9月 07 12:37:09 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: Driver Version: 512.78
9月 07 12:37:09 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: vGPU version: 0xd0001
9月 07 12:37:09 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: (0x0): vGPU license state: Unlicensed (Unrestricted)
9月 07 12:37:09 v-amd nvidia-vgpu-mgr[2647]: notice: vmiop_log: (0x0): Guest driver unloaded!

use of unstable library feature 'renamed_spin_loop' Error.

I'm running into the following when running cargo build --release

error[E0658]: use of unstable library feature 'renamed_spin_loop'
 --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.9.3/src/spinwait.rs:9:5
  |
9 | use core::hint::spin_loop;
  |     ^^^^^^^^^^^^^^^^^^^^^
  |
  = note: see issue #55002 <https://github.com/rust-lang/rust/issues/55002> for more information

error[E0658]: use of unstable library feature 'renamed_spin_loop'
  --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.9.3/src/spinwait.rs:16:9
   |
16 |         spin_loop()
   |         ^^^^^^^^^
   |
   = note: see issue #55002 <https://github.com/rust-lang/rust/issues/55002> for more information

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0658`.
error: could not compile `parking_lot_core`.

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error: build failed
```

VM will not start and nvidia-vgpu-mgr spits errors

Driver: NVIDIA-Linux-x86_64-510.47.03-vgpu-kvm with patch (tried with and without patch)
GPU: GTX 1050 Ti
Profile to override:

nvidia-286
    Available instances: 2
    Device API: vfio-pci
    Name: GRID P40-12C
    Description: num_heads=1, frl_config=60, framebuffer=12288M, max_resolution=4096x2160, max_instance=2

Error when booting vm:

mdev instance '00000000-0000-0000-0000-000000000100' already existed, using it.
kvm: -device vfio-pci,sysfsdev=/sys/bus/pci/devices/0000:06:00.0/00000000-0000-0000-0000-000000000100,id=hostpci0,bus=pci.0,addr=0x10: warning: vfio 00000000-0000-0000-0000-000000000100: Could not enable error recovery for the device
kvm: -device vfio-pci,sysfsdev=/sys/bus/pci/devices/0000:06:00.0/00000000-0000-0000-0000-000000000100,id=hostpci0,bus=pci.0,addr=0x10: vfio 00000000-0000-0000-0000-000000000100: failed to read device config space: Bad address
TASK ERROR: start failed: QEMU exited with code 1

systemctl status nvidia-vgpu-mgr

Apr 21 23:30:58 proxmox nvidia-vgpu-mgr[4090]: Patching nvidia-286/pci_id: 456659844 -> 456135072
Apr 21 23:30:58 proxmox nvidia-vgpu-mgr[4090]: Patching nvidia-286/pci_device_id: 6968 -> 6960
Apr 21 23:30:58 proxmox nvidia-vgpu-mgr[4090]: Patching nvidia-286/framebuffer: 12012486656 -> 1968526677
Apr 21 23:30:58 proxmox nvidia-vgpu-mgr[4090]: Patching nvidia-286/frl_enabled: 1 -> 0
Apr 21 23:30:58 proxmox nvidia-vgpu-mgr[4090]: error: vmiop_log: (0x0): Guest BAR1 is of invalid length (g: 0x400000000, h: 0x10000000)
Apr 21 23:30:58 proxmox nvidia-vgpu-mgr[4090]: error: vmiop_log: (0x0): init_device_instance failed for inst 0 with error 1 (error setting vGPU configuration information from RM)
Apr 21 23:30:58 proxmox nvidia-vgpu-mgr[4090]: error: vmiop_log: (0x0): Initialization: init_device_instance failed error 1
Apr 21 23:30:58 proxmox nvidia-vgpu-mgr[4090]: error: vmiop_log: display_init failed for inst: 0
Apr 21 23:30:58 proxmox nvidia-vgpu-mgr[4090]: error: vmiop_env_log: (0x0): vmiope_process_configuration: plugin registration error
Apr 21 23:30:58 proxmox nvidia-vgpu-mgr[4090]: error: vmiop_env_log: (0x0): vmiope_process_configuration failed with 0x1f

/etc/vgpu_unlock/profile_override.toml

[profile.nvidia-286]
num_displays = 1
display_width = 1920
display_height = 1080
max_pixels = 2073600
cuda_enabled = 1
frl_enabled = 0
framebuffer = 1968526677
pci_id = 0x1B3011A0
pci_device_id = 0x1B30

Any help would be greatly appreciated.

have anyone gotten rtx 30 series card to work yet?

Please create new 2.0.1 tag/release

Hey,

could you please create a new tag/release since 2.0.1 is available on master. That will make packaging much easier :)

Thanks,
TheAifam5.

Changing the name of the graphics card does not take effect?

Similar to this, I changed the graphics card to RTX8000, but it still shows RTX6000 on Windows

[profile.nvidia-435]
card_name = "GRID RTX8000"
vgpu_type = "NVS"
pci_id = 0x1e301437
pci_device_id = 0x1e30

VM won't start after shutdown: error allocating framebuffer

Hello, I'm having an issue allocating framebuffer even though there is enough VRAM.

• Host:
  • Ubuntu 22.04.02 LTS
  • Kernel: 5.19.0-35 Generic
  • Nvidia Merged drivers: 525.85.05 (15.1) from vGPU Community Drivers patch (patched by the script)
  • CPU: Xeon E5-2680V4
  • GPU: Quadro M4000 (8GB)
  • RAM: 64GB
  • vGPU Profile:

  nvidia-14
    Available instances: 6
    Device API: vfio-pci
    Name: GRID M60-1B
    Description: num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=8

• Guest1:
  • Windows 11 22h2
    • 16 threads
    • 32GB RAM
    • vGPU slice with these changes in profile_override: (fb values taken from PolloLoco's Proxmox guide)

[mdev.2b6976dd-8620-49de-8d8d-ae9ba47a50df]
# Windows 11 4GB
cuda_enabled = 1
frl_enabled = 0
framebuffer = 0xEC000000
framebuffer_reservation = 0x14000000

• Guest2:
  • Unraid 6.12
    • 8 threads
    • 10GB RAM
    • vGPU slice with these changes in profile_override:

[mdev.bcac6a5c-d5da-4f4d-a31f-ba7bc7de378c]
# Unraid 1GB
cuda_enabled = 1
frl_enabled = 0
framebuffer = 0x38000000
framebuffer_reservation = 0x8000000

How to reproduce:

• Boot host
• Start Unraid VM automatically on boot
• Start Windows VM
• All works as expected and both systems recognize the vGPU slices
• Shutdown both vms
• Start unraid VM
• Start Windows VM
• Windows VM wont start and the error appears

Why would I shutdown unraid? I had to change stuff on the unraid usb, doing it from the gui/cli of unraid will take a while. This usually doesn't happen that often, but I would do that from time to time.

Journalctl log:

Mar 16 17:32:10 X995 nvidia-vgpu-mgr[977]: Nv0000CtrlVgpuGetStartDataParams {
                                               mdev_uuid: {2b6976dd-8620-49de-8d8d-ae9ba47a50df},
                                               config_params: "vgpu_type_id=14",
                                               qemu_pid: 115078,
                                               gpu_pci_id: 0x300,
                                               vgpu_id: 2,
                                               gpu_pci_bdf: 768,
                                           }
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: notice: vmiop_env_log: vmiop-env: guest_max_gpfn:0x0
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: notice: vmiop_env_log: (0x0): Received start call from nvidia-vgpu-vfio module: mdev uuid 2b6976dd-8620-49de-8d8d-ae9ba47a50df GPU PCI id 00:03:00.0 confi>
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: notice: vmiop_env_log: (0x0): pluginconfig: vgpu_type_id=14
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: notice: vmiop_env_log: Successfully updated env symbols!
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: cmd: 0x20801322 failed.
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: cmd: 0x2080014b failed.
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: NvA081CtrlVgpuConfigGetVgpuTypeInfoParams {
                                                  vgpu_type: 14,
                                                  vgpu_type_info: NvA081CtrlVgpuInfo {
                                                      vgpu_type: 14,
                                                      vgpu_name: "GRID M60-1B",
                                                      vgpu_class: "NVS",
                                                      vgpu_signature: [],
                                                      license: "GRID-Virtual-PC,2.0;Quadro-Virtual-DWS,5.0;GRID-Virtual-WS,2.0;GRID-Virtual-WS-Ext,2.0",
                                                      max_instance: 8,
                                                      num_heads: 4,
                                                      max_resolution_x: 5120,
                                                      max_resolution_y: 2880,
                                                      max_pixels: 16384000,
                                                      frl_config: 45,
                                                      cuda_enabled: 0,
                                                      ecc_supported: 0,
                                                      gpu_instance_size: 0,
                                                      multi_vgpu_supported: 0,
                                                      vdev_id: 0x13f21177,
                                                      pdev_id: 0x13f2,
                                                      profile_size: 0x40000000,
                                                      fb_length: 0x38000000,
                                                      gsp_heap_size: 0x0,
                                                      fb_reservation: 0x8000000,
                                                      mappable_video_size: 0x400000,
                                                      encoder_capacity: 0x64,
                                                      bar1_length: 0x100,
                                                      frl_enable: 1,
                                                      adapter_name: "GRID M60-1B",
                                                      adapter_name_unicode: "GRID M60-1B",
                                                      short_gpu_name_string: "GM204GL-A",
                                                      licensed_product_name: "NVIDIA Virtual PC",
                                                      vgpu_extra_params: "",
                                                      ftrace_enable: 0,
                                                      gpu_direct_supported: 0,
                                                      nvlink_p2p_supported: 0,
                                                      multi_vgpu_exclusive: 0,
                                                      exclusive_type: 0,
                                                      exclusive_size: 0,
                                                      gpu_instance_profile_id: 0,
                                                  },
                                              }
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: Applying profile nvidia-14 overrides
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: Patching nvidia-14/num_heads: 4 -> 1
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: Patching nvidia-14/max_resolution_x: 5120 -> 1920
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: Patching nvidia-14/max_resolution_y: 2880 -> 1080
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: Patching nvidia-14/max_pixels: 16384000 -> 2073600
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: Patching nvidia-14/cuda_enabled: 0 -> 1
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: Patching nvidia-14/fb_length: 939524096 -> 939524096
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: Patching nvidia-14/fb_reservation: 134217728 -> 134217728
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: Patching nvidia-14/frl_enable: 1 -> 0
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: Applying mdev UUID 2b6976dd-8620-49de-8d8d-ae9ba47a50df profile overrides
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: Patching nvidia-14/cuda_enabled: 1 -> 1
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: Patching nvidia-14/fb_length: 939524096 -> 3959422976
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: Patching nvidia-14/fb_reservation: 134217728 -> 335544320
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: Patching nvidia-14/frl_enable: 0 -> 0
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: cmd: 0xa0810115 failed.
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: notice: vmiop_log: (0x0): gpu-pci-id : 0x300
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: notice: vmiop_log: (0x0): vgpu_type : NVS
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: notice: vmiop_log: (0x0): Framebuffer: 0xec000000
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: notice: vmiop_log: (0x0): Virtual Device Id: 0x13f2:0x1177
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: notice: vmiop_log: ######## vGPU Manager Information: ########
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: notice: vmiop_log: Driver Version: 525.85.07
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: cmd: 0x2080012f failed.
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: error: vmiop_log: NVOS status 0x51
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: error: vmiop_log: Assertion Failed at 0xc36bc541:143
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: error: vmiop_log: 13 frames returned by backtrace
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: error: vmiop_log: /lib/x86_64-linux-gnu/libnvidia-vgpu.so(_nv008399vgpu+0x35) [0x7fcac370eec5]
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: error: vmiop_log: /lib/x86_64-linux-gnu/libnvidia-vgpu.so(_nv008439vgpu+0x14e) [0x7fcac36bb73e]
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: error: vmiop_log: /lib/x86_64-linux-gnu/libnvidia-vgpu.so(_nv008520vgpu+0xe1) [0x7fcac36bc541]
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: error: vmiop_log: /lib/x86_64-linux-gnu/libnvidia-vgpu.so(+0xd1c67) [0x7fcac36d1c67]
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: error: vmiop_log: /lib/x86_64-linux-gnu/libnvidia-vgpu.so(+0xd4639) [0x7fcac36d4639]
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: error: vmiop_log: vgpu(+0x1893e) [0x56479981893e]
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: error: vmiop_log: vgpu(+0x19a39) [0x564799819a39]
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: error: vmiop_log: vgpu(+0x1389b) [0x56479981389b]
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: error: vmiop_log: vgpu(+0x11116) [0x564799811116]
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: error: vmiop_log: vgpu(+0x3e1a) [0x564799803e1a]
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: error: vmiop_log: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fcac3e29d90]
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: error: vmiop_log: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fcac3e29e40]
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: error: vmiop_log: vgpu(+0x3e5d) [0x564799803e5d]
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: error: vmiop_log: (0x0): Failed to alloc guest FB memory
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: error: vmiop_log: (0x0): init_device_instance failed for inst 0 with error 2 (vmiop-display: error allocating framebuffer)
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: error: vmiop_log: (0x0): Initialization: init_device_instance failed error 2
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: error: vmiop_log: display_init failed for inst: 0
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: error: vmiop_env_log: (0x0): vmiope_process_configuration: plugin registration error
Mar 16 17:32:10 X995 nvidia-vgpu-mgr[115107]: error: vmiop_env_log: (0x0): vmiope_process_configuration failed with 0x1a
Mar 16 17:48:29 X995 nvidia-vgpu-mgr[112701]: notice: vmiop_log: (0x0): vGPU license state: Unlicensed (Unrestricted)

CUDA in win/linux guest not available when device override used

It's a rather curious case that affects CUDA capabilities when there's a device name override. I've got a few Teslas, M40, M40 24G, P100, and P4 and they all seem to exhibit the same issue.

For the windows guest - the acceleration works fine, taking the physical P100 as an example, I can spoof it as Quadro GP100 and the 3d rendering works ok, but CUDA doesn't

For Linux guests (edit: same observed with Windows), as soon as the card device id and the subsystem id get overridden the CUDA capabilities are lost.
Without the override, everything works fine, including CUDA. Seeing that nvidia-smi doesn't report an overridden GPU name (taking it directly from the mdev profile name?) - but lspci does, I suspect that the driver has a dafaq moment and doesn't know whether it's a CUDA capable device or not. But that's just a guess.
Initially I thought it might have been caused by the ROM BAR region being set to only 256M but that's not the case here. Without the override it still shows 256M, so BAR has to be handled somehow differently with vGPU.

The host happily reports the GPU is in C+G mode, indicating both Compute and GFX should be available, so there's something in the device id override that is making CUDA unavailable.

Host smi:

another host smi:

guest SMI (with override active):

CUDA test with an override:

rliwoch@ubuntu-vpu-test:~$ docker run --rm --gpus all nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2
[Vector addition of 50000 elements]
Failed to allocate device vector A (error code all CUDA-capable devices are busy or unavailable)!

Windows CUDA test with an override:

Linux guest, CUDA without override:

rliwoch@ubuntu-vgpu-test:~$ docker run --rm --gpus all nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

I'm posting it as a rather curious case, as I said in the beginning.
I'm not sure if the override is needed in this for Linux Guest and CUDA, it happily took a normal driver (not a GRID one) - so there doesn't seem to be licensing requirements there. Having said that I don't want to start using it and find out that the performance of CUDA is crippled after some time (read somewhere that it can go into the limp mode after a while?)

Any ideas as to why with an override the device suddenly stops accepting CUDA API calls?

for competness:

[profile.nvidia-86]
num_displays = 1
cuda_enabled = 1
frl_enabled = 0     
#Quadro GP100
#pci_device_id = 0x15F0
#pci_id = 0x15F011C3

#Windows box
[mdev.00000000-0000-0000-0000-000000000103]
pci_device_id = 0x15F0
pci_id = 0x15F011C3

#Linux box
[mdev.00000000-0000-0000-0000-000000000102]

Unable to install driver on Debian

Using Proxmox 7.2 I've installed the vGPU unlocker on my host using NVIDIA-Linux-x86_64-510.85.03 drivers and have passed it in to windows with no issues.

I want to do the same on a linux VM, but since I can't do the vGPU spoofing in Linux I went to install the included guest drivers that came with the NVIDIA-Linux-x86_64-510.85.03 driver package.

I'm attempting to install it using dpkg --install , but I keep getting an error saying I haven't installed the linux-headers package. But I've already installed linux-headers-amd64.

Any ideas what I need to do to install the GPU in the guest OS for Linux?

vGPU License

Hi.
I have many questions about NVIDIA vGPU Licensing.

1. Does this still require a nvidia vGPU License ?
2. I just got a 90-day free license and unlocked my vGPU functionality of my consumer grades GPU cards and I used this vGPUs in my Proxmox VMs. Whats happen when that period (90 days) expires?
   Please Help me. Thanks.

The card_name parameter is invalid, the virtual graphics card name will still be displayed

I found that using the card_name parameter is invalid, and using GPU-Z can still display: GRID RTX6000-1B in: WDDM, opengl, and cuda, even if it has been replaced. This ID can be found in the registry

Kernel NULL pointer dereference

I am getting the following kernel error when starting up a KVM guest. The kvm process just hangs. GPU driver version is NVIDIA-Linux-x86_64-460.73.01-grid-vgpu-kvm. Tried kernel 5.13 and 5.15, two different vgpu profiles, overridden and not overridden.

Not really sure where to start with this. Any ideas? Thank you!!!

[  526.065253] BUG: kernel NULL pointer dereference, address: 0000000000000008
[  526.065279] #PF: supervisor read access in kernel mode
[  526.065293] #PF: error_code(0x0000) - not-present page
[  526.065307] PGD 0 P4D 0 
[  526.065316] Oops: 0000 [#1] SMP PTI
[  526.065327] CPU: 19 PID: 3230 Comm: kvm Tainted: P           O      5.15.30-2-pve #1
[  526.065347] Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.1a 10/24/2018
[  526.065365] RIP: 0010:__gup_longterm_locked+0x128/0x430
[  526.065382] Code: b8 ff ff ff ff c7 45 c8 c0 2c 10 00 48 c7 45 a0 00 00 00 00 eb 10 49 8d 76 01 49 39 f4 0f 84 99 00 00 00 49 89 f6 4e 8b 2c f3 <49> 8b 45 08 48 8d 50 ff a8 01 4c 0f 45 ea 4d 39 fd 74 d9 49 8b 45
[  526.065426] RSP: 0018:ffffaeab003dbac8 EFLAGS: 00010246
[  526.065441] RAX: ffffaeab003dbb08 RBX: ffff938829c9a000 RCX: 0000000000000000
[  526.065459] RDX: 0000000000001000 RSI: 00007efa24200000 RDI: ffffda1dc42c3428
[  526.065477] RBP: ffffaeab003dbb60 R08: ffffffffffffffef R09: ffffda1dc42c3400
[  526.065495] R10: 000000010b0d0067 R11: 0000000000000227 R12: 0000000000000200
[  526.065513] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[  526.065531] FS:  00007efe777fe700(0000) GS:ffff93973f6c0000(0000) knlGS:0000000000000000
[  526.065551] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  526.065566] CR2: 0000000000000008 CR3: 0000000119236004 CR4: 00000000003726e0
[  526.065585] Call Trace:
[  526.065592]  <TASK>
[  526.065601]  __get_user_pages_remote+0x55/0x320
[  526.065615]  pin_user_pages_remote+0x1c/0x30
[  526.065629]  vaddr_get_pfns+0x7d/0x260 [vfio_iommu_type1]
[  526.065645]  ? __alloc_pages+0x17b/0x320
[  526.065657]  vfio_pin_pages_remote+0x2ed/0x450 [vfio_iommu_type1]
[  526.065674]  vfio_iommu_type1_ioctl+0x7d4/0x1730 [vfio_iommu_type1]
[  526.065691]  ? __mod_lruvec_state+0x37/0x40
[  526.065705]  ? page_add_new_anon_rmap+0x69/0x100
[  526.065720]  ? set_pte+0x9/0x10
[  526.065729]  ? __handle_mm_fault+0x131f/0x15c0
[  526.065743]  vfio_fops_unl_ioctl+0x6b/0x290 [vfio]
[  526.065758]  __x64_sys_ioctl+0x91/0xc0
[  526.065770]  do_syscall_64+0x5c/0xc0
[  526.065782]  ? exit_to_user_mode_prepare+0x37/0x1b0
[  526.065798]  ? irqentry_exit_to_user_mode+0x9/0x20
[  526.065811]  ? irqentry_exit+0x19/0x30
[  526.066320]  ? exc_page_fault+0x89/0x160
[  526.066832]  ? asm_exc_page_fault+0x8/0x30
[  526.067333]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[  526.067833] RIP: 0033:0x7efe876c6cc7
[  526.068325] Code: 00 00 00 48 8b 05 c9 91 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 99 91 0c 00 f7 d8 64 89 01 48
[  526.069360] RSP: 002b:00007efe777f8ba8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[  526.069889] RAX: ffffffffffffffda RBX: 000055ec6ef7b990 RCX: 00007efe876c6cc7
[  526.070430] RDX: 00007efe777f8bb0 RSI: 0000000000003b71 RDI: 0000000000000021
[  526.070968] RBP: 0000000820000000 R08: 0000000000000000 R09: ffffffffffffffff
[  526.071498] R10: 0000000820000000 R11: 0000000000000246 R12: 0000000010000000
[  526.072017] R13: 0000000820000000 R14: 00007efe777f8bb0 R15: 0000000000000000
[  526.072526]  </TASK>
[  526.073023] Modules linked in: veth tcp_diag inet_diag ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog nfnetlink_log nfnetlink intel_rapl_msr intel_rapl_common sb_edac nvidia_vgpu_vfio(O) mdev x86_pkg_temp_thermal intel_powerclamp nvidia_drm(PO) nvidia_modeset(PO) coretemp zfs(PO) ipmi_ssif kvm_intel zunicode(PO) nvidia(O) zzstd(O) kvm ast zlua(O) zavl(PO) drm_vram_helper icp(PO) crct10dif_pclmul ghash_clmulni_intel drm_ttm_helper ttm snd_hda_intel aesni_intel snd_intel_dspcfg snd_intel_sdw_acpi crypto_simd cryptd rapl snd_hda_codec zcommon(PO) intel_cstate drm_kms_helper znvpair(PO) snd_hda_core joydev input_leds cec spl(O) snd_hwdep rc_core pcspkr fb_sys_fops snd_pcm vhost_net syscopyarea efi_pstore sysfillrect vhost snd_timer sysimgblt vhost_iotlb snd tap soundcore ib_iser rdma_cm iw_cm acpi_ipmi ipmi_si ipmi_devintf ib_cm ipmi_msghandler ib_core iscsi_tcp libiscsi_tcp libiscsi
[  526.073068]  scsi_transport_iscsi mac_hid vfio_pci vfio_pci_core vfio_virqfd irqbypass vfio_iommu_type1 vfio drm sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic xor zstd_compress raid6_pq hid_generic usbkbd usbmouse usbhid hid simplefb dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c ohci_pci crc32_pclmul i2c_i801 lpc_ich xhci_pci i2c_smbus xhci_pci_renesas ohci_hcd igb i2c_algo_bit ehci_pci ahci dca xhci_hcd ehci_hcd libahci wmi
[  526.080188] CR2: 0000000000000008
[  526.080827] ---[ end trace 22ecbdf9784a9b2a ]---
[  526.140694] RIP: 0010:__gup_longterm_locked+0x128/0x430
[  526.141433] Code: b8 ff ff ff ff c7 45 c8 c0 2c 10 00 48 c7 45 a0 00 00 00 00 eb 10 49 8d 76 01 49 39 f4 0f 84 99 00 00 00 49 89 f6 4e 8b 2c f3 <49> 8b 45 08 48 8d 50 ff a8 01 4c 0f 45 ea 4d 39 fd 74 d9 49 8b 45
[  526.142757] RSP: 0018:ffffaeab003dbac8 EFLAGS: 00010246
[  526.143429] RAX: ffffaeab003dbb08 RBX: ffff938829c9a000 RCX: 0000000000000000
[  526.144109] RDX: 0000000000001000 RSI: 00007efa24200000 RDI: ffffda1dc42c3428
[  526.144790] RBP: ffffaeab003dbb60 R08: ffffffffffffffef R09: ffffda1dc42c3400
[  526.145473] R10: 000000010b0d0067 R11: 0000000000000227 R12: 0000000000000200
[  526.146161] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[  526.146839] FS:  00007efe777fe700(0000) GS:ffff93973f6c0000(0000) knlGS:0000000000000000
[  526.147612] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  526.148296] CR2: 0000000000000008 CR3: 0000000119236004 CR4: 00000000003726e0

Profile overrides file shouldn't be required

If profile_overrides.toml doesn't exist, the library errors when booting a VM. It should continue, just with no overrides.

(I'll submit a PR for this, just wanted to throw up an issue so I don't forget)

mdev specific nvidia profile overrides

In PoIIoLocos vGPU Proxmox guide there is chapter about the possibility to override certain VgpuConfig options. Is it possible to override the option [profile.nvidia-XXX] for specific VMs with [mdev.00000000-0000-0000-0000-000000000XXX]? Or is this a global variable which can’t be defined for specific VMs? For example I want to use [profile.nvidia-47]for [mdev.00000000-0000-0000-0000-000000000100] and [profile.nvidia-49]for [mdev.00000000-0000-0000-0000-000000000102]. I don't want use the VMs at the same time because I do unterstand that it's not possible to mix and match profiles. Currently if I want to use a linux guest I will have to use a nvidia b profile because the performance of nvidia q profiles is utterly terrible. This conflicts with the nvidia q profile which I want to use for a windows guest. At line 146 of lib.rs I can see that there are options like gpu_type and vgpu_type. What are these for?