When you have hardware burning a hole in your desk, and you want to run a VM that can utilize physical hardware such as a Graphics Processing Unit, or GPU, this is how you set it up.
Docs and Blogs on how-to
- ArchLinux Wiki: PCI Passthrough via OVMF
- GitHub: BigOakley/gpu-passthrough
- mrzk.io Blog: Fedora 32 and GPU Passthrough VFIO
- N00b security Blog: Easy GPU Passthrough using KVM on Fedora
Fedora Discussions
You will need to have a second monitor with this setup. You could try to setup Looking Glass or Sunshine+Moonlight if you wanted to.
- Zotac GTX 970 Dual Fan - 145w
[!Info] Make sure you have all updates installed before doing any of this. Makes it go faster!
Run this grep command to find if your CPU has virtualization enabled.
sudo grep --color --regexp vmx --regexp svm /proc/cpuinfoIf there is no result, make sure to enable VT-d for Intel or AMD-V for AMD based motherboards. Consult your hardware's instructions on how to do that.
You may need to remove some Nvidia specific options if you had once used an Nvidia GPU.
sudo rpm-ostree kargs \
--delete-if-present=rd.driver.blacklist=nouveau \
--delete-if-present=modprobe.blacklist=nouveau \
--delete-if-present=nvidia-drm.modeset=1 \
--rebootRun the following command.
dmesg | grep -i -e DMAR -e IOMMUIf this doesn't show, make sure to set:
sudo rpm-ostree kargs \
--append-if-missing="intel_iommu=on" \
--append-if-missing="iommu=pt" \
--append-if-missing="rd.driver.pre=vfio_pci" \
--rebootas a Kernel parameter
Run the following command.
dmesg | grep -i -e DMAR -e IOMMUThis should already be enabled for AMD CPUs. Either way, you'll want to do this.
sudo rpm-ostree kargs \
--append-if-missing="amd_iommu=on" \
--append-if-missing="iommu=pt" \
--append-if-missing="rd.driver.pre=vfio_pci" \
--reboot#!/usr/bin/env bash
shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
echo "IOMMU Group ${g##*/}:"
for d in $g/devices/*; do
echo -e "\t$(lspci -nns ${d##*/})"
done;
done;It should spit out stuff that looks like this...
➜ ~
> bash ./02-check-pci-bus-groups.sh
IOMMU Group 0:
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 1:
00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
[...lots of output...]
IOMMU Group 32:
0d:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 22 [Radeon RX 6700/6700 XT/6750 XT / 6800M] [1002:73df] (rev c1)
IOMMU Group 33:
0d:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]
IOMMU Group 34:
0e:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
0e:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
[...more output...]You can also get the device IDs using lspci. In this case, I'm looking for my NVIDIA card to pass-through.
➜ ~
> lspci -vnn | grep -i --regexp NVIDIA
0e:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1) (prog-if 00 [VGA controller])
0e:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
➜ ~
> lspci -vnn | grep -i --regexp Radeon
0d:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 22 [Radeon RX 6700/6700 XT/6750 XT / 6800M] [1002:73df] (rev c1) (prog-if 00 [VGA controller])Now we'll setup the Kernel Args for disabling the PCI Buses for the GPU.
sudo rpm-ostree kargs \
--append-if-missing="vfio-pci.ids=10de:13c2,10de:0fbb" \
--rebootThen perform a dracut to make sure the initramfs has the Kernel module loaded.
sudo rpm-ostree initramfs \
--enable \
--arg="--add-drivers" \
--arg="vfio-pci" \
--rebootYou should now see when you perform
sudo lspci -nnvShould show something similar to...
➜ ~
> sudo lspci -vnn
[sudo] password for filbot:
[...lots of output...]
0e:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1) (prog-if 00 [VGA controller])
Subsystem: ZOTAC International (MCO) Ltd. Device [19da:1366]
Flags: bus master, fast devsel, latency 0, IRQ 255, IOMMU group 34
Memory at f5000000 (32-bit, non-prefetchable) [size=16M]
Memory at c0000000 (64-bit, prefetchable) [size=256M]
Memory at d0000000 (64-bit, prefetchable) [size=32M]
I/O ports at e000 [size=128]
Expansion ROM at f6000000 [disabled] [size=512K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Legacy Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [250] Latency Tolerance Reporting
Capabilities: [258] L1 PM Substates
Capabilities: [128] Power Budgeting <?>
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900] Secondary PCI Express
Kernel driver in use: vfio-pci
Kernel modules: nouveau
0e:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
Subsystem: ZOTAC International (MCO) Ltd. Device [19da:1366]
Flags: bus master, fast devsel, latency 0, IRQ 131, IOMMU group 34
Memory at f6080000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
[...lots of output...][!Warning] If it doesn't say
vfio-pciThen you're doing something wrong.
You can also check if the vfio Kernel modules made it in the initramfs by using:
➜ ~
> sudo lsinitrd /boot/ostree/fedora-af7516f20c0bee3df3aa77532024d9fb7c207d1b7246d1d4949b047d7ab916d9/initramfs-6.0.15-300.fc37.x86_64.img| grep -i --regexp vfio
Arguments: --reproducible -v --add 'ostree' --tmpdir '/tmp/dracut' -f --no-hostonly --kver '6.0.15-300.fc37.x86_64' --reproducible -v --add 'ostree' --tmpdir '/tmp/dracut' -f --no-hostonly --add-drivers ' vfio-pci' --kver '6.0.15-300.fc37.x86_64'
drwxr-xr-x 3 root root 0 Dec 31 1969 usr/lib/modules/6.0.15-300.fc37.x86_64/kernel/drivers/vfio
drwxr-xr-x 2 root root 0 Dec 31 1969 usr/lib/modules/6.0.15-300.fc37.x86_64/kernel/drivers/vfio/pci
-rw-r--r-- 1 root root 32140 Dec 31 1969 usr/lib/modules/6.0.15-300.fc37.x86_64/kernel/drivers/vfio/pci/vfio-pci-core.ko.xz
-rw-r--r-- 1 root root 8596 Dec 31 1969 usr/lib/modules/6.0.15-300.fc37.x86_64/kernel/drivers/vfio/pci/vfio-pci.ko.xz
-rw-r--r-- 1 root root 19884 Dec 31 1969 usr/lib/modules/6.0.15-300.fc37.x86_64/kernel/drivers/vfio/vfio_iommu_type1.ko.xz
-rw-r--r-- 1 root root 14880 Dec 31 1969 usr/lib/modules/6.0.15-300.fc37.x86_64/kernel/drivers/vfio/vfio.ko.xz
-rw-r--r-- 1 root root 3552 Dec 31 1969 usr/lib/modules/6.0.15-300.fc37.x86_64/kernel/drivers/vfio/vfio_virqfd.ko.xzThe /boot/ostree/fedora-???/initramfs-x.y.z-aaa.fc??.x86_64.img depends on what image is live. You may have two or three depending on how many backup images you keep of your ostree setup.
[!Info] Optional Stuff This is entirely optional. You don't need it.
- getlabsdone.com - How to connect kvm mv to host network
- getlabsdone.com - How to create a Bridge on Fedora
On Fedora, I installed bridge-utils.
sudo rpm-ostree install bridge-utils --rebootThen follow the blogs listed. However, this will make your host present a different IP address and route traffic differently. I think my issue for my Windows VM was the disk size. See setting up the Virtual Machine further on.
In this we're using qemu-kvm with virt-manager as a GUI to help us.
As of this writing, I'm using qemu-kvm version 7.0.0. This should be sufficient.
➜ ~
> qemu-kvm --version
QEMU emulator version 7.0.0 (qemu-7.0.0-12.fc37)
Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developersAt the time of this writing I'm using Windows 10 and I am downloading it from this link. Verify it before clicking.
[!Warning] Verify your ISO Download Remember to verify the ISO download you're performing!
This is going to use virt-manager or Virtual Machine Manager. Either way it's the GUI of the command-line application virt-install.
- Open
virt-manager. - Create a new Virtual Machine. Before installing, be sure to customize the configuration.
- Provide at least 8GB of RAM.
- Provide at least 4 CPUs.
- Provide at least 500GB of Storage.
- This is because Windows is stupid and huge.
- Ensure the following settings are set.
- Overview -> Chipset: Q35
- Overview -> Firmware: UEFI/OVMF
- Add Hardware -> PCI Host Device -> NVIDIA GPU
- Add Hardware -> PCI Host Device -> NVIDIA High Def Audio.
- Enable the SATA CD Drive.
- Change the boot order so that SATA CD Drive is first.
- Set CPUs -> Topology -> Sockets 1, Cores 4, Threads 2.
- This makes 8 Virtual Processors.
- Apply the changes.
- Begin Installation
Setup Windows as you can do the best. I had to move my monitor, then adjust the position of screens, update a bunch and then I could finally move on.
Domain is what virsh calls a Virtual Machine. We'll edit the XML for the VM. I use the Neovim Flatpak.
EDITOR=io.neovim.nvim virsh \
--connect=qemu:///system edit Windows-GamingHere, we'll insert the few elements.
<domain type='kvm'>
...
<features>
<kvm>
<hidden state='on' />
</kvm>
<hyperv>
<vendor_id state='on' value='whatever' />
</hyperv>
</features>
...
</domain>You'll need to have the VM powered off for these changes to take effect.
[!Warning] Secure Boot UEFI I disabled this in the VM's Bios. I had to enable the boot menu, then press F12.
This is from:
This will need to be downloaded on the Windows PC. Then reboot.
Download the official NVIDIA drivers.
Download and install SteamSetup.exe from:
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent