Support Intel IGD(Integrated Graphics Device) passthrough about kubevirt

[victortoso]

The VMI spec is missing. Have you set the externalResourceProvider to true ? from the docs.

[chenxinlong]

The VMI spec is missing. Have you set the externalResourceProvider to true ? from the docs.

Yes.

VM Spec :

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  annotations:
    kubevirt.io/latest-observed-api-version: v1
    kubevirt.io/storage-observed-api-version: v1
  creationTimestamp: "2024-02-26T06:13:46Z"
  finalizers:
  - kubevirt.io/virtualMachineControllerFinalize
  generation: 46
  name: vm-2o8x6yms
  namespace: hc-default-14
  resourceVersion: "1162561"
  uid: 35ccc0b3-c6d4-4d82-bf73-2444c5863698
spec:
  runStrategy: RerunOnFailure
  template:
    metadata:
      annotations:
        cni.projectcalico.org/hwAddr: 12:38:dd:a9:b1:eb
        cni.projectcalico.org/ipAddrs: '["100.67.166.175"]'
        hooks.kubevirt.io/hookSidecars: '[{"image":"example.com/hook-sidecar:igd"}]'
      creationTimestamp: null
    spec:
      architecture: amd64
      domain:
        cpu:
          cores: 1
          model: host-passthrough
        devices:
          clientPassthrough: {}
          disks:
          - bootOrder: 1
            disk:
              bus: virtio
            name: sysdisk
          - disk:
              bus: virtio
            name: cloudinitdisk
          - cdrom:
              bus: sata
            name: virtiocontainerdisk
          gpus:
          - deviceName: gpu.example.com/IGD
            name: gpu0
          inputs:
          - bus: usb
            name: tablet
            type: tablet
          interfaces:
          - bridge: {}
            name: default
        firmware:
          bootloader:
            efi:
              secureBoot: false
        machine:
          type: q35
        memory:
          guest: 1G
        resources: {}
      networks:
      - name: default
        pod: {}
      volumes:
      - name: sysdisk
        persistentVolumeClaim:
          claimName: vol-ntwvqx6q
      - cloudInitNoCloud:
          userDataBase64: xxx
        name: cloudinitdisk
      - containerDisk:
          image: example.com/virtio-container-disk:v0.38.1
        name: virtiocontainerdisk
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-02-28T02:10:32Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: null
    message: 'cannot migrate VMI: PVC vol-ntwvqx6q is not shared, live migration requires
      that all PVCs must be shared (using ReadWriteMany access mode)'
    reason: DisksNotLiveMigratable
    status: "False"
    type: LiveMigratable
  - lastProbeTime: "2024-02-28T02:11:02Z"
    lastTransitionTime: null
    status: "True"
    type: AgentConnected
  created: true
  desiredGeneration: 46
  observedGeneration: 45
  printableStatus: Running
  ready: true
  volumeSnapshotStatuses:
  - enabled: false
    name: sysdisk
    reason: 2 matching VolumeSnapshotClasses for default-btrfs
  - enabled: false
    name: cloudinitdisk
    reason: Snapshot is not supported for this volumeSource type [cloudinitdisk]
  - enabled: false
    name: virtiocontainerdisk
    reason: Snapshot is not supported for this volumeSource type [virtiocontainerdisk]

@victortoso Hi, has your team passed through Intel IGD successfully ? I've tried a lot of solution on the internet, but I still can't achieve it.

[victortoso]

I've tried a lot of solution on the internet, but I still can't achieve it

If it works with libvirt + qemu without KubeVirt, we should be able to make it work in KubeVirt too.

First, wrt to the device-plugin, you are sure it is working, correct? when you do a kubectl describe node you can see it there?

Second, we need to understand what is missing from KubeVirt side in order to make Intel IGD work. For example, I did a quick look at this reddit question and it might be necessary to add more configurations to the libvirt domain.

These are the current supported fields in schema.go, if you need to set something else, you'll need to use a hook sidecar to tweak the libvirt configuration prior to the VM being created.

So, we need to know what is missing in order to move forward. I have the following at hand, but I can't test it now.

toso@tapioca ~/s/k/kubevirt (main) [1]> lspci -s 00:02.0
00:02.0 VGA compatible controller: Intel Corporation CometLake-U GT2 [UHD Graphics] (rev 02)

[victortoso]

hooks.kubevirt.io/hookSidecars: '[{"image":"example.com/hook-sidecar:igd"}]'

Ah, you are already using the sidecar to tweak libvirt?
Any error message to share then?

Did you enable intel_iommu? (looking at the docs)

[chenxinlong]

1. Preparation

Did you enable intel_iommu

Yes, I've enabled the iommu.

root@node1:~# cat /etc/default/grub | grep GRUB_CMDLINE_LINUX
GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0 intel_iommu=on amd_iommu=on iommu=pt nouveau.modeset=0 video=vesafb:off,efifb:off "
root@node1:~#
root@node1:~# dmesg | grep -i "iommu enable"
[    0.030010] DMAR: IOMMU enabled

Switch the driver of Intel IGD from i915 to vfio-pci :

root@node1:~# echo vfio-pci > /sys/bus/pci/devices/0000\:00\:02.0/driver_override
root@node1:~# echo 0000:00:02.0 > /sys/bus/pci/devices/0000\:00\:02.0/driver/unbind
root@node1:~# echo 0000:00:02.0 > /sys/bus/pci/drivers/vfio-pci/bind
root@node1:~#
root@node1:~# lspci -s 00:02.0 -nnDk
0000:00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:46d1]
	DeviceName: Onboard - Video
	Subsystem: Device [0301:02f3]
	Kernel driver in use: vfio-pci
	Kernel modules: i915

2. Write a device plugin

First, wrt to the device-plugin, you are sure it is working, correct? when you do a kubectl describe node you can see it there?

Yes, I'm sure it works.

root@node1:~# kubectl describe node node1
Name:               node1
Roles:              control-plane,worker
...
Capacity:
  cpu:                              4
  devices.kubevirt.io/kvm:          1k
  devices.kubevirt.io/tun:          1k
  devices.kubevirt.io/vhost-net:    1k
  devices.kubevirt.io/vhost-vsock:  1k
  ephemeral-storage:                71645Mi
  gpu.example.com/IGD:              1
  hugepages-1Gi:                    0
  hugepages-2Mi:                    0
  memory:                           7839532Ki
  pods:                             110
Allocatable:
  cpu:                              3600m
  devices.kubevirt.io/kvm:          1k
  devices.kubevirt.io/tun:          1k
  devices.kubevirt.io/vhost-net:    1k
  devices.kubevirt.io/vhost-vsock:  1k
  ephemeral-storage:                71645Mi
  gpu.example.com/IGD:              1
  hugepages-1Gi:                    0
  hugepages-2Mi:                    0
  memory:                           7122732Ki
  pods:                             110

Create a windows vm with gpu.example.com/IGD required :

root@node1:~# kg vm -n  hc-default-14   vm-2o8x6yms -o yaml
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  annotations:
    kubevirt.io/latest-observed-api-version: v1
    kubevirt.io/storage-observed-api-version: v1
  creationTimestamp: "2024-02-26T06:13:46Z"
  finalizers:
  - kubevirt.io/virtualMachineControllerFinalize
  generation: 46
  name: vm-2o8x6yms
  namespace: hc-default-14
  resourceVersion: "1458695"
  uid: 35ccc0b3-c6d4-4d82-bf73-2444c5863698
spec:
  runStrategy: RerunOnFailure
  template:
    metadata:
      annotations:
        cni.projectcalico.org/hwAddr: 12:38:dd:a9:b1:eb
        cni.projectcalico.org/ipAddrs: '["100.67.166.175"]'
        hooks.kubevirt.io/hookSidecars: '[{"image":"images.example.com/hook-sidecar:igd"}]'
      creationTimestamp: null
    spec:
      architecture: amd64
      domain:
        cpu:
          cores: 1
          model: host-passthrough
        devices:
          clientPassthrough: {}
          disks:
          - bootOrder: 1
            disk:
              bus: virtio
            name: sysdisk
          - disk:
              bus: virtio
            name: cloudinitdisk
          - cdrom:
              bus: sata
            name: virtiocontainerdisk
          gpus:
          - deviceName: gpu.example.com/IGD
            name: gpu0
          inputs:
          - bus: usb
            name: tablet
            type: tablet
          interfaces:
          - bridge: {}
            name: default
        firmware:
          bootloader:
            efi:
              secureBoot: false
        machine:
          type: q35
        memory:
          guest: 1G
        resources: {}
      networks:
      - name: default
        pod: {}
      volumes:
      - name: sysdisk
        persistentVolumeClaim:
          claimName: vol-ntwvqx6q
      - cloudInitNoCloud:
          userDataBase64: xxx
        name: cloudinitdisk
      - containerDisk:
          image: images.example.com/virtio-container-disk:v0.38.1
        name: virtiocontainerdisk
Status:
   ...

The Allocate() api implementation of my device plugin :

func (dp *GenericPCIDevicePlugin) Allocate(ctx context.Context, reqs *pluginapi.AllocateRequest) (*pluginapi.AllocateResponse, error) {
	... 

        responses.ContainerResponses[0].Envs["PCI_RESOURCES_GPU_EXAMPLE_COM_IGD"] = "0000:00:02.0"
        responses.ContainerResponses[0].Devices = append(resp.Devices, &pluginapi.DeviceSpec{
	        HostPath:      "/dev/vfio",
	        ContainerPath: "/dev/vfio",
	        Permissions:   "mrw",
        })

	return responses, nil
}

After the virt-launcher pod running, make sure the related environment and the domain xml is correct :

root@node1:~# kubectl exec -it virt-launcher-vm-2o8x6yms-wvvb9 -n hc-default-14 -- env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=vm-2o8x6yms
PCI_RESOURCE_GPU_EXAMPLE_COM_IGD=0000:00:02.0
...

root@node1:~# kubectl exec -it virt-launcher-vm-2o8x6yms-wvvb9 -n hc-default-14 -- virsh dumpxml 1 | grep -i -10 ua-gpu-gpu0
    ...
    <hostdev mode='subsystem' type='pci' managed='no'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
      </source>
      <alias name='ua-gpu-gpu0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </hostdev>
    ...

3. hook-sidecar

Ah, you are already using the sidecar to tweak libvirt?

Yes, I'm already using hook-sidecar to tweak the domain xml to ensure the guest IGD pci address is 0000:00:02.0

Any error message to share then?

No, everything goes well.

[victortoso]

Thanks for all the info. I'll give a try when time allows for it.
/cc @vladikr, any ideas from the top of your head on why it seems it should work but doesn't without errors/etc ?