domains

A collection of (working/tested) domain/VM XML configuration files focused on integrated GPU dedicated passthrough (GVT-D "Legacy" mode on Intel 11700k).

Background

The 11700k includes an integrated graphics device ([8086:4c8a] 00:02.0 VGA compatible controller: Intel Corporation RocketLake-S GT1 [UHD Graphics 750]) which supports GVT-D but has lacked clear examples/tutorials of how to passthrough the GPU to a VM. After many attempts and failures, here are domain config with GVT-D working using OVMF with the GPU in apparent "Legacy" mode.

Hardware Configuration

• Intel Core i7 11700k
• ASUS Prime Z590-A
• Trident Z Neo DDR4-3600 CL16-19-19-39 1.35V 16GB x 2
• ASUS DUAL RTX 3070 OC 8GB
• WD Black SN750 NVMe SSD 1TB (Unraid Cache)
• WD Black SN850 NVMe SSD 1TB (Resevered for VM Passthrough)
• Fresco FL1100 4x USB 3.0 PCI Express Card PCI-e USB3.0 Host Controller Adapter (Resevered for VM passthrough; bus='0x07' slot='0x00' function='0x0)
• Marvell 88SE9215 PCIe 2.0 x1 4-port SATA 6 Gb/s Controller

6+1 Unraid Array (7 x 4TB SATA drives):

• Parity: ST4000VN008
• 1 x WDC_WD40EFRX
• 1 x ST4000DM000
• 3 x ST4000DM004
• 1 x ST4000VN008

Requirements:

• Host OS: Linux

  • Tested:
    • Unraid 6.11.5 (Kernel: 5.19.17-Unraid)
      • should work with most modern kernels/distros
    • Libvirt version 8.7.0
    • QEMU version 7.1.0

• Kernel modules blacklisted modprobe.blacklist=i2c_i801,i2c_smbus,snd_hda_intel,snd_hda_codec_hdmi,i915,drm,drm_kms_helper,i2c_algo_bit

  • Possibly excessive; more testing required

• EFI & VESA Framebuffers off video=efifb:off,vesafb:off

  • Possibly optional; more testing required

• (Likely optional) Both PCIe ACS overrides and allow VFIO unsafe interrupts pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1

  • used for passing through associated audio device [8086:43c8] 00:1f.3 Audio device: Intel Corporation Tiger Lake-H HD Audio Controller
  • may be important to allowing 00:1f.0 to be bound to VFIO

• PCI devices bound to VFIO:

  • 11700k/PRIME Z590-A devices:

  [8086:4c8a] 00:02.0 VGA compatible controller: Intel Corporation RocketLake-S GT1 [UHD Graphics 750]
  [8086:4385] 00:1f.0 ISA bridge: Intel Corporation Device 4385
  [8086:43c8] 00:1f.3 Audio device: Intel Corporation Tiger Lake-H HD Audio Controller
  [8086:43a3] 00:1f.4 SMBus: Intel Corporation Tiger Lake-H SMBus Controller
  [8086:43a4] 00:1f.5 Serial bus controller: Intel Corporation Tiger Lake-H SPI Controller
  

  • Devices specific to this configuration:

  [10de:2484] 01:00.0 VGA compatible controller: NVIDIA Corporation GA104 [GeForce RTX 3070]
  [10de:228b] 01:00.1 Audio device: NVIDIA Corporation GA104 High Definition Audio Controller
  [15b7:5011] 03:00.0 Non-Volatile memory controller: Sandisk Corp WD PC SN810 / Black SN850 NVMe SSD
  [1b73:1100] 07:00.0 USB controller: Fresco Logic FL1100 USB 3.0 Host Controller (rev 10)
  

• Secondary GPU or device to log in to Guest OS/VM via VNC/RDP

• Integrated graphics physically connected to powered-on monitor on HDMI port (be patient, watch for changes, always have a remote login method available)

• Specific QEMU XML:

  • <domain> tag must be updated to add QEMU namespace (replace id as appropriate):

  <domain type='kvm' id='#' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  

  • override integrated passthrough settings (add before </domain>):

  <qemu:override>
    <qemu:device alias='hostdev0'>
      <qemu:frontend>
        <qemu:property name='x-igd-opregion' type='bool' value='true'/>
        <qemu:property name='driver' type='string' value='vfio-pci-nohotplug'/>
      </qemu:frontend>
    </qemu:device>
  </qemu:override>
  

• (Windows Guests/VMs) ROM file: vbios_gvt_uefi.rom 😬

  • Hopefully, there is a better option because i915ovmf.rom did not work in testing

Configurations

Windows 10 - 6 Core (isolated) passthrough - NVIDIA passthrough - GVT-D

GVT-D WORKING:

• DP & HDMI output at 4K60 (3840x2160 @ 60Hz x 2)
• HDR

GVT-D NOT-WORKING (Yet?):

• HDMI audio does not appear to work in Windows nor do any of the analog audio ports
  • The SPDIF shows as working but has not been tested

NOTES:

• This VM uses 6 isolated cores:
  • Unraid boots using cores 0 and 7 (and HT) so that leaves cores 1-6 (and HT) isolated for VM usage

  isolcpus=1-6,9-14 nohz_full=1-6,9-14
  

  • see <vcpu>, <cputune>, and <cpu> blocks for related configuration
• Adding TPM emulator device and changing loader to OVMF TPM will fulfill requirements for Windows 11 upgrade
  • Driver re-install via RDP may be required after upgrade
• <qemu:commandline> block is entirely optional but <qemu:override> block is critical for GVT-D
• ROM file is required for Intel device to enable physical display on Windows
  • /mnt/cache/isos/ is not the best location for the ROM, this will be moved
• <name>Windows 10 (EXPERIMENTAL CONFIG 2)</name> will be updated in the near future
• see commit comments for details/history

Installation process:

• Set VNC as the only graphics device in XML, save, review to ensure changes saved successfully, start VM
• Install Windows, enable and test RDP login, and shutdown the VM
• With RDP enabled, remove VNC graphics device and add integrated graphics device to XML, save XML, review to ensure changes saved successfully, start VM
• Login with RDP and install Intel graphics driver (as many times as necessary until error goes away and monitor shows output; 2-3 times in my experience)
• Ensure Device Manager shows all devices working properly
• Install NVIDIA graphics drivers and shutdown VM
• Add NVIDIA graphics device to XML, save, review to ensure changes saved successfully, start VM
• Dual GPU passthrough should now work (4x4K tested: NVIDIA: DPx2 & Intel DPx1 HDMIx1)