Skip to content

ryohuang/relax-intel-rmrr

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

10 Commits
 
 
 
 
 
 
 
 

Repository files navigation

🍻 Relaxed RMRR Mapping for Linux 3.17+

🐧💨 Now you can use PCI passthrough on broken platforms

TL;DR

When you try to use PCI/PCIe passthrough in KVM/QEMU/Proxmox you get:

vfio-pci 0000:01:00.1: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor.

followed by vfio: failed to set iommu for container: Operation not permitted.

This kernel patch fixes the problem on kernels v3.17 and up (tested up to 5.9.1). You can skip to "Installation" section if you don't care about the rest. Reading of "Disclaimers" section to understand the risks, and "Solutions & hacks" to get the idea of different alternatives is highly recommended.


Table of Contents

  1. Installation
  2. Configuration
  3. Deep Dive - a throughout research on the problem written for mortals
  4. Disclaimers
  5. Acknowledgments & References
  6. License

Installation

Proxmox - premade packages

As I believe in eating your own dog food I run the kernel described here. Thus, I publish precompiled packages.

  1. Go to the releases tab and pick appropriate packages
  2. Download all *.debs packages to the server (you can copy links and use wget https://... on the server itself)
  3. Install all using dpkg -i *.deb in the folder where you downloaded the debs
  4. (OPTIONAL) Verify the kernel works with the patch disabled by rebooting and checking if uname -r shows a version ending with -pve-relaxablermrr
  5. Configure the kernel

Proxmox - building from sources

If you're running a version of Proxmox with no packages available you can compile the kernel yourself using patches provided.

  1. Prepare at least 60GB of free disk space
  2. Install required packages:
    apt update
    apt install git nano screen patch fakeroot build-essential devscripts libncurses5 libncurses5-dev libssl-dev bc flex bison libelf-dev libaudit-dev libgtk2.0-dev libperl-dev asciidoc xmlto gnupg gnupg2 rsync lintian debhelper libdw-dev libnuma-dev libslang2-dev sphinx-common asciidoc-base automake cpio dh-python file gcc kmod libiberty-dev libpve-common-perl libtool perl-modules python-minimal sed tar zlib1g-dev lz4
  3. Download everything:
    mkdir new-kernel ; cd new-kernel
    git clone --depth=1 git://git.proxmox.com/git/pve-kernel.git
    git clone --depth=1 https://github.com/kiler129/relax-intel-rmrr.git
  4. Add kernel patch & patch the toolchain
    cd pve-kernel
    cp ../relax-intel-rmrr/patches/add-relaxable-rmrr-below-5_8.patch ./patches/kernel/CUSTOM-add-relaxable-rmrr.patch
    patch -p1 < ../relax-intel-rmrr/patches/proxmox.patch
  5. Compile the kernel
    make

This step will take a lot of time (30m-3h depending on your machine).

  1. Install new kernel:
    dpkg -i *.deb
  2. (OPTIONAL) Verify the kernel works with the patch disabled by rebooting and checking if uname -r shows a version ending with -pve-relaxablermrr
  3. Configure the kernel

Other distros

  1. Download kernel sources appropriate for your distribution
  2. Apply an appropriate patch to the source tree
    • Go to the folder with your kernel source
    • For Linux 3.17 - 5.7: patch -p1 < ../patches/add-relaxable-rmrr-below-5_8.patch
    • For Linux >=5.8: patch -p1 < ../patches/add-relaxable-rmrr-5_8_and_up.patch
  3. Follow your distro kernel compilation & installation instruction

TODO: Add automation script


Configuration

By default, after the kernel is installed, the patch will be inactive (i.e. the kernel will behave like this patch was never applied). To activate it you have to add intel_iommu=relax_rmrr to your Linux boot args.

In most distros (including Proxmox) you do this by:

  1. Opening /etc/default/grub (e.g. using nano /etc/default/grub)
  2. Editing the GRUB_CMDLINE_LINUX_DEFAULT to include the option:
    • Example of old line:
      GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt intremap=no_x2apic_optout"
      
    • Example of new line:
      GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on,relax_rmrr iommu=pt intremap=no_x2apic_optout"
      
    • Side note: these are actually options which will make your PCI passthrough work and do so efficiently
  3. Running update-grub
  4. Rebooting

To verify if the the patch is active execute dmesg | grep 'Intel-IOMMU' after reboot. You should see a result similar to this:

root@sandbox:~# dmesg | grep 'Intel-IOMMU'
[    0.050195] DMAR: Intel-IOMMU: assuming all RMRRs are relaxable. This can lead to instability or data loss
root@sandbox:~# 

Disclaimers

  • I'm not a kernel programmer by any means, so if I got something horribly wrong correct me please :)
  • This path should be safe, as long as you don't try to remap devices which are used by the IPMI/BIOS, e.g.
    • Network port shared between your IPMI and OS
    • RAID card in non-HBA mode with its driver loaded on the host
    • Network card with monitoring system installed on the host (e.g. Intel Active Health System Agent)
  • This is not a supported solution by any of the vendors. In fact this is a direct violation of Intel's VT-d specs (which Linux already violates anyway, but this is increasing the scope). It may cause crashes or major instabilities. You've been warned.

Acknowledgments & References


License

This work (patches & docs) is dual-licensed under MIT and GPL 2.0 (or any later version), which should be treated as an equivalent of Linux Dual MIT/GPL (i.e. pick a license you prefer).