QEMU/KVM NVIDIA GPU Passthrough with AMD CPU in Arch Linux
Estimated reading time: 4 minutesThis document catalogs my working setup for configuring an Arch Linux host OS to run a Windows 10 VM while passing through a virtualised NVIDIA GPU with a AMD CPU.
Requirements:
Make sure you have 2 or more GPUs CPU with multiple cores and virtualization support, this can be checked with
lscpu | grep "Virtualization"
If not, go to BIOS and enable it
Prerequisites
# Download the required packages
yay -S qemu-full libvirt virt-manager virt-viewer bridge-utils
# Add yourself into groups so to solve permission issues
usermod -aG libvirt,kvm,libvirt-qemu,input,disk,audio,video [your username]
# Enable systemd service
systemctl enable libvirtd.socket
Install VM
Start Virtual Machine Manager(virt-manager) At VM Manager, you may not see there is QEMU/KVM - User session. For that, click File->Add Connection. Choose QEMU/KVM user session and press connect.
Creating the VM
Click plus symbol on top left corner
- Choose Local install media
- Choose the ISO file
- Pass half of your RAM and half of CPU cores
- If you choose to use qcow2 format, more than 50 GB is recommended, of course you can use a dedicated drive
- Make sure to click Customize confirmation before install
This will create a new window with your VM configuration, make sure chipset is Q35 and boot firmware is UEFI (it should be like UEFI x86_64: /usr/share/edk2/x64/OVMF_CODE.secboot.4m.fd
). After that, go to CPU section, untick copy host CPU configuration, make sure Model is host-passthrough. Open Topology sub-menu and click manually set CPU topology, then set 1 socket, half of your cores, 1 thread.
Isolate GPU
- Identify PCI ID of GPU
lspci -nn | grep -E "NVIDIA"
# Example output
# 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU117M [GeForce GTX 1650 Ti Mobile] [10de:1f95] (rev a1)
# 01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10fa] (rev a1)
# Copy the ID inside the square brackets and seperate them with a comma
# 10de:1f95,10de:10fa
# or alternative
shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
echo "IOMMU Group ${g##*/}:"
for d in $g/devices/*; do
echo -e "\t$(lspci -nns ${d##*/})"
done;
done;
- Enable IOMMU for AMD CPU
# In your /etc/default/grub, add these parameters to GRUB_CMDLINE_LINUX_DEFAULT(kernel parameters)
amd_iommu=on iommu=pt vfio-pci.ids=(ids you copied, without bracket)
# Example config
# GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amd_iommu=on iommu=pt vfio-pci.ids=10de:1f95,10de:10fa"
# Then update you grub config by
sudo grub-mkconfig -o /boot/grub/grub.cfg
reboot
- Edit mkinitcpio.conf
# Add these to MODULES
# vfio vfio_iommu_type1 vfio_pci
# Example Config
# MODULES=(vfio vfio_iommu_type1 vfio_pci amdgpu nvidia nvidia_modeset nvidia_uvm nvidia_drm)
# Add modconf to HOOKS
# Example Config
# HOOKS=(base udev plymouth autodetect modconf kms keyboard keymap consolefont block filesystems fsck)
- Edit modprobe
sudo touch /etc/modprobe.d/vfio.conf
sudoedit /etc/modprobe.d/vfio.conf
# Type these things in
options vfio-pci ids=(ids you copied, without bracket)
# Example config
options vfio-pci ids=10de:1f95,10de:10fa
# After that run
sudo mkinitcpio -p linux
reboot
- Prove that GPU have been isolated
lspci -k | grep -E "vfio-pci|NVIDIA"
# You should see output if it is successful
- Download virtio Windows drivers
Go to virtio Github README and download Stable virtio-win ISO
- Add virtio drivers
At VM Config, go to CDROM and import virtio drivers by selecting the ISO file.
- Add PCI Host Devices
At VM Config, click “Add Hardware” and select PCI Host Device then choose both NVIDIA devices. Then run the VM, first you might encounter some permissions issue(I did), for that try edit /etc/libvirt/qemu.conf
, add these lines
user = "night"
group = "libvirt-qemu"
cgroup_device_acl = [
"/dev/vfio/9", "/dev/null", "/dev/full", "/dev/zero",
"/dev/random", "/dev/urandom",
"/dev/ptmx", "/dev/kvm"
]
If you face memory issue, edit /etc/security/limits.conf
and add
night hard memlock 28388608
or try
sudo chmod a+rw /dev/vfio/9
- Add Drivers in Windows
Run the VM and navigate to the mounted CD-ROM and run virtio-win-guest-tools and go through the menu Reboot After reboot, go to NVIDIA and download official GPU drivers Then go to task manager to see if GPU exist
- Optimise VM
Shutdown the VM, go to VM configuration. At Video, normally “Video QXL”, change QXL to virtio. Then click Add Hardware, navigate to Channel and add org.qemu.guest_agent. Run the VM and check the State is connected at the guest agent channel
- If mouse don’t work
Navigate to CD-ROM using keyboard and execute virtio-win-gt-x64 using spacebar to navigate menu Choose Remove from the menu Reboot
- Passthrough disks
Run
virt-viewer --connect=qemu:///session --domain-name [vm name]
File->Preferences Click Share folder and choose the folder you want to share to the VM and click X and wait
Note: Right click at QEMU/KVM User session and Click Details Copy the Libvirt URI to –connect Domain name is the VM name
References
https://christitus.com/vm-setup-in-linux/
https://christitus.com/windows-inside-linux/
https://christitus.com/vm-setup-in-linux/
https://christitus.com/setup-qemu-in-archlinux/
https://github.com/tuh8888/libvirt_win10_vm
https://github.com/BeanGreen247/ArchLinux-KDE-Plasma-setup-script/blob/main/vfio-readme.md
https://askubuntu.com/questions/1406888/ubuntu-22-04-gpu-passthrough-qemu
https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
https://wiki.archlinux.org/title/QEMU/Guest_graphics_acceleration
https://www.reddit.com/r/VFIO/comments/fxvt2d/help_with_seamless_gpu_passthrough_guest_gpu/