BAR Allocation Failed & IOMMU Conflicts: Dual GPU (RTX 5060 + 4060) on Ryzen 5800X/B550 - "No Space" Errors
https://askubuntu.com/questions/1563112/bar-allocation-failed-iommu-conflicts-dual-gpu-rtx-5060-4060-on-ryzen-580
I am hitting a wall trying to configure a dual GPU setup on a B550/Ryzen platform running Ubuntu. I am facing a persistent Memory Mapped I/O (MMIO) resource allocation issue where the kernel cannot assign BAR addresses for the second GPU, leading to can't assign; no space errors.
Hardware Specs:
CPU: AMD Ryzen 7 5800X
Mobo: B550 Chipset [INSERT YOUR MOTHERBOARD MODEL HERE]
GPU 1: NVIDIA RTX 4060
GPU 2: NVIDIA RTX 5060 (New gen)
PSU: 850W
OS: Ubuntu (Kernel 6.x generic)
Driver: Manual install of NVIDIA-Linux-x86_64-580.126.09.run (Required for 5060 support)
The Problem: When booting with default BIOS settings, only one GPU is enumerated correctly. The second GPU fails with BAR allocation errors in dmesg. When I attempt to fix this by enabling "Above 4G Decoding" in BIOS, the system fails to mount the NVMe drive or panics during boot unless I use specific kernel parameters, and even then, I lose the proprietary driver binding.
Logs & Symptoms:
1. The BAR Error (dmesg output): Without 4G decoding, the kernel clearly sees the device but cannot map the memory:
Bash
[ 0.666193] pci 0000:07:00.0: BAR 1 [mem size 0x10000000 64bit pref]: can't assign; no space
[ 0.666195] pci 0000:07:00.0: BAR 1 [mem size 0x10000000 64bit pref]: failed to assign
[ 0.666196] pci 0000:07:00.0: VF BAR 2 [mem size 0x10000000 64bit pref]: can't assign; no space
NVRM: BAR1 is 0M @ 0x0 (PCI:0000:04:00.0)
2. The IOMMU/Disk Conflict: If I Enable Above 4G Decoding and Disable CSM (UEFI only), the boot process hangs with filesystem errors, losing access to the NVMe block devices:
Bash
/var/lib/snapd/snaps/firefox_2356.snap: Can't lookup blockdev
[FAILED] Failed to mount Mount unit for firefox.
[FAILED] Failed to mount Mount unit for snapd.
What I have tried so far:
BIOS Settings:
Kernel Parameters:
Tried pci=realloc: Did not resolve the no space error.
Tried iommu=soft: This allows the system to boot with "Above 4G Decoding" enabled (bypassing the blockdev error), BUT nvidia-smi fails to detect devices, or the system falls back to llvmpipe (software rendering) with a zoomed-in resolution.
Display Manager:
- Disabled Wayland in
/etc/gdm3/custom.conf (WaylandEnable=false).
Driver Installation:
Purged all apt nvidia packages.
Running the .run file for driver 580.126.09.
Installation completes successfully in Recovery Mode (Root Shell), but upon rebooting with iommu=soft, the driver doesn't seem to bind correctly to the cards.
The Question: How can I properly expose the BAR address space for both cards on a B550/Ryzen platform without breaking the NVMe/IOMMU controller? Is there a specific combination of GRUB parameters (e.g., pci=noalloc, iommu=pt, amd_iommu=on) that allows 4G decoding to coexist with a functional NVIDIA driver on this chipset?
Any insights on how to escape this loop would be appreciated.