BAR Allocation Failed & IOMMU Conflicts: Dual GPU (RTX 5060 + 4060) on Ryzen 5800X/B550 - "No Space" Errors
https://askubuntu.com/questions/1563112/bar-allocation-failed-iommu-conflicts-dual-gpu-rtx-5060-4060-on-ryzen-580
I am hitting a wall trying to configure a dual GPU setup on a B550/Ryzen platform running Ubuntu. I am facing a persistent Memory Mapped I/O (MMIO) resource allocation issue where the kernel cannot assign BAR addresses for the second GPU, leading to can't assign; no space errors.
Hardware Specs:
CPU: AMD Ryzen 7 5800X
Mobo: B550 Chipset [INSERT YOUR MOTHERBOARD MODEL HERE]
GPU 1: NVIDIA RTX 4060
GPU 2: NVIDIA RTX 5060 (New gen)
PSU: 850W
OS: Ubuntu (Kernel 6.x generic)
Driver: Manual install of NVIDIA-Linux-x86_64-580.126.09.run (Required for 5060 support)
The Problem: When booting with default BIOS settings, only one GPU is enumerated correctly. The second GPU fails with BAR allocation errors in dmesg. When I attempt to fix this by enabling "Above 4G Decoding" in BIOS, the system fails to mount the NVMe drive or panics during boot unless I use specific kernel parameters, and even then, I lose the proprietary driver binding.
Logs & Symptoms:
1. The BAR Error (dmesg output): Without 4G decoding, the kernel clearly sees the device but cannot map the memory:
Bash
[ 0.666193] pci 0000:07:00.0: BAR 1 [mem size 0x10000000 64bit pref]: can't assign; no space
[ 0.666195] pci 0000:07:00.0: BAR 1 [mem size 0x10000000 64bit pref]: failed to assign
[ 0.666196] pci 0000:07:00.0: VF BAR 2 [mem size 0x10000000 64bit pref]: can't assign; no space
NVRM: BAR1 is 0M @ 0x0 (PCI:0000:04:00.0)
2. The IOMMU/Disk Conflict: If I Enable Above 4G Decoding and Disable CSM (UEFI only), the boot process hangs with filesystem errors, losing access to the NVMe block devices:
Bash
/var/lib/snapd/snaps/firefox_2356.snap: Can't lookup blockdev
[FAILED] Failed to mount Mount unit for firefox.
[FAILED] Failed to mount Mount unit for snapd.
What I have tried so far:
BIOS Settings:
Kernel Parameters:
Tried pci=realloc: Did not resolve the no space error.
Tried iommu=soft: This allows the system to boot with "Above 4G Decoding" enabled (bypassing the blockdev error), BUT nvidia-smi fails to detect devices, or the system falls back to llvmpipe (software rendering) with a zoomed-in resolution.
Display Manager:
- Disabled Wayland in
/etc/gdm3/custom.conf (WaylandEnable=false).
Driver Installation:
Purged all apt nvidia packages.
Running the .run file for driver 580.126.09.
Installation completes successfully in Recovery Mode (Root Shell), but upon rebooting with iommu=soft, the driver doesn't seem to bind correctly to the cards.
The Question: How can I properly expose the BAR address space for both cards on a B550/Ryzen platform without breaking the NVMe/IOMMU controller? Is there a specific combination of GRUB parameters (e.g., pci=noalloc, iommu=pt, amd_iommu=on) that allows 4G decoding to coexist with a functional NVIDIA driver on this chipset?
Any insights on how to escape this loop would be appreciated.
I tested all suggestions, but the NVMe/Blockdev error persists. I have methodically tried all three proposed solutions, but unfortunately, they all result in the exact same boot failure. It seems that enabling "Above 4G Decoding" breaks the NVMe controller's ability to mount filesystems, regardless of the IOMMU parameters or BAR sizing.
Here is exactly what I tested:
Attempt 1 (`iommu=off`)
BIOS: Above 4G [Enabled], CSM [Disabled].
GRUB: iommu=off
Result: Boot failure. "Can't lookup blockdev".
Attempt 2 (pci=big_root_window):
BIOS: Above 4G [Enabled], CSM [Disabled].
GRUB: iommu=soft pci=big_root_window
Result: Boot failure. "Can't lookup blockdev".
Attempt 3 (Re-Size BAR Disabled):
BIOS: Above 4G [Enabled], Re-Size BAR [Disabled], CSM [Disabled].
GRUB: iommu=soft
Result: Boot failure. "Can't lookup blockdev".
The error log (identical in all 3 cases):
The system sees the drive initially (/dev/nvme0n1p2: clean...), but immediately loses access to block devices required for mounting snaps and the desktop environment:
/dev/nvme0n1p2: clean, 2954993/30498816 files, 87996494/121965088 blocks
[ 3.603861] /var/lib/snapd/snaps/firefox_2356.snap: Can't lookup blockdev
[ 3.603916] /var/lib/snapd/snaps/gnome-3-38-2004_119.snap: Can't lookup blockdev
[FAILED] Failed to mount Mount unit for firefox, revision 2356.
[FAILED] Failed to mount Mount unit for snapd, revision 18357.
[FAILED] Failed to start Process error reports when automatic reporting is enabled.
It appears impossible to keep the NVMe drive functional while "Above 4G Decoding" is Enabled on this B550/Ryzen setup with the current kernel parameters.
Is there any other specific IOMMU parameter (like amd_iommu=fullflush or iommu=pt) that specifically targets B550 NVMe conflicts? Or is moving the NVMe drive to a Chipset-controlled M.2 slot (instead of the CPU-controlled one) a potential workaround?
