The problem with PCIe is not bandwidth, it is the limit in lanes on consumer PCs: 20 lanes from CPU and a few more from the SouthBridge is not enough when the GPU is usually linked on a 16 lane connection. The easy way out is to reduce GPU lanes to 8, that leaves plenty of bandwidth for nVME SSDs and maybe for 10 or 25 Gbps NICs (it's about time).
For servers it is a different story, but the recent fast move from PCIe ver 3 to ver 5 improved the situation 4x, doubling again is nice, but it does not seem that much of a deal. Maybe moving NICs from the usual 8 lane to a lot less (8 lanes of ver 3 means 2 lanes of ver 5 or a single lane of ver 6) will also make some difference.
But doubling the bandwidth per lane allows one to use half as many lanes to a GPU and maintain bandwidth. As you mention, it allows an eight lane GPU to be a viable option. And better yet, due to how PCIe handles variable number of lanes between host and device, different users with the same CPU, GPU, and even motherboard can choose to run the GPU at eight lane with a couple of four-lane SSDs, or at sixteen lane for even more bandwidth if they don't need that bandwidth elsewhere.
8 lane GPU is viable for a long time (benchmarks on PCIe 8x versus 16x shows a 5% perf difference), but it did not change the physical layout the motherboard manufacturers use; you cannot use the existing lanes any way you want, on some motherboards you cannot even split it the way you want between physical connectors and video card manufacturers continue to push 16x everywhere.
It's an AMD problem as well. It's absolute nightmare trying to research for a computer today. What ports can you use in what circumstances. Which slots go the CPU directly and which go to a chipset.
Which lanes are disabled if you use nvme-slot 2. Which slot has which generation etc. A proper nightmare.
And while we are at it, dedicating pci-lanes to nvme-slots must be one of the most boneheaded decisions in modern computers. Just use a pci-card with up to four nvme-slots on it instead.
Maybe it’s because I bought a “gaming” motherboard, but the manual was pretty (for my understanding at least) as to what configuration of m.2 drives and PCIe lanes would run at what version, what went to cpu and what went to chipset.
The issue is cross comparison, not reading from the manual of a single board. In the past you could look at the physical motherboard layout and could compare it with another motherboard. Today, it's merely a suggestion. You take the physical layout with a huge grain of salt because it's nothing but lies. Which means going to the website of every single potential motherboard you wish to purchase, downloading their PDF, and digging through enough caveat fine print documentation to make your eyes bleed.
It's not just segmentation. Laptop buyers are not going to pay for 64 lanes. A regular Intel SKU of the 12th generation has 28 PCIe 4.0/5.0 lanes. A Xeon has 64, does not have 5.0, and costs way more, partly because it has 4189 pins on the bottom, which is insane.
> The problem with PCIe is not bandwidth, it is the limit in lanes on consumer PCs... For servers it is a different story
EPYC/Threadripper is awesome but also explicitly not the segment being discussed. But since it was mentioned it's 64 PCIe lanes per socket on Intel (up to 8 sockets/512 lanes) instead of 128 lanes for single socket/up to 160 lanes for dual socket on Epyc. Gonna cost you a damn arm and leg and you better know your NUMA though.
In the consumer segment AMD has 16 PCIe lanes for the graphics and up to 4 PCIe lanes for NVMe drives, the chipset link is and additional x4 PCIe instead of a different interconnect. This comes to 20 PCIe lanes for user devices and 4 lanes worth of PCIe bandwidth off the chipset. https://i.imgur.com/8Aug02l.png
Intel offers something similar, 16 PCIe lanes for graphics and 4 PCIe lanes for NVMe. They opt for a proprietary DMI connection to the chipset which is equivalent to 8 lanes of PCIe 3.0 bandwidth or 4 lanes of PCIe 3.0 bandwith. https://i.pcmag.com/imagery/reviews/070BdprI2Ik2Ecd2wzo0Asi-...
Each offers splitting the x16 for the GPU into 2 x8 as well as oversubscribing the downstream PCIe bandwidth from the chipset, devices through the chipset on each obviously hit different latency penalties than CPU direct lanes as well. In the end the offerings are both pretty much identical in the consumer space.
With the bandwidth you do get more flexibility though, and with pci express bifurcation you can get the bandwidth equivalent of 4 pcie4 16x from a single pcie6 16x.
And that's great since the vast majority don't need that much bandwidth anyway.
Today you typically have a whole slew of devices sharing 4x to the cpu. More bandwidth would open up for more usb and perhaps cheaper onboard 10gig Ethernet etc.
For servers it is a different story, but the recent fast move from PCIe ver 3 to ver 5 improved the situation 4x, doubling again is nice, but it does not seem that much of a deal. Maybe moving NICs from the usual 8 lane to a lot less (8 lanes of ver 3 means 2 lanes of ver 5 or a single lane of ver 6) will also make some difference.