I'm always bothered by seeing that PCI-SIG doubling graph. Starting the graph back at PCI in 1992 glosses over that PCI was too slow to begin with, which is why there was VESA local bus, and then AGP (Intel), and then finally the development of 3GIO by Intel, Dell, HP, and IBM, which was handed over to the PCI-SIG and renamed PCI-Express.
The graph should start with PCIe 1.0 in 2003. Then doubling every 3 years from there would get us on time to the speed that PCIe 6.0 achieves in 2021 - not the 4 years later interpretation they show starting from 1992 that relies on the work of outside groups that created their own specification to get around PCI-SIG - providing we ignore the fact that PCIe 1.0 devices were available in 2004, a year after the specification release, and it's now 2021, two years after the PCIe 5.0 spec release, and we still don't have access to PCIe 5.0 devices. Since PCIe 6.0 and PCIe 5.0 are unlikely to appear at almost same time I don't think we'll be getting access to PCIe 6.0 speeds next year.
I could see 5.0 and 6.0 being useful in reducing costs/power consumption for consumer CPUs by reducing the number of lanes needed. A 2x 5.0 or 1x 6.0 link would be just fine for any M.2 SSD on the market today, for example.
PCIe 4.0 x4 SSDs could be updated to use PCIe 5.0 x2 lanes and signaling, but they'd still consume the same power without redesigning the SSD controller for PCIe 5.0. CPU I/O might end up using more power to drive faster 5.0 signals too.
PCIe 5.0 and 6.0 may also increase development costs of mainboards, as they'll likely need more copper to handle extra bandwidth or redrivers to amplify signals (as we see with 4.0 boards today). If data integrity is an issue, more expensive retimers can be used to retransmit a fresh copy of data to devices that exceed signal loss limits or, as in 6.0, FEC can be used.
I don't expect to see any consumer PCIe 5.0 devices within the next year or two.
He didn't say never just that right now it isn't useful for getting better performance. Even a RTX 3090 isn't limited by pcie 3.0x16. For consumers the network limit is the internet connection. hence why 1gbe is still essentially the standard as barley anyone needs more due to internet speed limits and data caps.
I agree FatFlatulentGit that it can help with having to provide less lanes of a higher version. Plus pcie4 to the pch is IMHO the big benefit of rocket lake. But even then in real-world it probably doesn't add much value as few people are coping files from ssd to ssd on regular basis.
Sadly due to "good enough" we will see consumer and server hardware drift further apart again. race to bottom on consumer side. Bad for us enthusiast as it will make new great tech far more expensive. Just look at 10gbe.
That's not true, but you won't find a *huge* performance difference between 3.0 and 4.0, either.
> For consumers the network limit is the internet connection.
Did you ever imagine that people might use their network connection to copy data between PCs on their LAN? Have you ever heard of a NAS or a file server?
> Just look at 10gbe.
2.5 Gbps seems to be the new enthusiast standard. Not as high as I wish, but it's progress.
That's not strictly true. As linked by @fallaha56, high-end gaming cards show some additional gaming performance from PCIe 4.0 x16 vs. 3.0 x16 (which is roughly equivalent to 4.0 x8).
> it would seem for years to come we will need more bandwidth
Even with the benchmarks you just posted, I think gaming cards will stay at PCIe 4.0 for the foreseeable future.
Compute accelerators are likely to just go for CXL, though I think some might support both, for at least a while. CXL should be lower-latency and has coherency support built-in.
It shares the same PHY layer with PCIe 5, making it easy to implement a device which speaks both protocols. It does not mean that CXL support is free or automatic, for PCIe 5.0 devices.
> since almost every GPU is a compute accelerator these days
AMD, Nvidia, and Intel all have dedicated product lines for compute accelerators, now. To strengthen that market segmentation, I think they won't upgrade gaming products to PCIe 5.0 as soon, if ever.
> where high speed networking or GPU interconnects are needed.
Intel Optane DC P5800X can make good use of PCIe 4.0 x4. Its 512B IOPs are so high that I could believe it would measurably benefit from the lower latency of PCIe 5.
Intel has been making excuses about why people don't need I/O for a long time. Seems the only things that matter to them are thin, light, and anti-virus.
For a long time consumer platforms have been starved for I/O, particularly when you consider that most chips and motherboards have restrictions about how you can use the ports -- restrictions you'll usually learn about when you try to plug in a card and it doesn't work or some of the features of the motherboard (serial ports) quit working.
It's like the bad old days before Plug-N-Play except back then people knew what the rules were, now the rules are "buy an i7 chip, the most expensive motherboard you can find, and pray"
and this could be solved as simply as just adding more pcie lanes to the consumer platform, even if it is just on higher end boards. for my own usage, i could use a few more lanes.....
> this could be solved as simply as just adding more pcie lanes to the consumer platform
The socket from Sandybridge and Ivy Bridge supported 20 lanes of PCIe + the 4-lane DMI link. They dropped 4 of those lanes in Haswell, only to bring them back in Comet Lake, but that socket also widened the DMI link to x8, which benefits everything hanging off it.
Then, Rocket Lake boosted everything to PCIe 4.0. So, now the chipset has 16x of the bandwidth it had in Skylake times and you have a dedicated 4.0 x4 link for SSD. And because of their x8 chipset link, they provide even more bandwidth than AMD's desktop platform!
I'm really interested to learn why their mainstream socket jumped all the way to 1700-pins, in Alder Lake.
If you've been feeling I/O-starved, these are indeed good times!
You are forgetting all of the major speed revisions to the original PCI bus. First was 33MHz at 32 bit, then 66MHz, then finally 133MHz 64 bit by the time PCI 3.0 came along in some server variants. AGP was also a superset of the PCI bus, with changes to reduce latency which is sensitive for GPU operations. Also, VESA local bus was a competitor to PCI that was supposed to replace ISA, but PCI won out. VLB was developed before PCI and has nothing to do with PCI being too slow. The biggest reason we migrated to PCI-E was the transition from parallel to serial in order to prevent data skew.
OK, forget about what I said about VESA local bus. But PCI 1.0 was also not created by PCI-SIG. It was created by Intel. I doesn't matter if AGP was a superset of PCI. The work by the PCI-SIG itself was not enough for the demands of common computing devices wishing to use the bus. I'm not forgetting any speed revisions, I'm just going by the graphic that PCI-SIG themselves provided. It also doesn't matter why we migrated to PCI-E. It wasn't developed by the PCI-SIG, it was given to them. And it created a big jump in bandwidth. In fact it created THE big jump in bandwidth that puts them 4 years ahead of the 3 year doubling time in their graph.
We got PCIE 1.0 devices so quickly because of the all the work done leading up to it and more importantly how inadequate PCI, AGP and VESA were. No subsequent upgrade since 1.0 has arrived that quickly. PCIE3 was out for a few years before the first devices arrived and 4 was even worse. I expect the first 5.0 MB's and devices will be next year or 2023 and 6.0 production devices is a long way away.
Well first PCIe 5.0 will be used this year for Intel AlderLake and DG2.
It is only for the GPU link but I would see a general use if GPU, CPU and SSD are all supporting it. Guess that would be enough for most consumers.
I am not sure if ResizableBAR and DirectStorage profit from PCIe 5.0/6.0 a lot but if it would obviously be nice to have at least GPU, CPU and SSD connected with newer PCIe and maybe even iclude RAM addressing over CXL / CCIX.
You do not need the whole chipset supporting it - just the direct lines from the CPU to GPU, CPU, SSD and I/0 while the others stay on PCIe 4.0
> first PCIe 5.0 will be used this year for Intel AlderLake and DG2. It is only for the GPU link
How do you know this?
> You do not need the whole chipset supporting it
The board needs to support it, which means more layers and retimers. That adds cost.
And remember how AMD X570 boards started showing up with cooling fans on the chipsets? If PCIe 4.0 used that much more power, just imagine *another* doubling of clock speed! Yeah, maybe not in the southbridge, but no matter where you put it, get ready to burn some real power on I/O.
This is based on current information about AlderLake and DG2 which seems to indicate that AlderLake has a 16x PCIe 5.0 link and DG2 has a PCIe 5.0 link.
Yes the board needs to support it but you can limit it to one GPU link and one SSD link that are close to the CPU. The other lanes and chipset can be kept simple.
Yes X570 was hot but that is mainly because they used an Ryzen I/O instead of a specific chipset like the B550.
Yes, but how does that not make what I said less relevant? What you are saying is that the time stamp given to PCIe 1.0 is not at the same standard of the time stamp as PCIe 5.0 or PCIe 6.0, and the difference is in such a direction to give the impression that PCIe 6.0 is coming sooner after PCIe 1.0 than it really is. So if we had uniformity of time stamps then PCIe 6.0 would be behind a "doubling every three years" cadence from PCIe 1.0.
> more importantly how inadequate PCI, AGP and VESA were.
Woah, that's a huge blurring-together of technologies spanning a whole decade!
IIRC, AGP 8x wasn't too slow, though a x16 PCIe 1.0 slot was almost 2x as fast. The bigger benefit of PCIe was for the non-graphics slots, which were generally stuck on 32-bit PCI @ 33 MHz.
PCIe 5.0 is more important than PCIe 4.0 because CXL depends on it. For consumer applications the bus speed is not that important, but PCIe 4.0 was so late that it got to the point that the bus speed was becoming a bottleneck for consumers.
No, it won't. PCIe-6 is same speed as PCIe-5 but PAM4, that is 2 bit per clock instead on 1. So the power requirements will be almost the same but with double the bandwidth.
It's the same technology Nvidia ha adopted for NVLink 3 a couple of years earlier. Probably because even PCI consortium has understood that it is not possible to continue pumping up the frequencies to go faster.
PAM4 however means that the new standard is no more compatible with previous gen PICe, unless the controller has a fallback mode to PCIe-5.
> No, it won't. > PCIe-6 is same speed as PCIe-5 but PAM4, that is 2 bit per clock instead on 1. > So the power requirements will be almost the same but with double the bandwidth.
You don't think PAM4 is more costly to implement and burns more power?
It's definitely more noise-sensitive, meaning boards will have to use more layers and more expensive materials, as well as probably retimers (which will also be more expensive and consume more power, thanks to PAM4).
> It's the same technology Nvidia ha adopted for NVLink 3 a couple of years earlier.
That's only 50 Gbps and introduced just last year. Even on a RTX 3090, it doesn't hit PCIe 5.0 rates, much less PCIe 6.0.
> PAM4 however means that the new standard is no more compatible with previous gen PICe
Read the article. From one of the slides:
"Maintains backward compatibility with all previous generations of PCIe architecture"
> it got to the point that the bus speed was becoming a bottleneck for consumers.
Not really. More like: "GPUs got to the point where they could *start* to reach the limits of PCIe 3.0". Go back and look at PCIe 3.0 SSDs and you won't see them even hit the link speed.
> You don't think the release of SSDs faster than PCIe 3.0 speeds
I think AMD releasing support for PCIe 4 created a scramble among SSD makers to try and build a product for it. But you could take most 1st-gen PCIe 4 SSDs and run them at PCIe 3 and hardly measure a difference.
It's not at all like the situation we had with SATA SSDs, where they were link-limited for years.
> looks like we are going to go straight from PCIe 4.0 to PCIe 6.0
We're not. Sapphire Rapids and Alder Lake will use PCIe 5.0 this year or early next.
We'll see when PCIe 6.0 hits. Whatever happens with 5.0, in the consumer space (and I speculate it will not be much), don't expect 6.0 to follow any time soon.
Yes, I know Intel is adding support for 5.0. But I'm not sure why. There are no PCIe 5.0 products on the horizon: no NICs, no SSDs, no video cards, nothing. PCIe 4.0 will remain fast enough for the next two years.
So, in three years, if I am Nvidia, and CPUs added support for PCIe 6.0, why would I upgrade my products to 5.0 instead of 6.0? Low end cards don't need more than 4.0 - don't need to upgrade those. High end cards cost so much and consume so much power that any extra cost/power from 6.0 is insignificant. And there's a clear benefit from the increased bandwidth - otherwise why would Nvidia developed their proprietary NVLink?
Sure, Nvidia could upgrade their cards to 5.0 next year if they wanted to, but they just upgraded to 4.0 last year, so chances of that are slim.
> I know Intel is adding support for 5.0. But I'm not sure why.
I think it could be just for the CPU -> chipset link. That would give them PCIe 5.0 bragging rights and let them shrink the width of that link back down to its historical x4 width. Moreover, it's soldered potentially right next to the CPU, making it the cheapest and simplest thing to connect via PCIe 5.
It'd also mean they could then drop the CPU back to x16 lanes, since the chipset would have plenty of bandwidth for all the NVMe cards people want to use, and even a respectable amount of bandwidth for a second GPU.
What flies in the face of that scenario is that the socket is also ballooning up to more than 1700 contacts. So, that casts doubt on the idea they'll be cutting back on any CPU-direct connectivity.
> in three years, ... and CPUs added support for PCIe 6.0
Why would they? If 5.0 adoption is low and 5.0 burns more power and increases board and peripheral costs, why would consumer CPUs ever go to 6?
> High end cards cost so much and consume so much power that any extra cost/power from 6.0 is insignificant.
Not sure about that. Cost is cost. And with Intel in the GPU race and if mining cools down, we could see GPU prices come back to Earth.
> there's a clear benefit from the increased bandwidth - > otherwise why would Nvidia developed their proprietary NVLink?
That's not in most of their consumer cards, which is presumably what we're talking about.
Anyway, the RTX 3090 already has about 56 GB/s of NVLink bandwidth, compared to PCIe 5.0's 64 GB/s. However, the NVLink bandwidth is exclusive for GPU <-> GPU communication. NVidia has already announced a generation of NVLink beyond that.
I also don't see them ditching NVLink, because it scales better.
CXL is why. And PCIe 4.0 is going to be short-lived. Both AMD and Intel will be changing over to PCIe 5.0 quickly. It remains to be seen how fast PCIe 6.0 is taken up, especially in the consumer space.
Not sure why you think the chances are slim that NVIDIA will upgrade their cards to PCIe 5.0 The data center cards will certainly upgrade to 6.0 because 1) NVLink is built off PCIe and 2) they want to use CXL. As far as the gaming cards, what difference does it really make?
If you buy a motherboard an CPU Alder Lake or later it will be PCIe 5.0. Are you just going to hold off buying one because you think 6.0 will be out afterward? Why not just hold off for 7.0?
You seem to be trying very hard to make some sort of V with PCIe 4.0 and 6.0 at the tips and 5.0 at the trough. But it's not like that. People will buy PCIe 5.0 stuff because that's what will be available. And as far as consumers, they're going to need 6 even less than they will need 5. It seems SSDs will be able to take advantage of 5 when it comes out. But I wonder if they'll be able to take advantage of 6. Maybe if heterogeneous computing makes it to the PC then we'll see benefit of these higher bandwidths.
> CXL is why. And PCIe 4.0 is going to be short-lived. > Both AMD and Intel will be changing over to PCIe 5.0 quickly.
You're failing to draw any distinction between their consumer and server products. Consumer boards won't have CXL-support, for one thing. There's no reason for it.
> If you buy a motherboard an CPU Alder Lake or later it will be PCIe 5.0.
How do you know? Just because it said "PCIe 5" on a leaked roadmap? Nobody has yet shown me any evidence there will be actual PCIe 5 slots in a Alder Lake motherboard.
> It seems SSDs will be able to take advantage of 5
Just because there's server SSDs that burn like 25 W that can do it? That's not something you could fit in a M.2 slot. Not just for power and thermal reasons, but all the chips wouldn't even fit a M.2 board.
Again, consumer SSDs can barely peak above PCIe 4.0 speeds. Samsung just transitioned its mighty Pro line to TLC. Consumer SSDs are too limited by GB/$ to hit the kinds of speeds that $15k server SSDs can do, even in a AIC form factor.
For consumer level products sure there won't be much available for PCIe 5.0. That won't be the case in the enterprise market. There are already switches that have 400GbE ports and PCIe 5.0 NICs are on the way. NVidia has already announced their ConnectX-7 cards with PCIe 5.0 x16/x32 links.
In the consumer space PCIe 5.0 is only important for SSDs. We have finally gotten to the point that PCIe 4.0 SSDs can in theory saturate an x4 link. For GPUs we aren't anywhere close to being limited on the bus speed.
For the data center having PCIe 5.0 is VERY important for networking equipment. More and more hyperscalers are moving to faster and faster Ethernet. We were stuck on PCIe 3.0 for far too long in the data center and that stagnated network speeds and hyperconverged storage solutions. With the move to PCIe 4.0 we were able to get dual port 100GbE or single port 200GbE at bus speeds for each port. There were dual port PCIe 3.0 100GbE cards but you didn't get any benefit of the 2nd port as you were bus limited. Now with PCIe 5.0 you can get dual port 200GbE or single port 400GbE. This helps massively when you are running something like VMware vSAN for your storage. The idea is to make storage less of a bottleneck in the data center and the faster Ethernet in conjunction with faster SSD is making that possible. This ever increasing network bandwidth is also making it very hard on Fibre Channel as that is limited to 32Gb where iSCSI can do 200Gb right now.
> In the consumer space PCIe 5.0 is only important for SSDs. We have finally gotten to the > point that PCIe 4.0 SSDs can in theory saturate an x4 link. For GPUs we aren't anywhere > close to being limited on the bus speed.
This is a laughable argument. In 2012, the first PCIe 3.0 GPUs could probably saturate a x16 link, but that doesn't mean we needed PCIe 4.0
Even a PCIe 4.0 consumer SSD being able to touch the limit of an x4 link, in peak speeds, doesn't justify transitioning to 5.0.
> Starting the graph back at PCI in 1992 glosses over that PCI was too slow to begin with, > which is why there was VESA local bus
Huh? 486DX2 machines had VLB, because EISA wasn't fast enough! Once Pentiums came out, they pretty much immediately adopted the new PCI bus. And it was *plenty* fast, at the time!
It was only ~5 years later that PCI got stuck between 66 MHz and 64-bit, with neither seeming to gain mainstream acceptance. That's when Intel decided to push AGP, as the way forward. That lasted until 2004 or so, when PCIe hit the mainstream.
VESA local bus came before PCI, but, sorry to nitpick, the way you described it is as if VESA local bus came after PCI. Theoretical bandwidth wise VESA can go higher than the lowest iteration of PCI, indeed.
I am unfortunately old enough to witness those transitions.
There's not a need for PCIe x32 outside of servers, where they're used for riser cards. PCIe x2 doesn't really make sense and was only really used for cheap SSDs. x12 might make sense in a BiFurcation situation for x12 GPU and an x4 SSD, but now that both AMD and Intel already have a CPU x4 lane dedicated for SSDs it's not going to happen.
But, the CPU x4 lane should be dedicated as the OS drive.
The BiFuricated x12 + x4 = GPU + Game Drive where the Game Drive could have DMA to the GPU's RAM, ergo lowering latency & power requirements for sending data from the game to the GPU.
PCIe x2 is widespread, just not in slot form. In addition to some SSDs, a lot of integrated chips on motherboards use PCIe x2. For example, the ubiquitous ASMedia USB-C controllers all use PCIe x2.
x2 and x12 don't make sense in slot form because if you look at the pinout of the PCIe slot, an x2 or x12 card couldn't be made any smaller than an x4 or x16 card, respectively.
"Meanwhile, as the PCIe 6.0 specification reaches competition, ..." Competition, eh? With CXL maybe? I think you meant "completion". "Meanwhile, as the PCIe 6.0 specification reaches completion, ..."
This ain't about you, dude. This is about hyperscalers who want 800 Gigabit Ethernet and about datacenter GPUs that want to stream vast amounts of data in/out of system RAM.
Does PCIe 6.0's 1b/1b encoding mean zero encoding overhead? Is this really possible? It's also unclear if "FLIT mode" is always active or not, and what exactly does this mean (I looked it up and FLIT is apparently the acronym for "Flow Control Unit").
Elsewhere I read that that PCIe 6.0, due to the switch to PAM4 signaling, has a staggering 3 orders of magnitude (i.e. 1000 times) higher bit-error rate than PCIe 5.0, thus the need for FEC. While this sounds unreasonably high the error rate of PCIe up to version 5.0 is reportedly extremely low to begin with, so FEC can handle the error rate of PCIe 6.0 with minimal extra latency (the target is an astoundingly low 1 - 2 ns of extra latency due to FEC, while the FEC variant used on Ethernet adds ~100 ns of latency)
PAM4 certainly kills SNR. EQ tuning becomes massively important. It looks like they specify tuning of DFE and CTLE to minimize BER, but I suppose that only buys so much.
Idk what you consider "astonishingly low" for effective BER, but anything above 10^-18 is too high for an interconnect this fast and with this application is way too high imo. I don't see the numbers mentioned in the article.
Line coding's signal integrity benefits are made obsolete by FEC. I don't full understand FLIT, but it appears to be their "lightweight" FEC implementation that allows for low latency. IEEE 802.3 know their business. A lot of this PAM4 is copy and paste from bs. I'm wary of places where they deviate, such as saying "100 ns is too long, we'll just make the FEC worse and retransmit more". That sends of red flags to me. Your effective throughput will greatly suffer if you rely on retransmission. Thou shalt not violate Shannon's limit.
> the FEC variant used on Ethernet adds ~100 ns of latency
At what speed of Ethernet? At 1 Gbps, a bit takes 1 ns. The CRC of an Ethernet frame is 32-bits. So, if that's what you're talking about, it should add 32 ns, at Gigabit speeds.
Really, rather than get carried away with ns, just look at the relative data sizes of FEC and the frame or packet size. The controller has to process it at wire-speed, and is almost certainly doing so as it reads bits (or PAM4 symbols) off the wire.
It depends on how far you want to go with the signals, which max latency you are targeting and most of all, how much energy you want to spend for each bit. There's not a theoretical maximum speed you can achieve.
Oh, I forgot the costs. It also depends on how much you want your transmission to be complex to build and so how much it will cost in term of pure $$$.
This doesn't seem very useful. It would've been simpler and more accurate to say "I don't know", which is clearly the case. You probably never designed a PCB in your life.
I know that, but it hasn't been disclosed where or how. I'm asking how the author seems to know that "most" of its consumer motherboards will have "only a slot or two").
To the extent of any evidence I've seen, Alder Lake could just use PCIe 5.0 for its DMI link. If Ryan Smith definitely knows otherwise, I'd like him to confirm.
Making a public announcement of supporting a technology and then bury it only for internal use (which is not interesting to any consumer) would not be a good move. IMHO. So AL will probably have an external link PCIe 5 (4 lanes?) from CPU towards what can be simply a single SSD slot, just to say: we can be faster than the competition, and at the first device supporting that technology they can beat on that nail for months. The number of lanes will be few as they are very critical copper traces and cost a lot, so motherboards cannot be that expensive just for adding a technology that is not really beneficial today (and probably for the next 5 years but just for a small 0.x% amount of PC users).
I think that the balance will be having few lanes for imagine purpose but not too many for not going up with costs and energy consumption.
On laptop PCIe5 may be useful for limiting the lanes going towards the classic x4 connection towards the dGPU. GPU producer may spare some money and energy in supporting PCI5 with the added option to also advertising that feature just to give impression that the GPU will be even faster (you know, 3.0<4.0<5.0 and big numbers always matter! Give me a 200Mpixel sensor on that smartphone!).
> Making a public announcement of supporting a technology and then bury it only for > internal use (which is not interesting to any consumer) would not be a good move.
A lot of features and specs are just for bragging rights. Also, they could be trying to push AMD into a very expensive mistake.
> how would this push amd to make a very expensive mistake ?
If Intel is only adding PCIe 5.0 for the CPU-chipset link, all the noise they're making about it could scare AMD into adding it for all PCIe lanes, thinking that's what Intel is doing.
why would it scare them ? when amd added pcie 4 to x570, intel fans were saying so ? its no big deal, most devices cant use it yet any way. which was true, but now, not so much, going from a 970 pro to a 980 pro for my OS drive, i noticed a speed increase when i installed win10, and a little bit of an increase for other things. amd could just do the same, use pcie 5 for the cpu to chipset link, and leave everything else as 4, for check box parity. has amd even mentioned what " x670 " could have for pcie support yet ?
> amd could just do the same, use pcie 5 for the cpu to chipset link, > and leave everything else as 4, for check box parity.
If they know, for sure, that's what Intel is doing.
They might have been planning to design AM5 for PCIe 4.0, only. Intel's noise about PCIe 5 could have caused AMD to second guess themselves, and add in support for PCIe 5. That could cause delays and increased costs (due to more pins & larger footprint, if nothing else).
If PCIe 6.0 is approaching finalisation, that must mean work is starting on PCIe 7.0? Any rumours about the direction it will take? Do we have a date yet for the initial 0.3 draft?
I wonder if they could do like a 4.1 that runs at PCIe 4.0's clock speed, but uses PAM4 to deliver PCIe 5.0 bandwidth. I wonder if that would be cheaper (or more power-efficient) to implement on desktops & laptops than PCIe 5.
History is littered with the graves of many half-assed technology - and many fully-assed techs. I’m sorry that sounds like a half-assed short term effort that won’t go mainstream.
And the problem it solves isn't only short-term. The signal integrity requirements of PCIe 5 aren't going to relax with time. And that means they're always going to add costs to PCIe 5.0 boards vs. PCIe 4.0.
Does "Remaining System Base Board Budget" still provide enough margin for PCIe sockets for full device-to-device bandwidth compared to NVlink 2.0/3.0 (so for PCIe 5.0 for about (36 minus 2*(9.5 plus 1.5) minus (4-7 pcb copper safety margin)) remaining 9-12dB for burst data transfer states on ~4 GB/s per lane and direction)? While NVlink switches are especially designed for that purpose (up to now 25-50Gbit/s, while PCIe 5.0 is standardized to 32GT/s, means 16Ghz maximum clock rate for data for error correction, header data and precoding/decoding? also on (NRZ) signaling scheme for 128b/130b coding scheme). Physical limitations for NVlink show real 90-95% of standardized bandwidth, sample is ~35Gbit/s for 40Gbit/s link (2 sub-lanes uplink to P100 NV-Pascal GPU). PCISIG tells like this "The signal attenuation caused by the channel insertion loss (IL) is the biggest challenge of PCIe 5.0 technology system design.", what is recognized on about halfing maximum line length with doubling clock rates (>PCIe 3.x). Comparison of expectations/potential for PCIe 5.0 and NVlink 2.0/3.0 would be pretty useful and readworthy (especially PCIe device-to-device rates for AI accelerators and databases/inference models storage)
(PCI was a bus.) PCIe connects devices point to point and can support DMA for devices without affecting other PCIe connections (or devices) within a switch (or maybe the root package == chipset?). Seems there is no protocol overhead for "DMA"? accesses over switches, but for CPU/SoC connections for directly connected PCIe devices (hardware managed?) this might differ?
NVlink is called a bus system and NVlink bridge (for example) is a parallel bandwidth path additional to PCIe and data transfer is managed on each graphic card's NVlink controller? This changes on a NVlink switch that connects through 6-12 (V100-A100) switching interfaces up to 8-16 graphic devices (point to point through the switch)?
Maybe it is fair enough to summarize, that switches (for PCIe, NVlink) can connect devices like these were on physically paralleled lanes if suitable for "DMA"?
A comment made on Anandtech, mentioning that errors, because of error handling are a bigger influence to data bandwidth (what was probably within networking hardware news on x00Gbit/s devices) one might expect, could ask for the following (ethernet <> PCIe, NVlink overhead, transmit order?) %) What are error handling routines within PCIe protocol (transaction, data link, physical layer), received on "error reporting register(s)", what's their average impact and are there common stats available (PCIe2.x - PCIe5.0), so devices can be considered being within specification limits? %%) Are there new protocol versions or more likely driver conf parameters for data transfer that would not require, up to some amount of data, being retransmitted "free of errors" (like on AI inference methods, were predictions are made on probability) for then maybe increased throughput (configured from data analysts, application type) one reference https://www.kernel.org/doc/Documentation/PCI/pciea... (articles explaining are detailed on web, there are userspace log tools for summarized stats to that already before PCIe5-PCIe6?)
Looking forward to add-in cards for USB4, TB4, 10(+)GbE and M.2 SSDs with 1x or 2x interfaces on the consumer side. Eight 1x lanes with this kind of bandwidth is exciting for consumers.
We’ve updated our terms. By continuing to use the site and/or by logging into your account, you agree to the Site’s updated Terms of Use and Privacy Policy.
Facebook
Twitter
RSS
103 Comments
Back to Article
Yojimbo - Tuesday, May 25, 2021 - link
I'm always bothered by seeing that PCI-SIG doubling graph. Starting the graph back at PCI in 1992 glosses over that PCI was too slow to begin with, which is why there was VESA local bus, and then AGP (Intel), and then finally the development of 3GIO by Intel, Dell, HP, and IBM, which was handed over to the PCI-SIG and renamed PCI-Express.The graph should start with PCIe 1.0 in 2003. Then doubling every 3 years from there would get us on time to the speed that PCIe 6.0 achieves in 2021 - not the 4 years later interpretation they show starting from 1992 that relies on the work of outside groups that created their own specification to get around PCI-SIG - providing we ignore the fact that PCIe 1.0 devices were available in 2004, a year after the specification release, and it's now 2021, two years after the PCIe 5.0 spec release, and we still don't have access to PCIe 5.0 devices. Since PCIe 6.0 and PCIe 5.0 are unlikely to appear at almost same time I don't think we'll be getting access to PCIe 6.0 speeds next year.
cosmotic - Tuesday, May 25, 2021 - link
We barely have access to PCIe 4 devices.DigitalFreak - Tuesday, May 25, 2021 - link
PCIe 5.0 and 6.0 are only really useful in servers and high end workstation environments, where high speed networking or GPU interconnects are needed.FatFlatulentGit - Tuesday, May 25, 2021 - link
I could see 5.0 and 6.0 being useful in reducing costs/power consumption for consumer CPUs by reducing the number of lanes needed. A 2x 5.0 or 1x 6.0 link would be just fine for any M.2 SSD on the market today, for example.JasonMZW20 - Thursday, May 27, 2021 - link
PCIe 4.0 x4 SSDs could be updated to use PCIe 5.0 x2 lanes and signaling, but they'd still consume the same power without redesigning the SSD controller for PCIe 5.0. CPU I/O might end up using more power to drive faster 5.0 signals too.PCIe 5.0 and 6.0 may also increase development costs of mainboards, as they'll likely need more copper to handle extra bandwidth or redrivers to amplify signals (as we see with 4.0 boards today). If data integrity is an issue, more expensive retimers can be used to retransmit a fresh copy of data to devices that exceed signal loss limits or, as in 6.0, FEC can be used.
I don't expect to see any consumer PCIe 5.0 devices within the next year or two.
vol.2 - Wednesday, July 21, 2021 - link
probably like 5 years + until the industry is lined-up for a shift past 4dotjaz - Tuesday, June 1, 2021 - link
It doesn't reduce power consumption. PAM4 isn't cheap to process.While there might be fewer lanes but they are much higher quality lanes, so likely more expensive.
ct909 - Wednesday, May 26, 2021 - link
No one will ever need more than 640kb of memory.beginner99 - Wednesday, May 26, 2021 - link
He didn't say never just that right now it isn't useful for getting better performance. Even a RTX 3090 isn't limited by pcie 3.0x16. For consumers the network limit is the internet connection. hence why 1gbe is still essentially the standard as barley anyone needs more due to internet speed limits and data caps.I agree FatFlatulentGit that it can help with having to provide less lanes of a higher version. Plus pcie4 to the pch is IMHO the big benefit of rocket lake. But even then in real-world it probably doesn't add much value as few people are coping files from ssd to ssd on regular basis.
Sadly due to "good enough" we will see consumer and server hardware drift further apart again. race to bottom on consumer side. Bad for us enthusiast as it will make new great tech far more expensive. Just look at 10gbe.
fallaha56 - Wednesday, May 26, 2021 - link
@beginner not true, there are already PCIe bottlenecked scenarios eg emulationthere are also applications that will work far better with higher bi-directional bandwidth eg video editing / transcoding
fallaha56 - Wednesday, May 26, 2021 - link
https://www.pugetsystems.com/labs/articles/PCI-Exp...mode_13h - Wednesday, May 26, 2021 - link
Thank you.Wow, the difference in DaVinci Resolve is huge!
mode_13h - Wednesday, May 26, 2021 - link
> Even a RTX 3090 isn't limited by pcie 3.0x16.That's not true, but you won't find a *huge* performance difference between 3.0 and 4.0, either.
> For consumers the network limit is the internet connection.
Did you ever imagine that people might use their network connection to copy data between PCs on their LAN? Have you ever heard of a NAS or a file server?
> Just look at 10gbe.
2.5 Gbps seems to be the new enthusiast standard. Not as high as I wish, but it's progress.
ICT Buff - Friday, May 28, 2021 - link
"For consumers the network limit is the internet connection." That's a joke, right?dotjaz - Tuesday, June 1, 2021 - link
We still don't have any consumer product exceeding the need of PCIe 4.0 x8mode_13h - Wednesday, June 2, 2021 - link
That's not strictly true. As linked by @fallaha56, high-end gaming cards show some additional gaming performance from PCIe 4.0 x16 vs. 3.0 x16 (which is roughly equivalent to 4.0 x8).FLORIDAMAN85 - Saturday, July 10, 2021 - link
Lololol 😆fallaha56 - Wednesday, May 26, 2021 - link
Sure Digital, after all 640kb should be enough for everybody ;))Given moves in eg VR and AI in gaming and visualisations surely it would seem for years to come we will need more bandwidth
mode_13h - Wednesday, May 26, 2021 - link
> it would seem for years to come we will need more bandwidthEven with the benchmarks you just posted, I think gaming cards will stay at PCIe 4.0 for the foreseeable future.
Compute accelerators are likely to just go for CXL, though I think some might support both, for at least a while. CXL should be lower-latency and has coherency support built-in.
mode_13h - Wednesday, May 26, 2021 - link
> Even with the benchmarks you just posted, I think gaming cards will stay at PCIe 4.0In fact, precisely *because* your benchmarks show such a small impact on gaming, from stepping up to 4.0.
TeXWiller - Friday, July 2, 2021 - link
CXL is running on PCIe5, and since almost every GPU is a compute accelerator these days, and even more in the very near future..mode_13h - Sunday, July 4, 2021 - link
> CXL is running on PCIe5It shares the same PHY layer with PCIe 5, making it easy to implement a device which speaks both protocols. It does not mean that CXL support is free or automatic, for PCIe 5.0 devices.
> since almost every GPU is a compute accelerator these days
AMD, Nvidia, and Intel all have dedicated product lines for compute accelerators, now. To strengthen that market segmentation, I think they won't upgrade gaming products to PCIe 5.0 as soon, if ever.
mode_13h - Wednesday, May 26, 2021 - link
> where high speed networking or GPU interconnects are needed.Intel Optane DC P5800X can make good use of PCIe 4.0 x4. Its 512B IOPs are so high that I could believe it would measurably benefit from the lower latency of PCIe 5.
PaulHoule - Friday, July 2, 2021 - link
Intel has been making excuses about why people don't need I/O for a long time. Seems the only things that matter to them are thin, light, and anti-virus.For a long time consumer platforms have been starved for I/O, particularly when you consider that most chips and motherboards have restrictions about how you can use the ports -- restrictions you'll usually learn about when you try to plug in a card and it doesn't work or some of the features of the motherboard (serial ports) quit working.
It's like the bad old days before Plug-N-Play except back then people knew what the rules were, now the rules are "buy an i7 chip, the most expensive motherboard you can find, and pray"
Qasar - Friday, July 2, 2021 - link
and this could be solved as simply as just adding more pcie lanes to the consumer platform, even if it is just on higher end boards. for my own usage, i could use a few more lanes.....mode_13h - Sunday, July 4, 2021 - link
> this could be solved as simply as just adding more pcie lanes to the consumer platformThe socket from Sandybridge and Ivy Bridge supported 20 lanes of PCIe + the 4-lane DMI link. They dropped 4 of those lanes in Haswell, only to bring them back in Comet Lake, but that socket also widened the DMI link to x8, which benefits everything hanging off it.
Then, Rocket Lake boosted everything to PCIe 4.0. So, now the chipset has 16x of the bandwidth it had in Skylake times and you have a dedicated 4.0 x4 link for SSD. And because of their x8 chipset link, they provide even more bandwidth than AMD's desktop platform!
I'm really interested to learn why their mainstream socket jumped all the way to 1700-pins, in Alder Lake.
If you've been feeling I/O-starved, these are indeed good times!
Daeros - Monday, July 5, 2021 - link
RKL can have up to 20 lanes of PCIe 4.0 and has an x8 DMI link at PCIe 3.0 signaling rates (~8GiB/s to the chipset).That ties AM4 for total bandwidth, since it has 24 lanes all running at PCIe x4 signaling rates.
Tams80 - Saturday, July 3, 2021 - link
And?This is establishing a specification. Products still need to designed, validated, certified, and then produced.
jsncrso - Tuesday, May 25, 2021 - link
You are forgetting all of the major speed revisions to the original PCI bus. First was 33MHz at 32 bit, then 66MHz, then finally 133MHz 64 bit by the time PCI 3.0 came along in some server variants. AGP was also a superset of the PCI bus, with changes to reduce latency which is sensitive for GPU operations. Also, VESA local bus was a competitor to PCI that was supposed to replace ISA, but PCI won out. VLB was developed before PCI and has nothing to do with PCI being too slow. The biggest reason we migrated to PCI-E was the transition from parallel to serial in order to prevent data skew.Yojimbo - Tuesday, May 25, 2021 - link
OK, forget about what I said about VESA local bus. But PCI 1.0 was also not created by PCI-SIG. It was created by Intel. I doesn't matter if AGP was a superset of PCI. The work by the PCI-SIG itself was not enough for the demands of common computing devices wishing to use the bus. I'm not forgetting any speed revisions, I'm just going by the graphic that PCI-SIG themselves provided. It also doesn't matter why we migrated to PCI-E. It wasn't developed by the PCI-SIG, it was given to them. And it created a big jump in bandwidth. In fact it created THE big jump in bandwidth that puts them 4 years ahead of the 3 year doubling time in their graph.melgross - Wednesday, May 26, 2021 - link
None of that matters today. It’s an interesting historical line that I remember quite well. It also has little to do with what’s being done today.As far as consumers go, it’s all about price, at least for Window’s and Chrome users are concerned. That’s a fast race to the bottom.
Yojimbo - Thursday, May 27, 2021 - link
If it doesn't matter why do they put the "doubling every three years" graph there?rahvin - Tuesday, May 25, 2021 - link
We got PCIE 1.0 devices so quickly because of the all the work done leading up to it and more importantly how inadequate PCI, AGP and VESA were. No subsequent upgrade since 1.0 has arrived that quickly. PCIE3 was out for a few years before the first devices arrived and 4 was even worse. I expect the first 5.0 MB's and devices will be next year or 2023 and 6.0 production devices is a long way away.Matthias B V - Wednesday, May 26, 2021 - link
Well first PCIe 5.0 will be used this year for Intel AlderLake and DG2.It is only for the GPU link but I would see a general use if GPU, CPU and SSD are all supporting it. Guess that would be enough for most consumers.
I am not sure if ResizableBAR and DirectStorage profit from PCIe 5.0/6.0 a lot but if it would obviously be nice to have at least GPU, CPU and SSD connected with newer PCIe and maybe even iclude RAM addressing over CXL / CCIX.
You do not need the whole chipset supporting it - just the direct lines from the CPU to GPU, CPU, SSD and I/0 while the others stay on PCIe 4.0
mode_13h - Wednesday, May 26, 2021 - link
> first PCIe 5.0 will be used this year for Intel AlderLake and DG2. It is only for the GPU linkHow do you know this?
> You do not need the whole chipset supporting it
The board needs to support it, which means more layers and retimers. That adds cost.
And remember how AMD X570 boards started showing up with cooling fans on the chipsets? If PCIe 4.0 used that much more power, just imagine *another* doubling of clock speed! Yeah, maybe not in the southbridge, but no matter where you put it, get ready to burn some real power on I/O.
mode_13h - Wednesday, May 26, 2021 - link
> That adds cost.Also, more exotic PCB materials. And repeat for each device that supports it.
Matthias B V - Wednesday, June 9, 2021 - link
This is based on current information about AlderLake and DG2 which seems to indicate that AlderLake has a 16x PCIe 5.0 link and DG2 has a PCIe 5.0 link.Yes the board needs to support it but you can limit it to one GPU link and one SSD link that are close to the CPU. The other lanes and chipset can be kept simple.
Yes X570 was hot but that is mainly because they used an Ryzen I/O instead of a specific chipset like the B550.
Yojimbo - Wednesday, May 26, 2021 - link
Yes, but how does that not make what I said less relevant? What you are saying is that the time stamp given to PCIe 1.0 is not at the same standard of the time stamp as PCIe 5.0 or PCIe 6.0, and the difference is in such a direction to give the impression that PCIe 6.0 is coming sooner after PCIe 1.0 than it really is. So if we had uniformity of time stamps then PCIe 6.0 would be behind a "doubling every three years" cadence from PCIe 1.0.mode_13h - Wednesday, May 26, 2021 - link
> more importantly how inadequate PCI, AGP and VESA were.Woah, that's a huge blurring-together of technologies spanning a whole decade!
IIRC, AGP 8x wasn't too slow, though a x16 PCIe 1.0 slot was almost 2x as fast. The bigger benefit of PCIe was for the non-graphics slots, which were generally stuck on 32-bit PCI @ 33 MHz.
p1esk - Wednesday, May 26, 2021 - link
So looks like we are going to go straight from PCIe 4.0 to PCIe 6.0 in a couple of years. Why did they even bother talking about PCIe 5.0?Yojimbo - Wednesday, May 26, 2021 - link
PCIe 5.0 is more important than PCIe 4.0 because CXL depends on it. For consumer applications the bus speed is not that important, but PCIe 4.0 was so late that it got to the point that the bus speed was becoming a bottleneck for consumers.p1esk - Wednesday, May 26, 2021 - link
Not disputing that, but if in two years you have a choice to build something new with PCIe 5.0 or PCIe 6.0, why would you go with PCIe 5.0?mode_13h - Wednesday, May 26, 2021 - link
> why would you go with PCIe 5.0?Because 6.0 will cost even more and burn even more power.
CiccioB - Thursday, May 27, 2021 - link
No, it won't.PCIe-6 is same speed as PCIe-5 but PAM4, that is 2 bit per clock instead on 1.
So the power requirements will be almost the same but with double the bandwidth.
It's the same technology Nvidia ha adopted for NVLink 3 a couple of years earlier.
Probably because even PCI consortium has understood that it is not possible to continue pumping up the frequencies to go faster.
PAM4 however means that the new standard is no more compatible with previous gen PICe, unless the controller has a fallback mode to PCIe-5.
mode_13h - Saturday, May 29, 2021 - link
> No, it won't.> PCIe-6 is same speed as PCIe-5 but PAM4, that is 2 bit per clock instead on 1.
> So the power requirements will be almost the same but with double the bandwidth.
You don't think PAM4 is more costly to implement and burns more power?
It's definitely more noise-sensitive, meaning boards will have to use more layers and more expensive materials, as well as probably retimers (which will also be more expensive and consume more power, thanks to PAM4).
> It's the same technology Nvidia ha adopted for NVLink 3 a couple of years earlier.
That's only 50 Gbps and introduced just last year. Even on a RTX 3090, it doesn't hit PCIe 5.0 rates, much less PCIe 6.0.
> PAM4 however means that the new standard is no more compatible with previous gen PICe
Read the article. From one of the slides:
"Maintains backward compatibility with all previous generations of PCIe architecture"
mode_13h - Wednesday, May 26, 2021 - link
> it got to the point that the bus speed was becoming a bottleneck for consumers.Not really. More like: "GPUs got to the point where they could *start* to reach the limits of PCIe 3.0". Go back and look at PCIe 3.0 SSDs and you won't see them even hit the link speed.
Yojimbo - Thursday, May 27, 2021 - link
I was talking about SSDs, not GPUsYojimbo - Thursday, May 27, 2021 - link
You don't think the release of SSDs faster than PCIe 3.0 speeds had something to do with a lack of market for them due to a lack of PCIe bandwidth?mode_13h - Saturday, May 29, 2021 - link
> You don't think the release of SSDs faster than PCIe 3.0 speedsI think AMD releasing support for PCIe 4 created a scramble among SSD makers to try and build a product for it. But you could take most 1st-gen PCIe 4 SSDs and run them at PCIe 3 and hardly measure a difference.
It's not at all like the situation we had with SATA SSDs, where they were link-limited for years.
mode_13h - Wednesday, May 26, 2021 - link
> looks like we are going to go straight from PCIe 4.0 to PCIe 6.0We're not. Sapphire Rapids and Alder Lake will use PCIe 5.0 this year or early next.
We'll see when PCIe 6.0 hits. Whatever happens with 5.0, in the consumer space (and I speculate it will not be much), don't expect 6.0 to follow any time soon.
p1esk - Thursday, May 27, 2021 - link
Yes, I know Intel is adding support for 5.0. But I'm not sure why. There are no PCIe 5.0 products on the horizon: no NICs, no SSDs, no video cards, nothing. PCIe 4.0 will remain fast enough for the next two years.So, in three years, if I am Nvidia, and CPUs added support for PCIe 6.0, why would I upgrade my products to 5.0 instead of 6.0? Low end cards don't need more than 4.0 - don't need to upgrade those. High end cards cost so much and consume so much power that any extra cost/power from 6.0 is insignificant. And there's a clear benefit from the increased bandwidth - otherwise why would Nvidia developed their proprietary NVLink?
Sure, Nvidia could upgrade their cards to 5.0 next year if they wanted to, but they just upgraded to 4.0 last year, so chances of that are slim.
mode_13h - Thursday, May 27, 2021 - link
> I know Intel is adding support for 5.0. But I'm not sure why.I think it could be just for the CPU -> chipset link. That would give them PCIe 5.0 bragging rights and let them shrink the width of that link back down to its historical x4 width. Moreover, it's soldered potentially right next to the CPU, making it the cheapest and simplest thing to connect via PCIe 5.
It'd also mean they could then drop the CPU back to x16 lanes, since the chipset would have plenty of bandwidth for all the NVMe cards people want to use, and even a respectable amount of bandwidth for a second GPU.
What flies in the face of that scenario is that the socket is also ballooning up to more than 1700 contacts. So, that casts doubt on the idea they'll be cutting back on any CPU-direct connectivity.
> in three years, ... and CPUs added support for PCIe 6.0
Why would they? If 5.0 adoption is low and 5.0 burns more power and increases board and peripheral costs, why would consumer CPUs ever go to 6?
> High end cards cost so much and consume so much power that any extra cost/power from 6.0 is insignificant.
Not sure about that. Cost is cost. And with Intel in the GPU race and if mining cools down, we could see GPU prices come back to Earth.
> there's a clear benefit from the increased bandwidth -
> otherwise why would Nvidia developed their proprietary NVLink?
That's not in most of their consumer cards, which is presumably what we're talking about.
Anyway, the RTX 3090 already has about 56 GB/s of NVLink bandwidth, compared to PCIe 5.0's 64 GB/s. However, the NVLink bandwidth is exclusive for GPU <-> GPU communication. NVidia has already announced a generation of NVLink beyond that.
I also don't see them ditching NVLink, because it scales better.
Yojimbo - Thursday, May 27, 2021 - link
CXL is why. And PCIe 4.0 is going to be short-lived. Both AMD and Intel will be changing over to PCIe 5.0 quickly. It remains to be seen how fast PCIe 6.0 is taken up, especially in the consumer space.Not sure why you think the chances are slim that NVIDIA will upgrade their cards to PCIe 5.0 The data center cards will certainly upgrade to 6.0 because 1) NVLink is built off PCIe and 2) they want to use CXL. As far as the gaming cards, what difference does it really make?
If you buy a motherboard an CPU Alder Lake or later it will be PCIe 5.0. Are you just going to hold off buying one because you think 6.0 will be out afterward? Why not just hold off for 7.0?
You seem to be trying very hard to make some sort of V with PCIe 4.0 and 6.0 at the tips and 5.0 at the trough. But it's not like that. People will buy PCIe 5.0 stuff because that's what will be available. And as far as consumers, they're going to need 6 even less than they will need 5. It seems SSDs will be able to take advantage of 5 when it comes out. But I wonder if they'll be able to take advantage of 6. Maybe if heterogeneous computing makes it to the PC then we'll see benefit of these higher bandwidths.
mode_13h - Saturday, May 29, 2021 - link
> CXL is why. And PCIe 4.0 is going to be short-lived.> Both AMD and Intel will be changing over to PCIe 5.0 quickly.
You're failing to draw any distinction between their consumer and server products. Consumer boards won't have CXL-support, for one thing. There's no reason for it.
> If you buy a motherboard an CPU Alder Lake or later it will be PCIe 5.0.
How do you know? Just because it said "PCIe 5" on a leaked roadmap? Nobody has yet shown me any evidence there will be actual PCIe 5 slots in a Alder Lake motherboard.
> It seems SSDs will be able to take advantage of 5
Just because there's server SSDs that burn like 25 W that can do it? That's not something you could fit in a M.2 slot. Not just for power and thermal reasons, but all the chips wouldn't even fit a M.2 board.
Again, consumer SSDs can barely peak above PCIe 4.0 speeds. Samsung just transitioned its mighty Pro line to TLC. Consumer SSDs are too limited by GB/$ to hit the kinds of speeds that $15k server SSDs can do, even in a AIC form factor.
mode_13h - Saturday, May 29, 2021 - link
> Again, consumer SSDsI meant to say "most PCIe 4 consumer SSDs".
schujj07 - Friday, May 28, 2021 - link
For consumer level products sure there won't be much available for PCIe 5.0. That won't be the case in the enterprise market. There are already switches that have 400GbE ports and PCIe 5.0 NICs are on the way. NVidia has already announced their ConnectX-7 cards with PCIe 5.0 x16/x32 links.schujj07 - Friday, May 28, 2021 - link
In the consumer space PCIe 5.0 is only important for SSDs. We have finally gotten to the point that PCIe 4.0 SSDs can in theory saturate an x4 link. For GPUs we aren't anywhere close to being limited on the bus speed.For the data center having PCIe 5.0 is VERY important for networking equipment. More and more hyperscalers are moving to faster and faster Ethernet. We were stuck on PCIe 3.0 for far too long in the data center and that stagnated network speeds and hyperconverged storage solutions. With the move to PCIe 4.0 we were able to get dual port 100GbE or single port 200GbE at bus speeds for each port. There were dual port PCIe 3.0 100GbE cards but you didn't get any benefit of the 2nd port as you were bus limited. Now with PCIe 5.0 you can get dual port 200GbE or single port 400GbE. This helps massively when you are running something like VMware vSAN for your storage. The idea is to make storage less of a bottleneck in the data center and the faster Ethernet in conjunction with faster SSD is making that possible. This ever increasing network bandwidth is also making it very hard on Fibre Channel as that is limited to 32Gb where iSCSI can do 200Gb right now.
mode_13h - Saturday, May 29, 2021 - link
> In the consumer space PCIe 5.0 is only important for SSDs. We have finally gotten to the> point that PCIe 4.0 SSDs can in theory saturate an x4 link. For GPUs we aren't anywhere
> close to being limited on the bus speed.
This is a laughable argument. In 2012, the first PCIe 3.0 GPUs could probably saturate a x16 link, but that doesn't mean we needed PCIe 4.0
Even a PCIe 4.0 consumer SSD being able to touch the limit of an x4 link, in peak speeds, doesn't justify transitioning to 5.0.
mode_13h - Wednesday, May 26, 2021 - link
> Starting the graph back at PCI in 1992 glosses over that PCI was too slow to begin with,> which is why there was VESA local bus
Huh? 486DX2 machines had VLB, because EISA wasn't fast enough! Once Pentiums came out, they pretty much immediately adopted the new PCI bus. And it was *plenty* fast, at the time!
It was only ~5 years later that PCI got stuck between 66 MHz and 64-bit, with neither seeming to gain mainstream acceptance. That's when Intel decided to push AGP, as the way forward. That lasted until 2004 or so, when PCIe hit the mainstream.
29a - Friday, July 2, 2021 - link
VLB did not replace PCI.Davdoc - Wednesday, July 21, 2021 - link
VESA local bus came before PCI, but, sorry to nitpick, the way you described it is as if VESA local bus came after PCI. Theoretical bandwidth wise VESA can go higher than the lowest iteration of PCI, indeed.I am unfortunately old enough to witness those transitions.
Kamen Rider Blade - Tuesday, May 25, 2021 - link
https://en.wikipedia.org/wiki/PCI_ExpressThe PCIe standards support: ×1, ×2, ×4, ×8, ×12, ×16, & ×32 lane implementations
Yet I don't see nearly as much support for ×2, ×12, & ×32 implementations.
Even in BiFurication of a ×16 port into ×12 + ×4.
I wish more vendors follow the PCIe spec and truly implement support across the board for all lane configs properly, not just the popular ones.
DigitalFreak - Tuesday, May 25, 2021 - link
There's not a need for PCIe x32 outside of servers, where they're used for riser cards. PCIe x2 doesn't really make sense and was only really used for cheap SSDs. x12 might make sense in a BiFurcation situation for x12 GPU and an x4 SSD, but now that both AMD and Intel already have a CPU x4 lane dedicated for SSDs it's not going to happen.Kamen Rider Blade - Tuesday, May 25, 2021 - link
But, the CPU x4 lane should be dedicated as the OS drive.The BiFuricated x12 + x4 = GPU + Game Drive where the Game Drive could have DMA to the GPU's RAM, ergo lowering latency & power requirements for sending data from the game to the GPU.
brucethemoose - Wednesday, May 26, 2021 - link
If you need 2 SSDs, and you need the game ssd directly connected the CPU in the future, the OS drive would be fine hanging off the chipset.If you REALLY need multiple SSDs all directly connected to the CPU, well, that's what HEDT is for.
mode_13h - Wednesday, May 26, 2021 - link
> Even in BiFurication of a ×16 port into ×12 + ×4.Dude, it's just "bifurcation".
https://en.wiktionary.org/wiki/bifurcate
The Von Matrices - Friday, July 2, 2021 - link
PCIe x2 is widespread, just not in slot form. In addition to some SSDs, a lot of integrated chips on motherboards use PCIe x2. For example, the ubiquitous ASMedia USB-C controllers all use PCIe x2.The Von Matrices - Friday, July 2, 2021 - link
x2 and x12 don't make sense in slot form because if you look at the pinout of the PCIe slot, an x2 or x12 card couldn't be made any smaller than an x4 or x16 card, respectively.ballsystemlord - Tuesday, May 25, 2021 - link
Spelling and grammar errors:"Meanwhile, as the PCIe 6.0 specification reaches competition, ..."
Competition, eh? With CXL maybe? I think you meant "completion".
"Meanwhile, as the PCIe 6.0 specification reaches completion, ..."
ballsystemlord - Tuesday, May 25, 2021 - link
Addendum: I didn't read the whole article.Ryan Smith - Tuesday, May 25, 2021 - link
Thanks!damianrobertjones - Wednesday, May 26, 2021 - link
Double every three years? Why? To max profits? Just go for the fastest you can achieve at the time. Stop milking people year after year.willis936 - Wednesday, May 26, 2021 - link
Because the market demands it. Technology actually improving isn't milking. That would be Intel selling 4 cores for the same price for 10 years.melgross - Wednesday, May 26, 2021 - link
It’s technology. It always improves. Giving a three year schedule is helpful for manufactures. Like it or not, that’s important.mode_13h - Wednesday, May 26, 2021 - link
> Double every three years? Why? To max profits?This ain't about you, dude. This is about hyperscalers who want 800 Gigabit Ethernet and about datacenter GPUs that want to stream vast amounts of data in/out of system RAM.
Santoval - Wednesday, May 26, 2021 - link
Does PCIe 6.0's 1b/1b encoding mean zero encoding overhead? Is this really possible? It's also unclear if "FLIT mode" is always active or not, and what exactly does this mean (I looked it up and FLIT is apparently the acronym for "Flow Control Unit").Elsewhere I read that that PCIe 6.0, due to the switch to PAM4 signaling, has a staggering 3 orders of magnitude (i.e. 1000 times) higher bit-error rate than PCIe 5.0, thus the need for FEC. While this sounds unreasonably high the error rate of PCIe up to version 5.0 is reportedly extremely low to begin with, so FEC can handle the error rate of PCIe 6.0 with minimal extra latency (the target is an astoundingly low 1 - 2 ns of extra latency due to FEC, while the FEC variant used on Ethernet adds ~100 ns of latency)
willis936 - Wednesday, May 26, 2021 - link
PAM4 certainly kills SNR. EQ tuning becomes massively important. It looks like they specify tuning of DFE and CTLE to minimize BER, but I suppose that only buys so much.Idk what you consider "astonishingly low" for effective BER, but anything above 10^-18 is too high for an interconnect this fast and with this application is way too high imo. I don't see the numbers mentioned in the article.
Line coding's signal integrity benefits are made obsolete by FEC. I don't full understand FLIT, but it appears to be their "lightweight" FEC implementation that allows for low latency. IEEE 802.3 know their business. A lot of this PAM4 is copy and paste from bs. I'm wary of places where they deviate, such as saying "100 ns is too long, we'll just make the FEC worse and retransmit more". That sends of red flags to me. Your effective throughput will greatly suffer if you rely on retransmission. Thou shalt not violate Shannon's limit.
mode_13h - Wednesday, May 26, 2021 - link
> the FEC variant used on Ethernet adds ~100 ns of latencyAt what speed of Ethernet? At 1 Gbps, a bit takes 1 ns. The CRC of an Ethernet frame is 32-bits. So, if that's what you're talking about, it should add 32 ns, at Gigabit speeds.
Really, rather than get carried away with ns, just look at the relative data sizes of FEC and the frame or packet size. The controller has to process it at wire-speed, and is almost certainly doing so as it reads bits (or PAM4 symbols) off the wire.
EthiaW - Wednesday, May 26, 2021 - link
So where is the ceiling of copper sockets?CiccioB - Thursday, May 27, 2021 - link
It depends on how far you want to go with the signals, which max latency you are targeting and most of all, how much energy you want to spend for each bit.There's not a theoretical maximum speed you can achieve.
CiccioB - Thursday, May 27, 2021 - link
Oh, I forgot the costs. It also depends on how much you want your transmission to be complex to build and so how much it will cost in term of pure $$$.mode_13h - Saturday, May 29, 2021 - link
This doesn't seem very useful. It would've been simpler and more accurate to say "I don't know", which is clearly the case. You probably never designed a PCB in your life.mode_13h - Wednesday, May 26, 2021 - link
Ryan Smith> All told we’re not expecting much different from 5.0Ryan Smith> (in other words, only a slot or two on most consumer motherboards)
And which consumer motherboards are you seeing this on, Ryan Smith?
Wereweeb - Wednesday, May 26, 2021 - link
Alder Lake is confirmed to use PCIe 5.0mode_13h - Thursday, May 27, 2021 - link
> Alder Lake is confirmed to use PCIe 5.0I know that, but it hasn't been disclosed where or how. I'm asking how the author seems to know that "most" of its consumer motherboards will have "only a slot or two").
To the extent of any evidence I've seen, Alder Lake could just use PCIe 5.0 for its DMI link. If Ryan Smith definitely knows otherwise, I'd like him to confirm.
CiccioB - Thursday, May 27, 2021 - link
Making a public announcement of supporting a technology and then bury it only for internal use (which is not interesting to any consumer) would not be a good move. IMHO.So AL will probably have an external link PCIe 5 (4 lanes?) from CPU towards what can be simply a single SSD slot, just to say: we can be faster than the competition, and at the first device supporting that technology they can beat on that nail for months.
The number of lanes will be few as they are very critical copper traces and cost a lot, so motherboards cannot be that expensive just for adding a technology that is not really beneficial today (and probably for the next 5 years but just for a small 0.x% amount of PC users).
I think that the balance will be having few lanes for imagine purpose but not too many for not going up with costs and energy consumption.
On laptop PCIe5 may be useful for limiting the lanes going towards the classic x4 connection towards the dGPU. GPU producer may spare some money and energy in supporting PCI5 with the added option to also advertising that feature just to give impression that the GPU will be even faster (you know, 3.0<4.0<5.0 and big numbers always matter! Give me a 200Mpixel sensor on that smartphone!).
mode_13h - Saturday, May 29, 2021 - link
> Making a public announcement of supporting a technology and then bury it only for> internal use (which is not interesting to any consumer) would not be a good move.
A lot of features and specs are just for bragging rights. Also, they could be trying to push AMD into a very expensive mistake.
Qasar - Friday, July 2, 2021 - link
how would this push amd to make a very expensive mistake ? couldnt it also push intel to do the same ?mode_13h - Sunday, July 4, 2021 - link
> how would this push amd to make a very expensive mistake ?If Intel is only adding PCIe 5.0 for the CPU-chipset link, all the noise they're making about it could scare AMD into adding it for all PCIe lanes, thinking that's what Intel is doing.
Qasar - Sunday, July 4, 2021 - link
why would it scare them ? when amd added pcie 4 to x570, intel fans were saying so ? its no big deal, most devices cant use it yet any way. which was true, but now, not so much, going from a 970 pro to a 980 pro for my OS drive, i noticed a speed increase when i installed win10, and a little bit of an increase for other things. amd could just do the same, use pcie 5 for the cpu to chipset link, and leave everything else as 4, for check box parity. has amd even mentioned what " x670 " could have for pcie support yet ?mode_13h - Monday, July 5, 2021 - link
> amd could just do the same, use pcie 5 for the cpu to chipset link,> and leave everything else as 4, for check box parity.
If they know, for sure, that's what Intel is doing.
They might have been planning to design AM5 for PCIe 4.0, only. Intel's noise about PCIe 5 could have caused AMD to second guess themselves, and add in support for PCIe 5. That could cause delays and increased costs (due to more pins & larger footprint, if nothing else).
Tomatotech - Sunday, July 4, 2021 - link
Thanks for the July 2021 update!If PCIe 6.0 is approaching finalisation, that must mean work is starting on PCIe 7.0? Any rumours about the direction it will take? Do we have a date yet for the initial 0.3 draft?
mode_13h - Monday, July 5, 2021 - link
I wonder if they could do like a 4.1 that runs at PCIe 4.0's clock speed, but uses PAM4 to deliver PCIe 5.0 bandwidth. I wonder if that would be cheaper (or more power-efficient) to implement on desktops & laptops than PCIe 5.Tomatotech - Tuesday, July 6, 2021 - link
History is littered with the graves of many half-assed technology - and many fully-assed techs. I’m sorry that sounds like a half-assed short term effort that won’t go mainstream.mode_13h - Wednesday, July 7, 2021 - link
Thanks for sharing your opinion. I can think of less insulting ways to reject my suggestion.mode_13h - Wednesday, July 7, 2021 - link
And the problem it solves isn't only short-term. The signal integrity requirements of PCIe 5 aren't going to relax with time. And that means they're always going to add costs to PCIe 5.0 boards vs. PCIe 4.0.back2future - Tuesday, July 6, 2021 - link
Does "Remaining System Base Board Budget" still provide enough margin for PCIe sockets for full device-to-device bandwidth compared to NVlink 2.0/3.0 (so for PCIe 5.0 for about (36 minus 2*(9.5 plus 1.5) minus (4-7 pcb copper safety margin)) remaining 9-12dB for burst data transfer states on ~4 GB/s per lane and direction)?While NVlink switches are especially designed for that purpose (up to now 25-50Gbit/s, while PCIe 5.0 is standardized to 32GT/s, means 16Ghz maximum clock rate for data for error correction, header data and precoding/decoding? also on (NRZ) signaling scheme for 128b/130b coding scheme). Physical limitations for NVlink show real 90-95% of standardized bandwidth, sample is ~35Gbit/s for 40Gbit/s link (2 sub-lanes uplink to P100 NV-Pascal GPU).
PCISIG tells like this "The signal attenuation caused by the channel insertion loss (IL) is the biggest challenge of PCIe 5.0 technology system design.", what is recognized on about halfing maximum line length with doubling clock rates (>PCIe 3.x).
Comparison of expectations/potential for PCIe 5.0 and NVlink 2.0/3.0 would be pretty useful and readworthy (especially PCIe device-to-device rates for AI accelerators and databases/inference models storage)
back2future - Tuesday, July 6, 2021 - link
correction: remaining 7-10dB (on previous assumptions) on burst data transfer statesmode_13h - Wednesday, July 7, 2021 - link
I don't get your point about device-to-device. PCIe is a switched network, not a bus.back2future - Wednesday, July 7, 2021 - link
(PCI was a bus.) PCIe connects devices point to point and can support DMA for devices without affecting other PCIe connections (or devices) within a switch (or maybe the root package == chipset?). Seems there is no protocol overhead for "DMA"? accesses over switches, but for CPU/SoC connections for directly connected PCIe devices (hardware managed?) this might differ?NVlink is called a bus system and NVlink bridge (for example) is a parallel bandwidth path additional to PCIe and data transfer is managed on each graphic card's NVlink controller? This changes on a NVlink switch that connects through 6-12 (V100-A100) switching interfaces up to 8-16 graphic devices (point to point through the switch)?
Maybe it is fair enough to summarize, that switches (for PCIe, NVlink) can connect devices like these were on physically paralleled lanes if suitable for "DMA"?
back2future - Wednesday, July 7, 2021 - link
A comment made on Anandtech, mentioning that errors, because of error handling are a bigger influence to data bandwidth (what was probably within networking hardware news on x00Gbit/s devices) one might expect, could ask for the following (ethernet <> PCIe, NVlink overhead, transmit order?)%) What are error handling routines within PCIe protocol (transaction, data link, physical layer), received on "error reporting register(s)", what's their average impact and are there common stats available (PCIe2.x - PCIe5.0), so devices can be considered being within specification limits?
%%) Are there new protocol versions or more likely driver conf parameters for data transfer that would not require, up to some amount of data, being retransmitted "free of errors" (like on AI inference methods, were predictions are made on probability) for then maybe increased throughput (configured from data analysts, application type)
one reference https://www.kernel.org/doc/Documentation/PCI/pciea... (articles explaining are detailed on web, there are userspace log tools for summarized stats to that already before PCIe5-PCIe6?)
Golgatha777 - Wednesday, July 21, 2021 - link
Looking forward to add-in cards for USB4, TB4, 10(+)GbE and M.2 SSDs with 1x or 2x interfaces on the consumer side. Eight 1x lanes with this kind of bandwidth is exciting for consumers.damianrobertjones - Wednesday, July 21, 2021 - link
"I/O bandwidth doubles every 3 years" - To ensure that we can milk customers over and over again.