It always good that company work together on common protocols. USB4 is good example of that.,
But I hope this stuff can bring some order to this PCIe mess - with AMD doing 4.0 and Intel doing 5.0 - bad thing is that there are going to be product out that don't work for other products. Yes 4.0 and 5.0 are backwards compatible but far as I understand 4.0 or 5.0 does not run on 3.0
yep.. it would just work at the cards pcie speed, or the slots depending on which is older/newer.. example, plug a pcie 5 card in a 3 slot, card runs at pcie 3
A PCIe5 x16 card will work in a PCIe1 x1 slot, just at PCIe1 x1 speeds, if it doesn't, then either the PCIe5 card or the PCIe1 slot are not to specification. It may not be useful to run a PCIe5 x16 card in a PCIe1 x1 slot, but it should work.
A PCIe1 x1 card will work in a PCIe5 x16 slot.
One of the features of PCIe is that devices perform an auto-negotiation with the slot they are put into to determine what specifications they support, and the devices (the slot and the card) will operate at the lowest common denominator of capabilities.
What it comes down to is you are more likely to have issues with the appropriate drivers for the platform you are running on, e.g. drivers written for an O/S running on a PCIe5 platform (windows 10, recent versions of Linux, AIX, MacOS and so on) probably won't work on the likely platform that has PCIe1, e.g. Windows NT4, or Linux 1.x Kernels, and so on, then PCIe slot incompatibility.
However, there should be no issues in PCIe3/4/5 driver compatibility since all three will be simultaneously 'current' standards at the same time, with PCIe3 still being in production when PCIe5 is available, therefore the same set of O/Ses, at least as far as major versioning is concerned, are likely to be current across all 3 PCIe generations, therefore drivers for the cards will likely be current across any typical O/S that will be running on those platforms (excluding custom super-computers, lab devices and so on).
No, a PCIe 5 x16 card will not work in a PCIe 1 x1 slot.
An x16 card will fit into an x16 slot. An x16 card will NOT fit into an x8 slot. An x16 card will NOT fit into an x4 slot. An x16 card will NOT fit into an x1 slot.
An x8 card will fit into an x16 slot An x8 card will fit into an x8 slot. An x8 card will NOT fit into an x4 slot. An x8 card will NOT fit into an x1 slot.
An x4 card will fit into an x16 slot An x4 card will fit into an x8 slot. An x4 card will fit into an x4 slot. An x4 card will NOT fit into an x1 slot.
An x2 card will fit into an x16 slot An x2 card will fit into an x8 slot. An x2 card will fit into an x4 slot. An x2 card will NOT fit into an x1 slot.
An x1 card will fit into an x16 slot An x1 card will fit into an x8 slot. An x1 card will fit into an x4 slot. An x1 card will fit into an x1 slot.
Notice the pattern? The slot needs to be the same size or larger than the connector.
(Talking about the physical connector for the card. The electrical connection for the card adds another wrinkle, but you can generally ignore that - ex x16 physical but only x8 electrical. If the connector fits in the slot, the card will work.)
Sorry, but you're just wrong about that. Any PCIe card of any generation can fit into any x1 slot of any generation. x2, x4, and x8 slots won't fit larger cards unless they are open-ended. In consumer-grade hardware, x4 slots are rare (and usually open-ended when they show up in modern hardware), and x8 slots are basically non-existent*.
* PCIe slots wired for x4 and x8 connections often use x16 physical slots anyway.
"x2, x4, and x8 slots won't fit larger cards unless they are open-ended."
That is purely a manufacturer choosing to make the piece of plastic that makes the receptacle a solid-bordered rectangle piece of plastic instead of a piece of plastic with the end cut out to fit a longer card. This is nothing to do with the PCIe spec. It is solely how the manufacturer built the support and retaining mechanism around the connector. This can be remedied by grabbing your favourite cutting tool and etching out that piece of plastic so now the connector of the card can sink into the slot with the extra length of the connector potruding out the end.
As per the Spec, an X16 card has to work in an x1 slot. Sure, of course there has tp be physical room for the card to be jammed in. I mean, in putting an x1 slot in, the manufacture could ahve envisopned only tiny 5cm cards will ever be put in, so they could have put a power supply or other components 5cm beyond the edge of the slot so you can't put a 10cm car in, even that that 10cm long card might be an x1 card as well.
Those are all constriants of the shape of the case, motherboard, and so on. Nothing to do with the electrical and signalling capabilities of the slot itself and/or those of the type of card (X1, xX, PCIe1, PCIe5).
"That is purely a manufacturer choosing to make the piece of plastic that makes the receptacle a solid-bordered rectangle piece of plastic instead of a piece of plastic with the end cut out to fit a longer card."
IOW, an x16 card won't fit into an x8 slot.
An x16 card will work with an x8 connector (after all, increasing the number of lanes just extends the length of the connector), but I have yet to see an open-ended PCIe slot on any motherboard (desktop or server; Supermicro, Tyan, Gigabyte, Asus, Asrock, or MSI). Maybe they exist, maybe they're part of the PCIe spec, but they aren't commonplace by any definition of the word.
Thus, an x16 card won't fit into any existing x8 slots.
"In consumer-grade hardware, x4 slots are rare (and usually open-ended when they show up in modern hardware), and x8 slots are basically non-existent*."
There are lots of x4 and x8 slots on better-than-garbage motherboards, and they are very commonplace on server motherboards (which is what I use more often than not). NICs, HBAs, RAID controllers, etc are generally x8 cards. We look for motherboards that have lots of x8 slots to fit these into. EPYC makes for wonderful storage and VM hosting servers as there's plenty of PCIe lanes to directly connect to storage. :) Lots of x8 cards can go into EPYC motherboards.
"but I have yet to see an open-ended PCIe slot on any motherboard (desktop or server; Supermicro, Tyan, Gigabyte, Asus, Asrock, or MSI). Maybe they exist, maybe they're part of the PCIe spec, but they aren't commonplace by any definition of the word."
I just went to Asrock's motherboards page (first one that crossed my mind), and on the landing page I can see 4 motherboards alone that have open-ended (x1 by the looks of things) connectors:
"Some slots use open-ended sockets to permit physically longer cards and negotiate the best available electrical and logical connection.
The number of lanes actually connected to a slot may also be fewer than the number supported by the physical slot size. An example is a ×16 slot that runs at ×4, which will accept any ×1, ×2, ×4, ×8 or ×16 card, but provides only four lanes. Its specification may read as "×16 (×4 mode)", while "×size @ ×speed" notation ("×16 @ ×4") is also common. The advantage is that such slots can accommodate a larger range of PCI Express cards without requiring motherboard hardware to support the full transfer rate."
"...but far as I understand 4.0 or 5.0 does not run on 3.0". They actually do. They also run on PCIe 2.0 and even the original PCIe 1.0(a). Up to PCIe 5.0 the backwards & forward compatibility is relatively easy, because fundamentally all versions employ the same signaling scheme : NRZ (Non-Return-to-Zero).
From PCIe 6.0 onward NRZ is canned and PAM4 (Pulse Amplitude Modulation with 4 signal levels) will be adopted instead. So, while PCIe 6.0 will still be backward/forward compatible with all previous versions it will be trickier to do that and extra work for NRZ mode operation will certainly be required.
A similar situation, but at a higher level, occurred when PCIe 3.0 switched to 128b/130b encoding. To remain compatible with PCIe 2.0 and PCIe 1.0(a) slots & devices, PCIe 3.0 devices & slots needed to also have a 8b/10b encoding mode. PCIe up to at least 6.0 will still employ 128b/130b encoding, so at least this will not also have to change as well (with its 1.5% overhead noone even thinks of changing it again, it makes no sense).
"Intel doing 5.0" you just showed up to say this didnt you. at least try creating some alt accounts its too obvious dude. also isnt it sunday in israel now ? no respect for sabath ?? PepeThink
since when do intel have PCIE 5? they are working on it as are AMD, difference being AMD went with 4 for now as its currently possible to use it with all the Ryzen 3000 CPUs I imagine you will see 5 on the next gen chips @ 5nm coming 2021 and intel will probably have similar with I dont know what chip about then as well, PCIE is backwards comp irrespective of generation it just defaults to the slot or path speed. what may not be easy to swallow for all of us is the increase in board costs, look at the uplift in costs for the current 4.0 chip set due to the big gains in power and transmission environment requirements I cant see 5.0 being any better in this respect.
alufan, they dont, HStewart is saying intel will skip pcie 4, and go straight to 5. there " could " be other reasons why the x570 boards cost so much over the x470, maybe the boards them selves cost more to make, i hear the MSI godlike board, is 12 layers.
Agree that this is good news! I also have this question: Am I correct that full-speed PCIe 5 at 32 GT/sec is essentially the speed of current Dual-Channel memory-to-CPU buses (with DDR 4 RAM)? Wow.
Dual Channel memory read speed is in the area of 40GB/s or so for "average" memory sticks. PCIe 5 with 16 lanes would be ~63 GB/s in one direction (64 minus overhead)
I wonder if mainstream will eventually move to more memory channel, but probably not.
I would assume that eventually everyone will move to PCIe 5.0, even if it'll be a while. And sure, at that point we'll likely be at DDR5 as well, which increases the bandwidth. But frequency alone only gets you so far, at some point it might be beneficial to go to Quad Channel.
But of course the complexity of routing 4 memory channel instead of 2, and all that, will probably mean it'll never be "mainstream", and people that require higher memory bandwidth should just go with a HEDT platform.
People keep assuming that the Intel approach of doing things the same way, only faster is how the industry will continue to move forward. The idea of needing more memory channels to get more bandwidth is limited, and assumes a lack of innovation. Even PCI Express may be replaced in the next six years with something new.
One thing you need to learn about the tech industry is "never" say "never". Latency of PCIE 4.0 is only around 150ns. PCIE 5.0 is even lower. It won't be long before latency is equal for both interfaces.
Oh and one more thing to add, latency is not the most critical element of DRAM. It helps, sure, but latency has remained pretty much unchanged (very little variation) as memory speeds have increased. This is because the memory controller of the CPU itself along with the traces/interconnects add more latency than the DRAM modules. The actual latency of a DRAM module is between 8 and 10ns. The latency between the CPU and DRAM module is usually between 50 and 70ns.
Actually, it is DRAM latency that has remained basically unchanged for the last 20+ years. That's because the fundamental operation of DRAM has remained unchanged for 20+ years. The only changes that have taken place are in the DRAM interfaces in order to deliver more bandwidth via increasing levels of prefetch/multiplexing.
Latency is a fundamental aspect of computer memory and a first order performance impact on actual real world workloads. And no, there is no memory module that is doing 8 or 10ns except for some very very specialized designs that cost order of magnitude what commodity dram costs.
No one is going to shift main memory to PCIe based technologies. I'm fairly confident in that. You say only, I say that is higher than loaded latency in a 4 socket system....
And no, it will be forever. Memory is on a dedicated low latency non-switched interface with full dedicated pathing and logic. Memory hanging off of PCIe Infinity, or Gen-Z++++++++++++ isn't going to ever be competitive.
I have multiple actual high performance CPUs across multiple architectures under my belt. Its not a matter of saying never say never, its a matter of physics.
Firstly, mainstream is still at PCIe 3.0 (even with PCIe 4.0 now partially available due to Zen2), i. e. a x16 slot is well below the typical DDR4 single-channel speed (2666 MT, 21 Gbyte/s).
PCIe 4.0 moves this up to 32 Gbyte/s, but for these platforms ram has moved to typically 3200 MT DDR4, 50 Gbyte/s in dual-channel. So even for PCIe 4.0, dual-channel is just fine.
DDR5 is in the pipeline and it will allow at least 32 Gbyte/s per channel and thus satisfy even a PCIe 5.0 x16 configuration with dual-channel. It is expected that DDR5 will specify up to 6400 MT/s per pin and move from the 1*64 pin interface of DDR4 to a 2*40 pin interface.
This means, mainstream will not move up to more than 2 channels as we know it. But mainstream will move to DDR5 in the next years and DDR5 comes with narrower channels which will be doubled. So we will see 2*2*40 bit interfaces in mainstream platforms instead of 2*64 bit interface, allowing - in the same pincount - for up to 128 Gbyte/s.
First of all, it will likely take at least two years before we see PCI-E 5 hit mainstream desktop and by then DDR5 will be a thing. Secondly, mainstream desktop might be limited to 16 PCI-E 5 lanes of bandwidth TOTAL. I don't think many people realize the number of high speed lanes available in mainstream platforms has gone down with each PCI-E generation. Back in the 2.0 days we had mainstream platforms with 44 high speed lanes from the northbridge, then with 3.0 we moved to ~16-24 lanes, now with 4.0 again we have just 16-24 lanes, except now you have to shut some devices down(namely SATA) to use all of the lanes. With 5.0, your primary PCI-E slot might only be an 8x slot, instead of the 16x we are used to.
@Anton Shilov: Could you please share the source for the last graph with silicon rollout timelines? Puzzled as to why Gen-Z rollout is so late, despite it having great specs - bandwidth, latency, memory semantics, coherency, etc.
That is an easy one. Gen Z relies on a couple extra techs, like FPGA and PCIe 5.0 to operate, just like CXL relies on PCIe 5.0 Phy. That means, they have to wait for PCIe 5.0 to hit products, among other considerations. It takes 18-24 months from publication of the new PCIe standard into incorporation in products. So, considering that means PCIe 5.0 likely will not show up in products until 2021, about the time AMD is on 5nm and Intel, if they meet their roadmap, is on 7nm, we will have to wait for the implementation into hardware until then. PCIe 6.0 spec is set to be published in 2021, which CXL explains the acceleration of that standard so that 4, 5, and 6 are all published within 2 years of each other (it was 3 years each for up to PCIe 3.0, then slowed to a 6 year wait for PCIe 4.0 published in 2017, then 5 in 2019 when 4.0 is implemented, 6 in 2021 when 5 is implemented, setting up for the harder hitting coherency being in place for 2023 with PCIe 6.0 and future implementations of Gen Z and CXL).
So it is not really late relative to the other standards, it is waiting on Phy to be introduced supporting it, along with completion of development for incorporation, which is time and money. Patience is key.
Gen-Z is basically Infiniband++. It is the 3rd time its been tried (Infiniband was the second major attempt). Little point in Gen-Z coming out until required and the infrastructure in place to support it. It pretty much will need CXL as a host interface.
We’ve updated our terms. By continuing to use the site and/or by logging into your account, you agree to the Site’s updated Terms of Use and Privacy Policy.
Facebook
Twitter
RSS
43 Comments
Back to Article
extide - Friday, July 19, 2019 - link
Honestly, it's smart and good for everyone if Intel & AMD both support a common coherent protocol. This could be very good.HStewart - Saturday, July 20, 2019 - link
It always good that company work together on common protocols. USB4 is good example of that.,But I hope this stuff can bring some order to this PCIe mess - with AMD doing 4.0 and Intel doing 5.0 - bad thing is that there are going to be product out that don't work for other products. Yes 4.0 and 5.0 are backwards compatible but far as I understand 4.0 or 5.0 does not run on 3.0
Korguz - Saturday, July 20, 2019 - link
sorry hstewart, pcie 4 and 5, remains backwards compatible with pcie 3HStewart - Sunday, July 21, 2019 - link
So is a pcie 4 or pcie 5 card work in pice 3 slot.I know a pcie 3 card should work in pice 4 or pcie 5 slot - that would be very stupid if that did not work.
Korguz - Sunday, July 21, 2019 - link
yep.. it would just work at the cards pcie speed, or the slots depending on which is older/newer.. example, plug a pcie 5 card in a 3 slot, card runs at pcie 3eldakka - Monday, July 22, 2019 - link
A PCIe5 x16 card will work in a PCIe1 x1 slot, just at PCIe1 x1 speeds, if it doesn't, then either the PCIe5 card or the PCIe1 slot are not to specification. It may not be useful to run a PCIe5 x16 card in a PCIe1 x1 slot, but it should work.A PCIe1 x1 card will work in a PCIe5 x16 slot.
One of the features of PCIe is that devices perform an auto-negotiation with the slot they are put into to determine what specifications they support, and the devices (the slot and the card) will operate at the lowest common denominator of capabilities.
What it comes down to is you are more likely to have issues with the appropriate drivers for the platform you are running on, e.g. drivers written for an O/S running on a PCIe5 platform (windows 10, recent versions of Linux, AIX, MacOS and so on) probably won't work on the likely platform that has PCIe1, e.g. Windows NT4, or Linux 1.x Kernels, and so on, then PCIe slot incompatibility.
However, there should be no issues in PCIe3/4/5 driver compatibility since all three will be simultaneously 'current' standards at the same time, with PCIe3 still being in production when PCIe5 is available, therefore the same set of O/Ses, at least as far as major versioning is concerned, are likely to be current across all 3 PCIe generations, therefore drivers for the cards will likely be current across any typical O/S that will be running on those platforms (excluding custom super-computers, lab devices and so on).
phoenix_rizzen - Wednesday, July 24, 2019 - link
No, a PCIe 5 x16 card will not work in a PCIe 1 x1 slot.An x16 card will fit into an x16 slot.
An x16 card will NOT fit into an x8 slot.
An x16 card will NOT fit into an x4 slot.
An x16 card will NOT fit into an x1 slot.
An x8 card will fit into an x16 slot
An x8 card will fit into an x8 slot.
An x8 card will NOT fit into an x4 slot.
An x8 card will NOT fit into an x1 slot.
An x4 card will fit into an x16 slot
An x4 card will fit into an x8 slot.
An x4 card will fit into an x4 slot.
An x4 card will NOT fit into an x1 slot.
An x2 card will fit into an x16 slot
An x2 card will fit into an x8 slot.
An x2 card will fit into an x4 slot.
An x2 card will NOT fit into an x1 slot.
An x1 card will fit into an x16 slot
An x1 card will fit into an x8 slot.
An x1 card will fit into an x4 slot.
An x1 card will fit into an x1 slot.
Notice the pattern? The slot needs to be the same size or larger than the connector.
(Talking about the physical connector for the card. The electrical connection for the card adds another wrinkle, but you can generally ignore that - ex x16 physical but only x8 electrical. If the connector fits in the slot, the card will work.)
asgallant - Wednesday, July 24, 2019 - link
Sorry, but you're just wrong about that. Any PCIe card of any generation can fit into any x1 slot of any generation. x2, x4, and x8 slots won't fit larger cards unless they are open-ended. In consumer-grade hardware, x4 slots are rare (and usually open-ended when they show up in modern hardware), and x8 slots are basically non-existent*.* PCIe slots wired for x4 and x8 connections often use x16 physical slots anyway.
eldakka - Wednesday, July 24, 2019 - link
"x2, x4, and x8 slots won't fit larger cards unless they are open-ended."That is purely a manufacturer choosing to make the piece of plastic that makes the receptacle a solid-bordered rectangle piece of plastic instead of a piece of plastic with the end cut out to fit a longer card. This is nothing to do with the PCIe spec. It is solely how the manufacturer built the support and retaining mechanism around the connector. This can be remedied by grabbing your favourite cutting tool and etching out that piece of plastic so now the connector of the card can sink into the slot with the extra length of the connector potruding out the end.
As per the Spec, an X16 card has to work in an x1 slot. Sure, of course there has tp be physical room for the card to be jammed in. I mean, in putting an x1 slot in, the manufacture could ahve envisopned only tiny 5cm cards will ever be put in, so they could have put a power supply or other components 5cm beyond the edge of the slot so you can't put a 10cm car in, even that that 10cm long card might be an x1 card as well.
Those are all constriants of the shape of the case, motherboard, and so on. Nothing to do with the electrical and signalling capabilities of the slot itself and/or those of the type of card (X1, xX, PCIe1, PCIe5).
phoenix_rizzen - Thursday, July 25, 2019 - link
"That is purely a manufacturer choosing to make the piece of plastic that makes the receptacle a solid-bordered rectangle piece of plastic instead of a piece of plastic with the end cut out to fit a longer card."IOW, an x16 card won't fit into an x8 slot.
An x16 card will work with an x8 connector (after all, increasing the number of lanes just extends the length of the connector), but I have yet to see an open-ended PCIe slot on any motherboard (desktop or server; Supermicro, Tyan, Gigabyte, Asus, Asrock, or MSI). Maybe they exist, maybe they're part of the PCIe spec, but they aren't commonplace by any definition of the word.
Thus, an x16 card won't fit into any existing x8 slots.
"In consumer-grade hardware, x4 slots are rare (and usually open-ended when they show up in modern hardware), and x8 slots are basically non-existent*."
There are lots of x4 and x8 slots on better-than-garbage motherboards, and they are very commonplace on server motherboards (which is what I use more often than not). NICs, HBAs, RAID controllers, etc are generally x8 cards. We look for motherboards that have lots of x8 slots to fit these into. EPYC makes for wonderful storage and VM hosting servers as there's plenty of PCIe lanes to directly connect to storage. :) Lots of x8 cards can go into EPYC motherboards.
eldakka - Friday, July 26, 2019 - link
"but I have yet to see an open-ended PCIe slot on any motherboard (desktop or server; Supermicro, Tyan, Gigabyte, Asus, Asrock, or MSI). Maybe they exist, maybe they're part of the PCIe spec, but they aren't commonplace by any definition of the word."I just went to Asrock's motherboards page (first one that crossed my mind), and on the landing page I can see 4 motherboards alone that have open-ended (x1 by the looks of things) connectors:
https://www.asrock.com/mb/AMD/X570%20Steel%20Legen...
https://www.asrock.com/mb/Intel/B365M%20Pro4/index...
https://www.asrock.com/mb/AMD/X570%20Pro4/index.as...
https://www.asrock.com/mb/AMD/X570%20Extreme4/inde...
eldakka - Friday, July 26, 2019 - link
Umm, ever heard of open ended slots? Obviously not.https://en.wikipedia.org/wiki/PCI_Express#Form_fac...
"Some slots use open-ended sockets to permit physically longer cards and negotiate the best available electrical and logical connection.
The number of lanes actually connected to a slot may also be fewer than the number supported by the physical slot size. An example is a ×16 slot that runs at ×4, which will accept any ×1, ×2, ×4, ×8 or ×16 card, but provides only four lanes. Its specification may read as "×16 (×4 mode)", while "×size @ ×speed" notation ("×16 @ ×4") is also common. The advantage is that such slots can accommodate a larger range of PCI Express cards without requiring motherboard hardware to support the full transfer rate."
Santoval - Saturday, July 20, 2019 - link
"...but far as I understand 4.0 or 5.0 does not run on 3.0".They actually do. They also run on PCIe 2.0 and even the original PCIe 1.0(a). Up to PCIe 5.0 the backwards & forward compatibility is relatively easy, because fundamentally all versions employ the same signaling scheme : NRZ (Non-Return-to-Zero).
From PCIe 6.0 onward NRZ is canned and PAM4 (Pulse Amplitude Modulation with 4 signal levels) will be adopted instead. So, while PCIe 6.0 will still be backward/forward compatible with all previous versions it will be trickier to do that and extra work for NRZ mode operation will certainly be required.
A similar situation, but at a higher level, occurred when PCIe 3.0 switched to 128b/130b encoding. To remain compatible with PCIe 2.0 and PCIe 1.0(a) slots & devices, PCIe 3.0 devices & slots needed to also have a 8b/10b encoding mode. PCIe up to at least 6.0 will still employ 128b/130b encoding, so at least this will not also have to change as well (with its 1.5% overhead noone even thinks of changing it again, it makes no sense).
azfacea - Sunday, July 21, 2019 - link
"Intel doing 5.0"you just showed up to say this didnt you. at least try creating some alt accounts its too obvious dude.
also isnt it sunday in israel now ? no respect for sabath ?? PepeThink
alufan - Monday, July 22, 2019 - link
since when do intel have PCIE 5? they are working on it as are AMD, difference being AMD went with 4 for now as its currently possible to use it with all the Ryzen 3000 CPUs I imagine you will see 5 on the next gen chips @ 5nm coming 2021 and intel will probably have similar with I dont know what chip about then as well, PCIE is backwards comp irrespective of generation it just defaults to the slot or path speed.what may not be easy to swallow for all of us is the increase in board costs, look at the uplift in costs for the current 4.0 chip set due to the big gains in power and transmission environment requirements I cant see 5.0 being any better in this respect.
Qasar - Tuesday, July 23, 2019 - link
alufan, they dont, HStewart is saying intel will skip pcie 4, and go straight to 5. there " could " be other reasons why the x570 boards cost so much over the x470, maybe the boards them selves cost more to make, i hear the MSI godlike board, is 12 layers.eastcoast_pete - Friday, July 19, 2019 - link
Agree that this is good news! I also have this question: Am I correct that full-speed PCIe 5 at 32 GT/sec is essentially the speed of current Dual-Channel memory-to-CPU buses (with DDR 4 RAM)? Wow.nevcairiel - Friday, July 19, 2019 - link
Dual Channel memory read speed is in the area of 40GB/s or so for "average" memory sticks.PCIe 5 with 16 lanes would be ~63 GB/s in one direction (64 minus overhead)
I wonder if mainstream will eventually move to more memory channel, but probably not.
willis936 - Friday, July 19, 2019 - link
Main memory can’t afford the latency.mode_13h - Saturday, July 20, 2019 - link
The latency of what - more channels?DanNeely - Sunday, July 21, 2019 - link
The latency of PCIe I assumenevcairiel - Saturday, July 20, 2019 - link
Quad Channel memory on eg. the Skylake-X platform doesn't really have a much higher latency, while doubling the theoretical bandwidth.mode_13h - Friday, July 19, 2019 - link
I wonder if mainstream will eventually move to PCIe 5.0, but probably not.ats - Saturday, July 20, 2019 - link
Never. The latency overhead of all these coherent I/O protocols is way to high to be viable for main memory.mode_13h - Saturday, July 20, 2019 - link
My point was that nevcairiel's concerns with memory bandwidth seemed rather premature, if not altogether presumptuous.Besides, the era of DDR4 will be past, if PCIe 5.0 ever hits the mainstream.
nevcairiel - Saturday, July 20, 2019 - link
I would assume that eventually everyone will move to PCIe 5.0, even if it'll be a while.And sure, at that point we'll likely be at DDR5 as well, which increases the bandwidth. But frequency alone only gets you so far, at some point it might be beneficial to go to Quad Channel.
But of course the complexity of routing 4 memory channel instead of 2, and all that, will probably mean it'll never be "mainstream", and people that require higher memory bandwidth should just go with a HEDT platform.
mode_13h - Tuesday, July 23, 2019 - link
Well, PCIe 5.0 uses yet more power and increases board costs. That's why I'm not sure it's destined for the mainstream.As for memory bandwidth, perhaps CPUs with in-package HBM-style memory will eventually happen.
Targon - Sunday, July 21, 2019 - link
People keep assuming that the Intel approach of doing things the same way, only faster is how the industry will continue to move forward. The idea of needing more memory channels to get more bandwidth is limited, and assumes a lack of innovation. Even PCI Express may be replaced in the next six years with something new.eek2121 - Saturday, July 20, 2019 - link
One thing you need to learn about the tech industry is "never" say "never". Latency of PCIE 4.0 is only around 150ns. PCIE 5.0 is even lower. It won't be long before latency is equal for both interfaces.eek2121 - Saturday, July 20, 2019 - link
Oh and one more thing to add, latency is not the most critical element of DRAM. It helps, sure, but latency has remained pretty much unchanged (very little variation) as memory speeds have increased. This is because the memory controller of the CPU itself along with the traces/interconnects add more latency than the DRAM modules. The actual latency of a DRAM module is between 8 and 10ns. The latency between the CPU and DRAM module is usually between 50 and 70ns.ats - Sunday, July 21, 2019 - link
Actually, it is DRAM latency that has remained basically unchanged for the last 20+ years. That's because the fundamental operation of DRAM has remained unchanged for 20+ years. The only changes that have taken place are in the DRAM interfaces in order to deliver more bandwidth via increasing levels of prefetch/multiplexing.Latency is a fundamental aspect of computer memory and a first order performance impact on actual real world workloads. And no, there is no memory module that is doing 8 or 10ns except for some very very specialized designs that cost order of magnitude what commodity dram costs.
ats - Sunday, July 21, 2019 - link
No one is going to shift main memory to PCIe based technologies. I'm fairly confident in that. You say only, I say that is higher than loaded latency in a 4 socket system....And no, it will be forever. Memory is on a dedicated low latency non-switched interface with full dedicated pathing and logic. Memory hanging off of PCIe Infinity, or Gen-Z++++++++++++ isn't going to ever be competitive.
I have multiple actual high performance CPUs across multiple architectures under my belt. Its not a matter of saying never say never, its a matter of physics.
thomasg - Saturday, July 20, 2019 - link
Firstly, mainstream is still at PCIe 3.0 (even with PCIe 4.0 now partially available due to Zen2), i. e. a x16 slot is well below the typical DDR4 single-channel speed (2666 MT, 21 Gbyte/s).PCIe 4.0 moves this up to 32 Gbyte/s, but for these platforms ram has moved to typically 3200 MT DDR4, 50 Gbyte/s in dual-channel.
So even for PCIe 4.0, dual-channel is just fine.
DDR5 is in the pipeline and it will allow at least 32 Gbyte/s per channel and thus satisfy even a PCIe 5.0 x16 configuration with dual-channel.
It is expected that DDR5 will specify up to 6400 MT/s per pin and move from the 1*64 pin interface of DDR4 to a 2*40 pin interface.
This means, mainstream will not move up to more than 2 channels as we know it. But mainstream will move to DDR5 in the next years and DDR5 comes with narrower channels which will be doubled.
So we will see 2*2*40 bit interfaces in mainstream platforms instead of 2*64 bit interface, allowing - in the same pincount - for up to 128 Gbyte/s.
voicequal - Friday, July 19, 2019 - link
DDR4-3200 dual channel:64-bit * 1600 MHz * 2 DDR * 2 channel = 51.2 GB/s
PCIe 5.0 x16:
32.0 GT/s * 128b/130b * 16 lanes = 63.02 GB/s
PCIe 5.0 has more bandwidth and has 16 lanes in each direction, whereas the DDR4 memory bus is shared for both reads & writes.
qwertymac93 - Saturday, July 20, 2019 - link
First of all, it will likely take at least two years before we see PCI-E 5 hit mainstream desktop and by then DDR5 will be a thing.Secondly, mainstream desktop might be limited to 16 PCI-E 5 lanes of bandwidth TOTAL. I don't think many people realize the number of high speed lanes available in mainstream platforms has gone down with each PCI-E generation. Back in the 2.0 days we had mainstream platforms with 44 high speed lanes from the northbridge, then with 3.0 we moved to ~16-24 lanes, now with 4.0 again we have just 16-24 lanes, except now you have to shut some devices down(namely SATA) to use all of the lanes. With 5.0, your primary PCI-E slot might only be an 8x slot, instead of the 16x we are used to.
Luffy1piece - Friday, July 19, 2019 - link
@Anton Shilov: Could you please share the source for the last graph with silicon rollout timelines? Puzzled as to why Gen-Z rollout is so late, despite it having great specs - bandwidth, latency, memory semantics, coherency, etc.ajc9988 - Friday, July 19, 2019 - link
That is an easy one. Gen Z relies on a couple extra techs, like FPGA and PCIe 5.0 to operate, just like CXL relies on PCIe 5.0 Phy. That means, they have to wait for PCIe 5.0 to hit products, among other considerations. It takes 18-24 months from publication of the new PCIe standard into incorporation in products. So, considering that means PCIe 5.0 likely will not show up in products until 2021, about the time AMD is on 5nm and Intel, if they meet their roadmap, is on 7nm, we will have to wait for the implementation into hardware until then. PCIe 6.0 spec is set to be published in 2021, which CXL explains the acceleration of that standard so that 4, 5, and 6 are all published within 2 years of each other (it was 3 years each for up to PCIe 3.0, then slowed to a 6 year wait for PCIe 4.0 published in 2017, then 5 in 2019 when 4.0 is implemented, 6 in 2021 when 5 is implemented, setting up for the harder hitting coherency being in place for 2023 with PCIe 6.0 and future implementations of Gen Z and CXL).So it is not really late relative to the other standards, it is waiting on Phy to be introduced supporting it, along with completion of development for incorporation, which is time and money. Patience is key.
catavalon21 - Friday, July 19, 2019 - link
Ian tweeted the link a few weeks agohttps://www.plda.com/blog/category/technical-artic...
catavalon21 - Friday, July 19, 2019 - link
Also, it's the "PLDA" source at the end of the article.Luffy1piece - Monday, July 22, 2019 - link
Thanks for the link :)ats - Saturday, July 20, 2019 - link
Gen-Z is basically Infiniband++. It is the 3rd time its been tried (Infiniband was the second major attempt). Little point in Gen-Z coming out until required and the infrastructure in place to support it. It pretty much will need CXL as a host interface.Elstar - Saturday, July 20, 2019 - link
Does Anandtech have editors? You want "and" in the following sentence (not "but"):The industry group is led by nine industry giants including Intel, Alibaba, Google, and Microsoft, AND has over 35 members.
ksec - Saturday, July 20, 2019 - link
And we have PCI-E 6.0 in development, that gives upgrade path to CXL 2.0