Your concept of "internal throughput" has no basis in reality. File copies (on a filesystem that does not do copy-on-write) require the file data to be read from the SSD into system DRAM, then written back to the SSD. There are no "copy" commands in the NVMe command set.
"There are no "copy" commands in the NVMe command set." --------------------------------------------------------------------------------- That might be fixed with a few more onboard processors in the future but does not answer my question
How fast can you copy/paste 100GB on THAT specific drive?
How will a new processor change that there is no way to tell the drive to do what you want? We don't trust storage devices to "do what I mean", because the cost of a mistake is too high. No device anyone should be using will say "it looks like they're writing back the data they just read in, I'mma ignore the input and duplicate it from the cache to save time." Especially since they can't know if the data is changed in advance.
Barring a new interface standard, it will take exactly as long to copy a file to another location on the same drive as it will to read the file and then write the file, because that is the only provision within the NVMe command set.
What would happen if Intel Colludes with AMD to implement this technology into onboard graphics instead of AMD's plan to use Flash in their graphics cards ?
Seems to me like Internal throughput would be very important to the design
That is also file system dependent. For example, in Mac OS High Sierra, you can copy and paste (duplicate) any size file instantly on any drive formatted with APFS.
But your question of a block by block transfer of a file internally for a 100GB file would likely take 50 seconds if not factoring in file system efficiency.
That's not a copy of the file though. It's just a duplicate file entry referencing the same blocks. That and things like snapshots are possible thanks to the copy on write nature of that file system. But, if any of those blocks were to become corrupted, both 'copies' of the file are corrupt.
I noticed that the Windows Fall Crappier Edition takes MUCH longer to copy/move/paste in several tests than the earlier versions of "Spyware Platform 10"
as well as gives me a "Format Disk Now?" Prompt after formatting a new disk with Disk Manager as well as making a zero byte backup with Acronis 2015 (Possible anomaly, Will test again) as well as breaking compatibility with several programs that worked in earlier versions as well as asking permission to do anything with my files and then often failing to do it as well as, well.....you get the idea, they fix one thing and break 5 more
Disclaimer: I do NOT believe that xpoint could be used in its current form for onboard graphics! But I'd like to know that the numbers you are getting at AnandTech match your/my expectations and if not, why not?
Sorry if I'm sounding like an AHole but I'd like to know what this drive can really do, and then what Microsoft graciously allows it to do ?
It doesn't make sense to even care what a client OS would do with this drive. It's enterprise hardware and is priced/positioned accordingly. Instead of the P4800X, check out the consumer 900p version of Octane instead. It's a lot more likely that model will end up sitting in a consumer PC or an office workstation.
The drives dissipate 12-18W, which is 2-3x more than flash based SSDs. I don't think the m.2 form factor would make heat dissipation easy, unless you scale the design back down to the 32-120GB range.
More than just capacity, you'd almost certainly have to scale down performance by a lot. It's possible to do better than the current PCIe x2 32GB module, but a 7 channel controller would be hard to fit physically, and it would have to run slower to not melt. I think a ~3 channel M.2 drive is plausible, but I'm not sure its write speeds would be high enough.
it looks like you could physically do 7 (or even 8) channels on a dual sided 22110 m.2 module with the current package size. The thermals would be a real challenge though; and are why I really wish U.2 would've gained at least a little traction in the consumer market.
I was thinking more along the lines of a 22x80mm card. I agree that 7 3D XPoint packages is probably possible on 22x110, but I'm not sure how dense the controller's BGA pad can get.
If you went 2 sided you could fit 6 on a 2280 card. With a bare bottom they're just short on enough room for a 3rd package on top; move some of that misc stuff underneath and you'd free up the PCB area needed.
This is the datacenter model, good for releasing higher capacity sooner with its size. Plus rackmount cooling design will help. U.2. Will be for hot swap, the card will not. The M.2 enthusiast consumer model is the 900P
XPoint technology is just one of the reasons Intel released the "ruler" form factor for future SSD's. I suspect a mobile variant will replace m2 in the future.
Optane as a gen 1 product is pretty fantastic. I can't wait to see how the rest of the market will compete, and what Intel and Micron (where are their drives?) can do for round two.
I thought the whole point of 3D XPoint memory would be that it is DIMM friendly too. 1) When will we see this in DIMM form? 2) Would the DIMM version also need/have the reserved/spare capacity? 3) Why is this spare capacity even needed? 1/6th seems like a potentially high internal failure rate (or other fundamental problem.)
It seems like current 3D XPoint doesn't have enough endurance yet to sit in a DIMM slot unless it's gonna be just a storage drive in DIMM form factor. That and because we're only just now seeing early enterprise and retail products, I bet that we're gonna need another generation or two before we get DIMM versions. :(
Intel hasn't said much about 3D XPoint DIMMs this year, other than to promise we'll hear more next year.
It's not clear how much fault tolerance Intel is trying to build in to the Optane SSDs with the extra capacity. A bit of it is necessary to support sector formats with protection information (eg. 16 extra bytes per 512B sector, or 128B extra for each 4kB sector). Beyond that, there needs to be room for the drive's internal data structures, which aren't as complicated as a flash translation layer but still impose some space overhead. The rest is probably taken by a fairly simple erasure coding/ECC scheme, because it's almost impossible to do LDPC at the speed necessary for this drive. (That's also why DIMMs use simple ECC instead of more space-efficient codes.)
Most all intel SSDs have a parity die, so one full die likely provides internal raid protection of data. The rest is for ECC, internal system information, media management and mapping out defects... Impossible to know which of these is driving the actual spare implemented. I count 14 packages, so 1/14th (7%) is already the internal parity. 16% is big relative to nand consumer SSDs but comparable to enterprise. Doesn't seem particularly out of line or indicative of something wrong.
Intel is working on DIMMs... "Now, of course, SSDs are important, but in the long run, Intel also wants to have Optane 3D XPoint memory slot into the same sockets as DDR4 main memory, and Krzanich brought a mechanical model of an Optane DIMM to show off." https://www.nextplatform.com/2015/10/28/intel-show...
I’m curious as to how this will perform when PCI 4 is out next year. That is, one with a PCI 4 interface. How throughput limiting is PCI 3 for this right now?
It shouldn't be that limiting, as PCIe 3.0 x4 allows for a higher throughput than 2.4 GB/s. There could be some latency improvements (probably small), but I don't think throughput is the issue
If the question is "could a drive be made to saturate gen 4?" then, yes, of course, if intel chooses to do so. Will require a whole drive. Latency is a more interesting question because that is what 3dxp is really providing. QD1 latency is <10us ((impressive!). I don't expect that to improve since it should be limited by the 3dxp itself. The PCIe and driver overhead is probably 5us of that. Maybe gen 4 will improve that part.
And SSD is a system and would neither over nor under engineer any one piece significantly. So, altering one piece should not be expected to suddenly pop to some much higher performance. Will require a whole new drive.
He is always good for a giggle. I suppose he's busy directing hard drive manufacturers to make special hard drive platters for him solely out of hand-gathered sand from the Sahara. Or something.
Still it's a shame to miss the laughs. It's always the second thing I do on SSD articles - first read the conclusion, then go and see what deedee has said. Ah well.
30 DWPD over the course of 5 years turns into a really large amount of data when you're talking about 750GB of capacity. Isn't the typical endurance rating more like 0.3 DPWD for enterprise solid state?
So this thing about Optane on DIMMs is really interesting to me. Is the plan for it to replace RAM and storage all at once or to act as a cache or some sort between faster DRAM and conventional solid state? Even with the endurance its offering right now, it seems like it would need to be more durable still for it to replace RAM.
Oh (sorry case of shinies) could this be like a DIMM behind HBM on the CPU package where HBM does more of the write heavy stuff and then Optane lives between HBM and SSD or HDD storage? Has Intel let much out of the bag about this sorta thing?
Enterprise SSDs are usually sorted into two or three endurance tiers. Drives meant for mostly-read workloads typically have endurance ratings of 0.3 DWPD. High-endurance drives for write-intensive uses are usually 10, 25 or 30 DWPD, but the ratings of high-endurance drives have decayed somewhat in recent years as the market realized few applications really need that much endurance.
Yes. It makes for a great swap device, especially with a recent Linux kernel. Alternatively, Intel will sell it bundled with a hypervisor that presents the guest OS with a pool of memory equal in size to the system's DRAM plus about 85% of the Optane drive's capacity. The hypervisor manages memory placement, so from the guest OS's perspective the memory is a homogeneous pool, not x GB of DRAM and y GB of Optane.
It's a bit odd Intel would go for the hypervisor solution since the kernel can handle tiered pmem and it's in a better position to know where to place data. I suppose it's useful because it's cross-platform?
I’d guess a hypervisor solution would also allow any critical fixes to be propagated faster/easier than having to go through a 3rd party (kernel) provider?
Since this is for enterprise, the os vendor would be the one responsible (so, yes, third party) and one of the reasons why you pay them ridiculous support fees is for them to be your single point of contact for most issues.
Very nice write-up. Might it be possible for us to get an idea of the difference in cell access times by running a couple tests on a loop device, and, even better, purely dram-based storage accessed over pcie?
Is there any tests of the 4800X in a virtual host? Either Hyper-V or ESX, running multiple server OS clients with a variety of workloads. With the kind of low latency shown, I'd love to see how much more responsive Optane is compared to all flash storage like a P3608. Sort of a" rising tide floats all ships" kind of improvement, I hope.
I'm not sure what you mean. Nobody seriously considers the Windows I/O system to be more advanced than what Linux provides. Even Intel's documentation states that the best latency they can get out of the Optane SSD on Windows is a few microseconds slower than on the Linux NVMe driver, and on Linux a few more microseconds can be saved using SPDK.
"Advanced" may be the wrong way to look at it because ntkrnl can perform both sync and async operations, while Linux is essentially a sync-based kernel (the limitations surrounding its aio system are legendary). However, by focusing on doing that one thing well the block subsystem has become highly optimized for enterprise workloads. Btw, is there any chance you could run that block system (and nvme protocol, if possible) overhead test i asked about?
We’ve updated our terms. By continuing to use the site and/or by logging into your account, you agree to the Site’s updated Terms of Use and Privacy Policy.
Facebook
Twitter
RSS
58 Comments
Back to Article
"Bullwinkle J Moose" - Thursday, November 9, 2017 - link
Humor me.....How fast can you copy and paste a 100GB file from and to the same Optane SSD
I don't believe your mixed mode results adequately demonstrate the internal throughput
At least not until you demonstrate a direct comparison
Billy Tallis - Thursday, November 9, 2017 - link
Your concept of "internal throughput" has no basis in reality. File copies (on a filesystem that does not do copy-on-write) require the file data to be read from the SSD into system DRAM, then written back to the SSD. There are no "copy" commands in the NVMe command set."Bullwinkle J Moose" - Thursday, November 9, 2017 - link
"There are no "copy" commands in the NVMe command set."---------------------------------------------------------------------------------
That might be fixed with a few more onboard processors in the future but does not answer my question
How fast can you copy/paste 100GB on THAT specific drive?
"Bullwinkle J Moose" - Thursday, November 9, 2017 - link
Better yet, I'd like you to GUESS how fast it can copy and paste based on your mixed mode analysis and then go measure itLord of the Bored - Friday, November 10, 2017 - link
How will a new processor change that there is no way to tell the drive to do what you want? We don't trust storage devices to "do what I mean", because the cost of a mistake is too high. No device anyone should be using will say "it looks like they're writing back the data they just read in, I'mma ignore the input and duplicate it from the cache to save time." Especially since they can't know if the data is changed in advance.Barring a new interface standard, it will take exactly as long to copy a file to another location on the same drive as it will to read the file and then write the file, because that is the only provision within the NVMe command set.
"Bullwinkle J Moose" - Thursday, November 9, 2017 - link
What would happen if Intel Colludes with AMD to implement this technology into onboard graphics instead of AMD's plan to use Flash in their graphics cards ?Seems to me like Internal throughput would be very important to the design
Samus - Thursday, November 9, 2017 - link
That is also file system dependent. For example, in Mac OS High Sierra, you can copy and paste (duplicate) any size file instantly on any drive formatted with APFS.But your question of a block by block transfer of a file internally for a 100GB file would likely take 50 seconds if not factoring in file system efficiency.
cygnus1 - Thursday, November 9, 2017 - link
That's not a copy of the file though. It's just a duplicate file entry referencing the same blocks. That and things like snapshots are possible thanks to the copy on write nature of that file system. But, if any of those blocks were to become corrupted, both 'copies' of the file are corrupt."Bullwinkle J Moose" - Thursday, November 9, 2017 - link
Good call SamusI noticed that the Windows Fall Crappier Edition takes MUCH longer to copy/move/paste in several tests than the earlier versions of "Spyware Platform 10"
as well as gives me a "Format Disk Now?" Prompt after formatting a new disk with Disk Manager
as well as making a zero byte backup with Acronis 2015 (Possible anomaly, Will test again)
as well as breaking compatibility with several programs that worked in earlier versions
as well as asking permission to do anything with my files and then often failing to do it
as well as, well.....you get the idea, they fix one thing and break 5 more
Disclaimer:
I do NOT believe that xpoint could be used in its current form for onboard graphics!
But I'd like to know that the numbers you are getting at AnandTech match your/my expectations and if not, why not?
Sorry if I'm sounding like an AHole but I'd like to know what this drive can really do, and then what Microsoft graciously allows it to do ?
make sense?
PeachNCream - Friday, November 10, 2017 - link
It doesn't make sense to even care what a client OS would do with this drive. It's enterprise hardware and is priced/positioned accordingly. Instead of the P4800X, check out the consumer 900p version of Octane instead. It's a lot more likely that model will end up sitting in a consumer PC or an office workstation.AlaskanDruid - Thursday, November 9, 2017 - link
Ok. what am I missing? What is the use of these when you can have one in a m.2 form which takes up MUCH less room?I know I'm missing something, but what is it?
Drumsticks - Thursday, November 9, 2017 - link
The drives dissipate 12-18W, which is 2-3x more than flash based SSDs. I don't think the m.2 form factor would make heat dissipation easy, unless you scale the design back down to the 32-120GB range.Billy Tallis - Thursday, November 9, 2017 - link
More than just capacity, you'd almost certainly have to scale down performance by a lot. It's possible to do better than the current PCIe x2 32GB module, but a 7 channel controller would be hard to fit physically, and it would have to run slower to not melt. I think a ~3 channel M.2 drive is plausible, but I'm not sure its write speeds would be high enough.DanNeely - Thursday, November 9, 2017 - link
it looks like you could physically do 7 (or even 8) channels on a dual sided 22110 m.2 module with the current package size. The thermals would be a real challenge though; and are why I really wish U.2 would've gained at least a little traction in the consumer market.Billy Tallis - Thursday, November 9, 2017 - link
I was thinking more along the lines of a 22x80mm card. I agree that 7 3D XPoint packages is probably possible on 22x110, but I'm not sure how dense the controller's BGA pad can get.DanNeely - Thursday, November 9, 2017 - link
If you went 2 sided you could fit 6 on a 2280 card. With a bare bottom they're just short on enough room for a 3rd package on top; move some of that misc stuff underneath and you'd free up the PCB area needed.Rwbeckman - Thursday, November 9, 2017 - link
This is the datacenter model, good for releasing higher capacity sooner with its size. Plus rackmount cooling design will help. U.2. Will be for hot swap, the card will not. The M.2 enthusiast consumer model is the 900PSamus - Thursday, November 9, 2017 - link
XPoint technology is just one of the reasons Intel released the "ruler" form factor for future SSD's. I suspect a mobile variant will replace m2 in the future.https://www.anandtech.com/show/11702/intel-introdu...
CheapSushi - Thursday, November 9, 2017 - link
Or just use the new M.3 format (wider) that Samsung came out with.Drumsticks - Thursday, November 9, 2017 - link
Great review Billy, thank you!Optane as a gen 1 product is pretty fantastic. I can't wait to see how the rest of the market will compete, and what Intel and Micron (where are their drives?) can do for round two.
Elstar - Thursday, November 9, 2017 - link
I thought the whole point of 3D XPoint memory would be that it is DIMM friendly too.1) When will we see this in DIMM form?
2) Would the DIMM version also need/have the reserved/spare capacity?
3) Why is this spare capacity even needed? 1/6th seems like a potentially high internal failure rate (or other fundamental problem.)
PeachNCream - Thursday, November 9, 2017 - link
It seems like current 3D XPoint doesn't have enough endurance yet to sit in a DIMM slot unless it's gonna be just a storage drive in DIMM form factor. That and because we're only just now seeing early enterprise and retail products, I bet that we're gonna need another generation or two before we get DIMM versions. :(Billy Tallis - Thursday, November 9, 2017 - link
Intel hasn't said much about 3D XPoint DIMMs this year, other than to promise we'll hear more next year.It's not clear how much fault tolerance Intel is trying to build in to the Optane SSDs with the extra capacity. A bit of it is necessary to support sector formats with protection information (eg. 16 extra bytes per 512B sector, or 128B extra for each 4kB sector). Beyond that, there needs to be room for the drive's internal data structures, which aren't as complicated as a flash translation layer but still impose some space overhead. The rest is probably taken by a fairly simple erasure coding/ECC scheme, because it's almost impossible to do LDPC at the speed necessary for this drive. (That's also why DIMMs use simple ECC instead of more space-efficient codes.)
woggs - Thursday, November 9, 2017 - link
Most all intel SSDs have a parity die, so one full die likely provides internal raid protection of data. The rest is for ECC, internal system information, media management and mapping out defects... Impossible to know which of these is driving the actual spare implemented. I count 14 packages, so 1/14th (7%) is already the internal parity. 16% is big relative to nand consumer SSDs but comparable to enterprise. Doesn't seem particularly out of line or indicative of something wrong.CheapSushi - Thursday, November 9, 2017 - link
Micron is probably working on the DIMM version.woggs - Tuesday, November 14, 2017 - link
Intel is working on DIMMs... "Now, of course, SSDs are important, but in the long run, Intel also wants to have Optane 3D XPoint memory slot into the same sockets as DDR4 main memory, and Krzanich brought a mechanical model of an Optane DIMM to show off." https://www.nextplatform.com/2015/10/28/intel-show...MajGenRelativity - Thursday, November 9, 2017 - link
I enjoyed the review. Keep up the good work!melgross - Thursday, November 9, 2017 - link
I’m curious as to how this will perform when PCI 4 is out next year. That is, one with a PCI 4 interface. How throughput limiting is PCI 3 for this right now?MajGenRelativity - Thursday, November 9, 2017 - link
It shouldn't be that limiting, as PCIe 3.0 x4 allows for a higher throughput than 2.4 GB/s. There could be some latency improvements (probably small), but I don't think throughput is the issuewoggs - Thursday, November 9, 2017 - link
If the question is "could a drive be made to saturate gen 4?" then, yes, of course, if intel chooses to do so. Will require a whole drive. Latency is a more interesting question because that is what 3dxp is really providing. QD1 latency is <10us ((impressive!). I don't expect that to improve since it should be limited by the 3dxp itself. The PCIe and driver overhead is probably 5us of that. Maybe gen 4 will improve that part.woggs - Thursday, November 9, 2017 - link
This drive is PCIe 3, so nothing will change because it will link as gen 3. Will need a whole new drive...MajGenRelativity - Thursday, November 9, 2017 - link
I believe OP was talking about a drive that was identical to this in every way, but supporting PCIe 4woggs - Thursday, November 9, 2017 - link
And SSD is a system and would neither over nor under engineer any one piece significantly. So, altering one piece should not be expected to suddenly pop to some much higher performance. Will require a whole new drive.Hurr Durr - Thursday, November 9, 2017 - link
Where is requisite ddriver hysterics session about hypetane?Reflex - Thursday, November 9, 2017 - link
It appears the hysteric is gone at least for the time being. As a result the discussion has been far easier to read.Great drive, can't wait to see future generations where presumably they will increase density and reduce power consumption...
woggs - Thursday, November 9, 2017 - link
Yep!woggs - Thursday, November 9, 2017 - link
(on both counts)Hixbot - Thursday, November 9, 2017 - link
He must be taking the day off. I'm sure he'll flood the comments tomorrow.Makaveli - Thursday, November 9, 2017 - link
If you say his name 3 times he will show up....DON'T DO IT!
abhaxus - Thursday, November 9, 2017 - link
I came here looking for him, and was disappointed.Lord of the Bored - Thursday, November 9, 2017 - link
Me too. ddriver is most of why I read the comments.mkaibear - Friday, November 10, 2017 - link
He is always good for a giggle. I suppose he's busy directing hard drive manufacturers to make special hard drive platters for him solely out of hand-gathered sand from the Sahara. Or something.Still it's a shame to miss the laughs. It's always the second thing I do on SSD articles - first read the conclusion, then go and see what deedee has said. Ah well.
extide - Friday, November 10, 2017 - link
Please.. don't jinx us!rocky12345 - Thursday, November 9, 2017 - link
Interesting drive to say the least. Also a well written review thanks.PeachNCream - Thursday, November 9, 2017 - link
30 DWPD over the course of 5 years turns into a really large amount of data when you're talking about 750GB of capacity. Isn't the typical endurance rating more like 0.3 DPWD for enterprise solid state?So this thing about Optane on DIMMs is really interesting to me. Is the plan for it to replace RAM and storage all at once or to act as a cache or some sort between faster DRAM and conventional solid state? Even with the endurance its offering right now, it seems like it would need to be more durable still for it to replace RAM.
Oh (sorry case of shinies) could this be like a DIMM behind HBM on the CPU package where HBM does more of the write heavy stuff and then Optane lives between HBM and SSD or HDD storage? Has Intel let much out of the bag about this sorta thing?
Billy Tallis - Thursday, November 9, 2017 - link
Enterprise SSDs are usually sorted into two or three endurance tiers. Drives meant for mostly-read workloads typically have endurance ratings of 0.3 DWPD. High-endurance drives for write-intensive uses are usually 10, 25 or 30 DWPD, but the ratings of high-endurance drives have decayed somewhat in recent years as the market realized few applications really need that much endurance.lazarpandar - Thursday, November 9, 2017 - link
Can this be used to supplement addressable system memory? I remember Intel talking about that during the product launch.Billy Tallis - Thursday, November 9, 2017 - link
Yes. It makes for a great swap device, especially with a recent Linux kernel. Alternatively, Intel will sell it bundled with a hypervisor that presents the guest OS with a pool of memory equal in size to the system's DRAM plus about 85% of the Optane drive's capacity. The hypervisor manages memory placement, so from the guest OS's perspective the memory is a homogeneous pool, not x GB of DRAM and y GB of Optane.tuxRoller - Friday, November 10, 2017 - link
It's a bit odd Intel would go for the hypervisor solution since the kernel can handle tiered pmem and it's in a better position to know where to place data.I suppose it's useful because it's cross-platform?
xype - Friday, November 10, 2017 - link
I’d guess a hypervisor solution would also allow any critical fixes to be propagated faster/easier than having to go through a 3rd party (kernel) provider?tuxRoller - Friday, November 10, 2017 - link
Since this is for enterprise, the os vendor would be the one responsible (so, yes, third party) and one of the reasons why you pay them ridiculous support fees is for them to be your single point of contact for most issues.tuxRoller - Friday, November 10, 2017 - link
Very nice write-up.Might it be possible for us to get an idea of the difference in cell access times by running a couple tests on a loop device, and, even better, purely dram-based storage accessed over pcie?
Pork@III - Friday, November 10, 2017 - link
Has no normal only speed test? What are these creepy creations of this vc that?romrunning - Friday, November 10, 2017 - link
Is there any tests of the 4800X in a virtual host? Either Hyper-V or ESX, running multiple server OS clients with a variety of workloads. With the kind of low latency shown, I'd love to see how much more responsive Optane is compared to all flash storage like a P3608. Sort of a" rising tide floats all ships" kind of improvement, I hope.Klimax - Sunday, November 12, 2017 - link
That's nice review. How about some test using Windows too. (Aka something with more advanced I/O subsystem)Billy Tallis - Monday, November 13, 2017 - link
I'm not sure what you mean. Nobody seriously considers the Windows I/O system to be more advanced than what Linux provides. Even Intel's documentation states that the best latency they can get out of the Optane SSD on Windows is a few microseconds slower than on the Linux NVMe driver, and on Linux a few more microseconds can be saved using SPDK.tuxRoller - Tuesday, November 14, 2017 - link
"Advanced" may be the wrong way to look at it because ntkrnl can perform both sync and async operations, while Linux is essentially a sync-based kernel (the limitations surrounding its aio system are legendary). However, by focusing on doing that one thing well the block subsystem has become highly optimized for enterprise workloads.Btw, is there any chance you could run that block system (and nvme protocol, if possible) overhead test i asked about?