Now that you're explicitly telling the OEM's which tests you perform, it's easier for them to cheat on those specific benchmarks without a real world benefit. Do you have anything in mind to reduce this threat?
It weren't a secret what kind of benchmarks he was running before. Many of the things they need to implement or improve to boost benchmarks for his potential upgrade path would benefit real world use too.
Is there a package I could download to test my own ZFS server to see how it compares with the units tested on Anandtech? I'm quite curious. Also, how would I test my server that has 32 GB memory? It could easily cache the whole bench run.
NASPT - probably yes, but, then, the caching would really help home scenarios also.
SPEC SFS - unfortunately, caching can only be of limited effect. For example, each load point / client in the VDI workload does 24 GB of accesses. So, over 10 clients, that would be 240 GB simultaneously active.
Exactly - but you might be surprised how many support calls these NAS vendors field when, for example, a customer tries to run a virtual machine off a Marvell ARM-based NAS.
The reviews aim to educate users as to what is possible and what is not possible with these NAS units.
Also, our aim to use the test setup (both hardware and software) even for high-end SMB / SME NAS units. You will soon see a review of the QNAP TVS-871 which is able to do very well with the SPEC SFS 2014 benchmarks.
Realistically, how many people use an SMB-class NAS to host a database that has even a modicum of a minimal SLA? And VDI...thats a hoot. How many years do these people want to wait for the desktop to load?
With SSDs, it is not impossible for 2-bay or 4-bay units to be able to run databases and VDI workloads. The main problem here is the SATA 7200 RPM disks.
So, who are these filthy rich SMB's that are plunking down ridiculous money for flash in a NAS? No one. If they had the money to spend they wouldn't be buying this class of products, period. They would be buying actual equipment suited for this purpose. Testing a storage device for what it cannot do makes no sense. It also cannot win a Formula 1 race, but do you slap tires on it and try? No. It wouldn't make sense to do that, as it does not make sense to test quasi-SMB NAS with SSDs, or with enterprise workloads that no one in their right mind would run on them.
The TVS-871 can come with 2x 10G BASE-T ports (our review unit did). It costs around $3K. 8x SSDs (even Pro series) can be had for less than that. If your usage scenario demands it, you can have it - and for a fraction of the enterprise gear.
It all depends on what the usage scenario is, what is the budget and so on.
As I mentioned earlier, we only present what is possible to do with a particular NAS model, and we also make it a point to indicate where the bottleneck is. Again, as I said in another post - I hear lots of anecdotes from the support teams of various NAS vendors - particularly users who buy a 'Honda Civic' and expect 'Porsche' performance. You will be surprised what happens in the field. As always, we aim to educate.
At 3k, ZFS-based whiteboxes are competitive (both price and performance) as well, and will satisfy the need for "enterprise-class" reliability and rack mounting many need.
The real difference between the SMB-class NAS and enterprise-class solutions would be in the hardware reliability, but with 3k, you can build a pretty hefty dual-PSU, dual-Xeon system with a decent amount of ECC RAM and LSI HBAs to drop SSDs into.
We’ve updated our terms. By continuing to use the site and/or by logging into your account, you agree to the Site’s updated Terms of Use and Privacy Policy.
Facebook
Twitter
RSS
17 Comments
Back to Article
Shadow7037932 - Thursday, October 29, 2015 - link
That was an interesting read. I like that you made a point to test the system capabilities during various scenarios (streaming, multi user, etc).Gigaplex - Thursday, October 29, 2015 - link
Now that you're explicitly telling the OEM's which tests you perform, it's easier for them to cheat on those specific benchmarks without a real world benefit. Do you have anything in mind to reduce this threat?Penti - Thursday, October 29, 2015 - link
It weren't a secret what kind of benchmarks he was running before. Many of the things they need to implement or improve to boost benchmarks for his potential upgrade path would benefit real world use too.Shadow7037932 - Friday, October 30, 2015 - link
In the past, AnandTech has called out smartphone OEMs for cheating in benchmarks. I'm sure they'll look in to it if.buxe2quec - Friday, October 30, 2015 - link
Is there a package I could download to test my own ZFS server to see how it compares with the units tested on Anandtech? I'm quite curious. Also, how would I test my server that has 32 GB memory? It could easily cache the whole bench run.ganeshts - Friday, October 30, 2015 - link
NASPT - probably yes, but, then, the caching would really help home scenarios also.SPEC SFS - unfortunately, caching can only be of limited effect. For example, each load point / client in the VDI workload does 24 GB of accesses. So, over 10 clients, that would be 240 GB simultaneously active.
buxe2quec - Sunday, November 1, 2015 - link
Tried NASPT, got reasonable results and only HD playback gives unrealistic data (148 MB/s, due to caching on Windows 10 side). Thanksmervincm - Friday, October 30, 2015 - link
"Many NAS vendors are supporting SMB multi-channel now"Really? any of the SMB vendors like Synology / QNap ? Netgear etc.?
ganeshts - Friday, October 30, 2015 - link
Yes, Western Digital, QNAP and Netgear to my knowledge. Others might be doing it too, but these are the ones for sure.Gigaplex - Saturday, October 31, 2015 - link
How are they doing it? Are they making use of Samba without releasing the patches? Or using a fully custom stack?JellyRoll - Monday, November 2, 2015 - link
Has the concept that the majority of the NAS units tested fail the SPEC SFS 2014 benchmark because they simply aren't intended for that use case?ganeshts - Monday, November 2, 2015 - link
Exactly - but you might be surprised how many support calls these NAS vendors field when, for example, a customer tries to run a virtual machine off a Marvell ARM-based NAS.The reviews aim to educate users as to what is possible and what is not possible with these NAS units.
Also, our aim to use the test setup (both hardware and software) even for high-end SMB / SME NAS units. You will soon see a review of the QNAP TVS-871 which is able to do very well with the SPEC SFS 2014 benchmarks.
JellyRoll - Monday, November 2, 2015 - link
Realistically, how many people use an SMB-class NAS to host a database that has even a modicum of a minimal SLA? And VDI...thats a hoot. How many years do these people want to wait for the desktop to load?ganeshts - Monday, November 2, 2015 - link
With SSDs, it is not impossible for 2-bay or 4-bay units to be able to run databases and VDI workloads. The main problem here is the SATA 7200 RPM disks.JellyRoll - Monday, November 2, 2015 - link
So, who are these filthy rich SMB's that are plunking down ridiculous money for flash in a NAS? No one. If they had the money to spend they wouldn't be buying this class of products, period. They would be buying actual equipment suited for this purpose. Testing a storage device for what it cannot do makes no sense. It also cannot win a Formula 1 race, but do you slap tires on it and try? No. It wouldn't make sense to do that, as it does not make sense to test quasi-SMB NAS with SSDs, or with enterprise workloads that no one in their right mind would run on them.ganeshts - Monday, November 2, 2015 - link
The TVS-871 can come with 2x 10G BASE-T ports (our review unit did). It costs around $3K. 8x SSDs (even Pro series) can be had for less than that. If your usage scenario demands it, you can have it - and for a fraction of the enterprise gear.It all depends on what the usage scenario is, what is the budget and so on.
As I mentioned earlier, we only present what is possible to do with a particular NAS model, and we also make it a point to indicate where the bottleneck is. Again, as I said in another post - I hear lots of anecdotes from the support teams of various NAS vendors - particularly users who buy a 'Honda Civic' and expect 'Porsche' performance. You will be surprised what happens in the field. As always, we aim to educate.
ZeDestructor - Tuesday, November 3, 2015 - link
At 3k, ZFS-based whiteboxes are competitive (both price and performance) as well, and will satisfy the need for "enterprise-class" reliability and rack mounting many need.The real difference between the SMB-class NAS and enterprise-class solutions would be in the hardware reliability, but with 3k, you can build a pretty hefty dual-PSU, dual-Xeon system with a decent amount of ECC RAM and LSI HBAs to drop SSDs into.