Intent of the article is to show what Servermark VDI is, and how it can be used. Benchmark numbers are secondary.
Also, benchmark numbers can be compared within the same 'set' to show how increasing the load on the server causes a drop in the effectiveness of the server.
Lastly, the benchmark numbers are just PCMark 8 Work preset scores. Plenty of PCs have been evaluated with that benchmark. Here is a graph from our latest mini-PC review with PCMark 8 work scores : http://images.anandtech.com/graphs/graph10595/pcm8... - so that you can get an idea of the performance of the Xeon-D 1540 -equipped server against some of the more powerful mini-PCs.
While I did not read the full article and I agree this is more about what the software can do. I believe it would help to see the benchmark numbers from the PCMark 8 Work preset to show how much of a difference there is from a single session to 2+ VMs.
It does seem like this server is a bit under powered for VM work. Any word on when there might be a follow up? It would be a lot more interesting to see some numbers that show how much of a change there is as you add more VMs.
Oh, right.. that is why we mention competitors like SPEC and LoginVSI? Never knew advertising meant talking about competitors in good light!!
Futuremark is well respected in benchmarking circles, and we were given the opportunity to beta-test their upcoming product. Anyone interested in performance evaluation of computing systems would jump in at this opportunity. We did, and we thought it would be interesting to share our impressions with readers - they can give their own feedback to Futuremark.
No solicitation is made in this article to get readers to go out and 'purchase' Servermark - In fact, it is not even available for purchase yet.
Don't impugn the editorial integrity of the writers here who spend countless hours in attempting to present a fair and balanced view of hardware and software of interest to the computing community. It is insulting to read comments like this.
FM is thoroughly decent company which I have no issue with. But I do have issues with articles about software which have no base for comparisons. You might mean well and intend to present in a fair and balanced method, but it sure doesn't come across like it.
Then again, if you don't care for the opinions of your readers, then I can clearly see why these kind of articles are becoming more common.
What is your rationale for saying : "it sure doesn't come across like it" ?
You have made an insinuation without properly backing up your comment.
We value opinions of all readers, and we will hear them out if they are presented in the right manner with proper reasoning - definitely can't respond with 'care' if they are off-the-cuff remarks that are baseless and demeaning to the efforts put by the editor.
My rationale is that this type of software ideally needs to be OS agnostic, and not limited to a Windows platform. There is no critique in the article to mention that, and it comes off as a one dimensional viewpoint, which is quite far away from typical server usage, especially in VM workloads.
That would not be a valid criticism of Futuremark's efforts. The aim of Servermark *VDI* is to test how many virtual desktops a server can support. Almost all office environments that use *VDI* do so with Windows as the guest environment. I do agree that when it comes to *VMs*, a lot of them are Linux-based and the like - but it is very rare to see Linux desktops as part of VDI in office environments.
Futuremark had a particular scenario in mind while framing Servermark VDI, and I have to say that they are doing the right thing for that scenario.
What would be valid is that they did not supply benchmarking environment scripts for non-VMWare environments, and we had to develop our own. Futuremark has taken this feedback and promised to provide support for Hyper-V also in the final release. This, I have pointed out as a drawback that Futuremark will address in the future.
We can only point out valid issues in the review, and we have done that on the basis of our hands-on evaluation of the benchmark.
Although FM puts "2000" as their recommended benchmark level, I think you will find for VDI that the bar is much, much lower. The main gist of VDI is the fact that most users need very little computational power continuously. They need the boot and load software bumps and then most productivity software sips at a few percent of CPU. I've noticed web browsers, especially Chrome, have become the biggest consumers of CPU on workstations, and it seems almost vindictive the way certain browsers will hose a system when resource starved. Mozilla/Firefox is especially designed to make virtualization and RDC scenarios bad (the thousands of cache folders is intentionally shit design for roaming profiles and causes huge login hangs).
I noticed a NUC gets around 5000 in the benchmark. From my experience in VDI as well as XenApp and RDS architectures, a NUC could support around 10 users with sufficient RAM (16GB would do it - 32GB and we're talking 20+ users, maybe 40 in XenApp/RDS instead of VDI).
Also, for the love of all that is holy, do not benchmark against RAID 0, it invalidates much of the testing. Test against both RAID 10 and 5 as those are industry standards for such infrastructures.
If you really want to deep dive, check out differences in memory consumption in HyperV vs VMware with VDI. While both support memory deduplication, experience shows that HyperV is much faster and more effective at VDI memory dedupe, while VMware seems to rely a tad heavy on memory ballooning. In server farm / VDI situations, RAM and storage I/O are ALWAYS more limiting than CPU throughput. You're not putting Pro-E and Solidworks into a VDI farm most of the time, just MS Office, a line of business application, and a browser.
maybe i am approaching this too focused on a consumer's mind, but all of that hardware needs to be shipped, unpacked, installed, and plugged in. For me, I prefer to log-in to a cloud PC for gaming and virtual PC. There are several services similar to what I'm talking about, www.paperspace.com being the one I happened to pick.
I'm sure FM goes for more enterprise-level agreements, but what about the small teams and individuals?
It seems to me this is completely a WINTEL benchmark. That's fine for a certain class of users, I guess, but is, I think, a lot less interesting for many readers. It does nothing to clarify whether either - a Linux PC would be a better server solution OR - whether an ARM server could handle a particular load.
Obviously AnandTech can do what they want in their reviews, but I, for one, am a lot less interested in comparisons that are focussed on the few percent differences between Wintel box A and Wintel box B than in comparisons between very different systems.
Well, they were looking at VDI. I don't know what kind of monster would try VDI+Linux+end user workloads in the workplace. But yeah, keep saying dumbass shit about Linux and ARM.
Lots of hostility unfairly being directed at the authors/editors here. I thought using the benchmark preview as a way to couch a hardware review was genuinely interesting both as a nerd with a home lab, and a professional wanting to keep abreast of how the enterprise class gear is being evaluated. Keep up the good work, haters gonna hate!
We’ve updated our terms. By continuing to use the site and/or by logging into your account, you agree to the Site’s updated Terms of Use and Privacy Policy.
Facebook
Twitter
RSS
18 Comments
Back to Article
ddriver - Thursday, September 1, 2016 - link
Absent a base for comparison, those marks are a meaningless metric.ganeshts - Thursday, September 1, 2016 - link
Intent of the article is to show what Servermark VDI is, and how it can be used. Benchmark numbers are secondary.Also, benchmark numbers can be compared within the same 'set' to show how increasing the load on the server causes a drop in the effectiveness of the server.
Lastly, the benchmark numbers are just PCMark 8 Work preset scores. Plenty of PCs have been evaluated with that benchmark. Here is a graph from our latest mini-PC review with PCMark 8 work scores : http://images.anandtech.com/graphs/graph10595/pcm8... - so that you can get an idea of the performance of the Xeon-D 1540 -equipped server against some of the more powerful mini-PCs.
lioncat55 - Thursday, September 1, 2016 - link
While I did not read the full article and I agree this is more about what the software can do. I believe it would help to see the benchmark numbers from the PCMark 8 Work preset to show how much of a difference there is from a single session to 2+ VMs.It does seem like this server is a bit under powered for VM work. Any word on when there might be a follow up? It would be a lot more interesting to see some numbers that show how much of a change there is as you add more VMs.
extide - Wednesday, September 7, 2016 - link
You probably should have read the article then, because it DOES show the scores as they change vs the number of VM's...powerarmour - Thursday, September 1, 2016 - link
What is the point of this article exactly?, Advertising?ganeshts - Thursday, September 1, 2016 - link
Oh, right.. that is why we mention competitors like SPEC and LoginVSI? Never knew advertising meant talking about competitors in good light!!Futuremark is well respected in benchmarking circles, and we were given the opportunity to beta-test their upcoming product. Anyone interested in performance evaluation of computing systems would jump in at this opportunity. We did, and we thought it would be interesting to share our impressions with readers - they can give their own feedback to Futuremark.
No solicitation is made in this article to get readers to go out and 'purchase' Servermark - In fact, it is not even available for purchase yet.
Don't impugn the editorial integrity of the writers here who spend countless hours in attempting to present a fair and balanced view of hardware and software of interest to the computing community. It is insulting to read comments like this.
powerarmour - Thursday, September 1, 2016 - link
FM is thoroughly decent company which I have no issue with. But I do have issues with articles about software which have no base for comparisons. You might mean well and intend to present in a fair and balanced method, but it sure doesn't come across like it.Then again, if you don't care for the opinions of your readers, then I can clearly see why these kind of articles are becoming more common.
ganeshts - Thursday, September 1, 2016 - link
What is your rationale for saying : "it sure doesn't come across like it" ?You have made an insinuation without properly backing up your comment.
We value opinions of all readers, and we will hear them out if they are presented in the right manner with proper reasoning - definitely can't respond with 'care' if they are off-the-cuff remarks that are baseless and demeaning to the efforts put by the editor.
powerarmour - Thursday, September 1, 2016 - link
My rationale is that this type of software ideally needs to be OS agnostic, and not limited to a Windows platform. There is no critique in the article to mention that, and it comes off as a one dimensional viewpoint, which is quite far away from typical server usage, especially in VM workloads.ganeshts - Thursday, September 1, 2016 - link
That would not be a valid criticism of Futuremark's efforts. The aim of Servermark *VDI* is to test how many virtual desktops a server can support. Almost all office environments that use *VDI* do so with Windows as the guest environment. I do agree that when it comes to *VMs*, a lot of them are Linux-based and the like - but it is very rare to see Linux desktops as part of VDI in office environments.Futuremark had a particular scenario in mind while framing Servermark VDI, and I have to say that they are doing the right thing for that scenario.
What would be valid is that they did not supply benchmarking environment scripts for non-VMWare environments, and we had to develop our own. Futuremark has taken this feedback and promised to provide support for Hyper-V also in the final release. This, I have pointed out as a drawback that Futuremark will address in the future.
We can only point out valid issues in the review, and we have done that on the basis of our hands-on evaluation of the benchmark.
Ryan Smith - Thursday, September 1, 2016 - link
Ultimately this is an article we worked on because Futuremark's software looked neat, and we wanted to see if it could tell us anything useful.We're nerds at heart. We like looking at new things.=)
aaronb1138 - Friday, September 2, 2016 - link
Although FM puts "2000" as their recommended benchmark level, I think you will find for VDI that the bar is much, much lower. The main gist of VDI is the fact that most users need very little computational power continuously. They need the boot and load software bumps and then most productivity software sips at a few percent of CPU. I've noticed web browsers, especially Chrome, have become the biggest consumers of CPU on workstations, and it seems almost vindictive the way certain browsers will hose a system when resource starved. Mozilla/Firefox is especially designed to make virtualization and RDC scenarios bad (the thousands of cache folders is intentionally shit design for roaming profiles and causes huge login hangs).I noticed a NUC gets around 5000 in the benchmark. From my experience in VDI as well as XenApp and RDS architectures, a NUC could support around 10 users with sufficient RAM (16GB would do it - 32GB and we're talking 20+ users, maybe 40 in XenApp/RDS instead of VDI).
Also, for the love of all that is holy, do not benchmark against RAID 0, it invalidates much of the testing. Test against both RAID 10 and 5 as those are industry standards for such infrastructures.
If you really want to deep dive, check out differences in memory consumption in HyperV vs VMware with VDI. While both support memory deduplication, experience shows that HyperV is much faster and more effective at VDI memory dedupe, while VMware seems to rely a tad heavy on memory ballooning. In server farm / VDI situations, RAM and storage I/O are ALWAYS more limiting than CPU throughput. You're not putting Pro-E and Solidworks into a VDI farm most of the time, just MS Office, a line of business application, and a browser.
madisson - Thursday, September 1, 2016 - link
maybe i am approaching this too focused on a consumer's mind, but all of that hardware needs to be shipped, unpacked, installed, and plugged in. For me, I prefer to log-in to a cloud PC for gaming and virtual PC. There are several services similar to what I'm talking about, www.paperspace.com being the one I happened to pick.I'm sure FM goes for more enterprise-level agreements, but what about the small teams and individuals?
name99 - Thursday, September 1, 2016 - link
It seems to me this is completely a WINTEL benchmark.That's fine for a certain class of users, I guess, but is, I think, a lot less interesting for many readers. It does nothing to clarify whether either
- a Linux PC would be a better server solution OR
- whether an ARM server could handle a particular load.
Obviously AnandTech can do what they want in their reviews, but I, for one, am a lot less interested in comparisons that are focussed on the few percent differences between Wintel box A and Wintel box B than in comparisons between very different systems.
powerarmour - Thursday, September 1, 2016 - link
Exactly.aaronb1138 - Friday, September 2, 2016 - link
Well, they were looking at VDI. I don't know what kind of monster would try VDI+Linux+end user workloads in the workplace. But yeah, keep saying dumbass shit about Linux and ARM.gabemcg - Tuesday, September 6, 2016 - link
Lots of hostility unfairly being directed at the authors/editors here. I thought using the benchmark preview as a way to couch a hardware review was genuinely interesting both as a nerd with a home lab, and a professional wanting to keep abreast of how the enterprise class gear is being evaluated. Keep up the good work, haters gonna hate!simran sidhki - Thursday, October 18, 2018 - link
hello