Standardized benchmarks with industry-wide acceptance are a must for evaluating computing systems. These benchmarks may evaluate the system as a whole, or target specific aspects. Vendors such as Futuremark and BAPCo have various offerings for both consumer and business-use PCs. However, the market for such programs in the server space is largely untapped. Futuremark has been working on Servermark for more than two years now, and they finally felt it was stable enough to let us test drive one of their recent beta versions. Servermark will have a number of sub-categories, but, our focus was on VDI (virtual desktop infrastructure) - a way to determine how many virtual machines a server can support while providing an acceptable level of performance for all users.

Introduction and System Setup

Modern servers are very powerful and come with virtualization capabilities. This capability allows the work done by multiple physical machines to be be consolidated into a single physical server using the concept of 'virtual machines' (VMs). Virtualization brings about a host of benefits to system administrators. VMs can be used in a headless manner (for example, as a mail server or IoT controller), or as a virtual desktop with a thin client front-end. The latter can enable businesses to provide basic low-cost / low-power computers at the employee's desk. Business applications which require plenty of processing power can be processed on a virtual desktop running on the VM host / server. This aspect is the focus of Servermark VDI and our investigation today.

Prior to going into the details of Servermark VDI and its various features, let us take a look at our test machine for the evaluation. The Supermicro SYS-5028D-TN4T is a very popular server in the mini-tower form factor amongst home-lab enthusiasts. We already covered it in our Xeon-D review. Our unit (from last year) came with a Xeon-D 1540 (8C / 16T @ 2.00 GHz) SoC and four hot-swappable 3.5" drive bays. The following table summarizes the specifications of the Supermicro SYS-5028D-TN4T used in this review.

We used four Crucial MX200 500GB SSDs in the four hot-swap bays and configured them in RAID-0 with Intel's BIOS-assisted software RAID.

Futuremark's Servermark VDI came by default with scripts for a VMWare-based virtualization setup. However, being personally used to Hyper-V, I decided to perform the evaluation using Windows Server 2016 TP5.
The RAID-0 volume with the Crucial MX200 SSDs was used as the VM data store. A switch was setup for the VMs and they were uplinked to the main network using one of the 10GBASE-T LAN ports. For home lab setups, this is acceptable, though we must make a note here that the Xeon-D 1540 doesn't support SR-IOV and ends up increasing CPU load for this type of internal switch setup. The newer version of the SYS-5028D-TN4T comes with the Xeon-D 1541. It has better I/O virtualization support, and doesn't have this problem.

In addition to the Hyper-V role, Windows Server 2016 TP5 was also configured with a DHCP Server role on an internal network accessible to both the host OS and the guest OS. Windows 10 VMs were used as guests. Each VM was configured with two network adapters - the first one connected to one of the 10GBASE-T LAN ports of the host and the second one connected to the internal network. Both interfaces were configured as DHCP clients.

In the next section, we will talk about Futuremark Servermark VDI in detail, followed by details of how we benchmarked the Supermicro SYS-5028D-TN4T. At this juncture, we must stress that the Xeon-D machine is not really geared for large-scale virtualization workloads. Our purpose here is to take Futuremark's Servermark VDI for a test drive and create a comprehensive evaluation script that can be utilized for evaluating 'production' machines.

Futuremark Servermark VDI

The previous section briefly mentioned that VDI is quite popular with business / enterprise use-cases, particularly for office applications. Futuremark has kept this in mind while formulating the benchmarking strategy for virtualization capabilities. On the desktop side, Futuremark already has the well-respected PCMark 8 system benchmark. It considers various use-cases and can report scores for 'Work', 'Creative' and 'Home' workloads. Servermark VDI is based on PCMark.

Multiple PCMark instances are executed on the VDI server simultaneously (one instance per VM) in order to determine the server performance. Based on the scores from each instance, it is possible to determine the maximum number of virtual desktops that the server can support with acceptable performance. Correspondingly, it can also report the performance of the server with a particular number of clients.

The various PCMark instances are controlled by a Servermark Control Console that runs on a separate machine that is on the same network as that of the virtual machines. It is possible, but, not advisable, to run the Control Console on one of the VMs under test. The Control Console also takes the server configuration as input (through a JSON file) and uses it for generating the final report.

Servermark Control Console takes a number of parameters. The most important amongst those include the IP addresses / ports of the VMs (in either IPv4 or IPv6 format), the number of times that PCMark needs to be run on each VM, the duty cycle (fraction of time, as percentage, that should run the benchmark simultaneously on the VMs), the stagger time for the start of PCMark on each of the VMs and a timeout for processing each PCMark run.

By default, the Control Console runs PCMark three times on each VM. The duty cycle is set at 30%, stagger time at 10s and processing timeout of 60 minutes. The PCMark workload utilized is the 'Work' preset - this goes well with primary use-case for VDIs being in the office / business space. For VMs with OpenCL capabilities, the PCMark 8 benchmark can be run with accelerated compute turned on.

Servermark VDI suggests that each VM score at least 2000 on the 'Work' preset. If the minimum score is not achieved, the PCMark 8 scores of each VDI client are reported along with various statistical parameters such as the minimum, median, maximum, average and standard deviation. If the minimum acceptable score is met, the benchmark also creates a PDF file with all the hardware and score details. This serves as a certification from Futuremark that the server configuration is able to run the specified number of concurrent VDI clients with an acceptable level of performance.

Benchmarking the Supermicro SYS-5028D-TN4T

The guides and scripts that are supplied with Servermark VDI assume a VMWare-based virtualization setup. However, for our Hyper-V setup, the scripts had to be reworked in Powershell. Two distinct scripts were developed. The first script took a base VM and cloned it N times with different copies of the base image (not differencing VHDs). This script was run on the Supermiro SYS-5028D-TN4T.

An Intel D54250WYK Haswell NUC running Windows 8.1 Pro x64 was used as the console node on the main network. The second script was run on this machine. It started the VMs in sets of 2 (start with 2, then 4, 6 and so on till a configurable number of VMs based on N in the first script) and processed the ServermarkControlConsole command with the appropriate arguments based on the number of active powered on VMs in that particular iteration. One of the challenges here was determining the DHCP IPs of the active VMs on their main network interface to prepare the arguments.

The Xeon-D 1540 is a 8C/16T processor, and our system was configured with 64GB of RAM. To keep things reasonable, we set the number of maximum simultaneously active VMs to be 16. Each of the Hyper-V Gen 2 VMs was allocated one vCPU and 2GB of RAM. Other VM configurations were left at default. The graph below plots the average PCMark score obtained on each VM as a function of the number of simultaneously active VMs.

Other statistics from the benchmark runs are summarized in the table below.

A number of tests were carried out by allocating more vCPUs per VM and also allowing dynamic memory amounts for each according to their requirements. Irrespective of the VM configuration, we were unable to score more than the 2000 required for Servermark VDI to certify the server. This is likely due to the single-threaded performance of the Xeon-D 1540 not being good enough for the generic office workloads from PCMark 8's 'Work' preset.

Concluding Remarks

Coming to the business end of the article, there are two separate aspects to discuss - the hardware that we used in this evaluation, and the Servermark VDI benchmark itself.

Supermicro's SYS-5028D-TN4T is the perfect machine for IT enthusiasts and tinkerers to experiment with various virtualization tools. In addition to being compact and relatively silent, it is also frugal in terms of power consumption. Our review configuration idled around 35W, and even subjecting the unit to AIDA64's system stability test with all aspects loaded pushed the unit up to 105W only. The availability of dual native 10GBASE-T ports more than justify its current street price. It does have some drawbacks in terms of chassis design (placement of USB ports and the design of the hot-swap bays) and the access to internal components is not hassle-free. However, these are minor aspects in the grand scheme of things. Readers looking to invest in a virtualization lab machine would do little wrong in opting for the Supermicro SYS-5028D-TN4T.

Futuremark's Servermark has been in development for quite some time. Their PCMark offering is undoubtedly very well respected in the desktop benchmarking circles. It is therefore logical that they are trying to use that to move into the server benchmarking space. There are plenty of configurable options to make the usage of multiple PCMark instances a valid virtualization benchmark. In particular, the duty cycle parameter enables IT administrators / benchmark users to have a better fit between the evaluation and the expected workloads on the server.

Futuremark's aim with Servermark VDI appears to be two-fold. On one hand, they want to provide IT administrators with a tool to evaluate how many virtual desktops they can allocate on a particular server. On the other hand, they also want to supply server vendors with a marketing tool. Armed with a Servermark VDI certificate, the vendors can make a better pitch to business users. The certificate can also help the latter to consider the right server offering for their needs.

Futuremark's Servermark VDI faces stiff competition from existing well-established benchmarks such as SPEC's SPECvirt_sc2013, LoginVSI's offerings and even VMWare's own VMMark. Perfecting a server / VDI benchmark is a tough task, and this field is obviously not as crowded as the desktop benchmarking arena. Therefore, Servermark VDI is definitely a welcome addition. In our opinion, the results delivered by it are based on solid benchmarking and can help users arrived at independent and reliable conclusions.

Futuremark indicated that the future plans for Servermark VDI include expanded support for more virtualization environments beyond VMWare, including Hyper-V. In addition, once PCMark 10 gets released, Servermark will support it as well.

VDI is only one server application. We believe Futuremark is evaluating other server application scenarios also. Specific server benchmarks focusing on particular server aspects like, say, the data store / disk subsystem would also be welcome. In its current beta state, Servermark VDI seems to have had a promising start in the evaluation of server systems. We expect / hope that more stakeholders are providing feedback to Futuremark in order to add more features to Servermark and fine-tune its current features.