One of the consequences of the Meltdown and Spectre vulnerabilities, besides the immediate security concerns, is what it has meant for our ongoing testing and benchmarking operations. Along with accounting for the immediate performance changes from the required patches, we've also needed to look at how we present data and how we can (or can't) compare new patched systems to older unpatched systems. From an editorial perspective we need to ensure the continuity of our data as well as providing meaningful benchmark collections.

What we've found so far is that the impact of the Meltdown and Spectre patches varies with the workload and the type of test. Nate's GPU testing shrugs it off almost entirely, while Billy's SSD testing especially feels the pinch. In the middle of course are consumer system reviews, where workloads are more varied but also much more realistic and often more relevant as well. Of course, this is also the type hardware that we most often have to return when we're done, making it the most challenging to correct.

As everyone at AnandTech has been integrating the patches and updating their datasets in their own way, I wanted to offer some insight into what's going on for system testing. With several important systems set to launch in the first half of this year – particularly Intel's Hades Canyon – I've been going back and collecting updated results for use in those future reviews. In the process, I also wanted to document how performance has been impacted by these security patches and which benchmarks in particular have been affected.

Evaluation Setup

The Intel Kaby Lake NUC (NUC7i7BNH) was used for all the benchmarks presented in this article. It was chosen for two reasons. Being a relatively new system that consumers can still purchase in the market, evaluating it presents data of particular relevance to readers. Intel was prompt in providing BIOS updates to resolve the Meltdown and Spectre issues compared to other OEMs whose systems I had in hand for evaluation. The NUC was configured with the following specifications:

Three different patch configurations were benchmarked:

• Completely unpatched
• Only OS patches activated (Graph entry : Patch OS)
• Completely patched with updates to both the OS and the CPU microcode (Graph entry : Patch OS + uCode)

While it will (eventually) be an uncommon configuration, right now a lot of systems are still only running OS patches without their associated microcode updates, as the latter is still being distributed. At the same time, because the microcode patches are primarily for Spectre – Meltdown fixes are implemented at an OS level – this allows us to see just how much of an impact the Spectre fixes have on top of the existing Meltdown fixes.

The completely unpatched system used Windows 10 version 16299.125 with BIOS v0054. As evident from the screenshot below, the system is susceptible to both Spectre and Meltdown.

The second configuration was processed with Windows 10 version 16299.309 with BIOS v0060. The screenshot below shows that the system is protected against Meltdown, but, not Spectre.

The final fully patched configuration was process with Windows 10 version 16299.214 and BIOS v0061. The current BIOS is v0062 (the one we used was available briefly before it was pulled to undergo additional QA for possible random restarts). This configuration shows full protection against both Meltdown and Spectre.

We used our standard test suite for low power desktops / industrial PCs in order to evaluate the impact of the patching on the performance of the system.

BAPCo and Futuremark Benchmarks

BAPCo's SYSmark 2014 SE is an application-based benchmark that uses real-world applications to replay usage patterns of business users in the areas of office productivity, media creation and data/financial analysis. In addition, it also addresses the responsiveness aspect which deals with user experience as related to application and file launches, multi-tasking etc. Scores are meant to be compared against a reference desktop (the SYSmark 2014 SE calibration system in the graphs below). While the SYSmark 2014 benchmark used a Haswell-based desktop configuration, the SYSmark 2014 SE makes the move to a Lenovo ThinkCenter M800 (Intel Core i3-6100, 4GB RAM and a 256GB SATA SSD). The calibration system scores 1000 in each of the scenarios. A score of, say, 2000, would imply that the system under test is twice as fast as the reference system.

We see that the patching does have an impact on the performance. It ranges from around 5% for the Office Productivity scenario to around 10% for the Responsiveness metric.

SYSmark 2014 SE also adds energy measurement to the mix. We see that the patched system, despite being slower, consumes lower energy for the completion of the tasks.

Futuremark PCMark 10

UL's PCMark 10 evaluates computing systems for various usage scenarios (generic / essential tasks such as web browsing and starting up applications, productivity tasks such as editing spreadsheets and documents, gaming, and digital content creation). We benchmarked the configurations with the PCMark 10 Extended profile and recorded the scores for various scenarios. These scores are heavily influenced by the CPU and GPU in the system, though the RAM and storage device also play a part. The power plan was set to Balanced for all the PCs while processing the PCMark 10 benchmark.

Similar to SYSmark 2014 SE, we see performance impacts in all the PCMark 10 scenarios also. For example, the Essentials workload score for the fully patched system is around 6.5% lower than the unpatched system. The gaming workload doesn't seem to be affected much by the patching, though. The overall score is around 2.3% lower.

Futuremark PCMark 8

We continue to present PCMark 8 benchmark results (as those have more comparison points) while our PCMark 10 scores database for systems grows in size. PCMark 8 provides various usage scenarios (home, creative and work) and offers ways to benchmark both baseline (CPU-only) as well as OpenCL accelerated (CPU + GPU) performance. We benchmarked select PCs for the OpenCL accelerated performance in all three usage scenarios. Since these scores are heavily influenced by the CPU in the system, it is not surprising to see a performance impact. The patched configuration, on average, scores around 2% lower than the unpatched one.

Miscellaneous Futuremark Benchmarks

PCMark 7's PCMark Suite shows a 4% performance loss.

3DMark 11's entry level score and 3DMark 2013's Cloud Gate score actually end up better after the patch, but, 3DMark 2013's Ice Storm score shows a performance loss of more than 13%.

Futuremark PCMark 8 Storage Bench

PCMark 8 has a storage bench where certain common workloads such as loading games and document processing are replayed on the target drive. Results are presented in two forms, one being a benchmark number and the other, a bandwidth figure. We ran the PCMark 8 storage bench on the ADATA NVMe SSD in all the configurations.

While the absolute score is just around 1.3% lower for the patched configuration, the storage bandwidth is around 29% worse. This significant performance loss is partly due to the NVMe drive performance now being CPU bound. It is highly likely that SATA drives will not experience such a drastic performance loss due to the patching.

Miscellaneous Benchmarks

CINEBENCH R15 is our benchmark of choice for 3D rendering. It provides three benchmark modes - OpenGL, single threaded and multi-threaded. This benchmark is largely unaffected by the patching. All the recorded numbers are within the margin of expected errors from one run to another.

x264 v5.0 is another benchmark that is unaffected by the Meltdown and Spectre patches.

7-Zip is a very effective and efficient compression program, often beating out OpenCL accelerated commercial programs in benchmarks even while using just the CPU power. 7-Zip has a benchmarking program that provides tons of details regarding the underlying CPU's efficiency. In our benchmark suite, we are interested in the compression and decompression MIPS ratings when utilizing all the available threads. This benchmark also remains unaffected, with the results happening to be within the margin of error from run to run.

Agisoft PhotoScan is a commercial program that converts 2D images into 3D point maps, meshes and textures. The program designers sent us a command line version in order to evaluate the efficiency of various systems that go under our review scanner. The command line version has two benchmark modes, one using the CPU and the other using both the CPU and GPU (via OpenCL). We have been using an old version of the program with 50 photographs in our reviews till now. The updated benchmark (v1.3) now takes around 84 photographs and does four stages of computation:

• Stage 1: Align Photographs (capable of OpenCL acceleration)
• Stage 2: Build Point Cloud (capable of OpenCL acceleration)
• Stage 3: Build Mesh
• Stage 4: Build Textures

We record the time taken for each stage. Since various elements of the software are single threaded, others multithreaded, and some use GPUs, it is a very relevant benchmark from a media editing and content creation perspective.

Since this is a real-world benchmark, we can see performance impacts in some of the stages. While the first and last ones do not have any significant deviation, stages 2 and 3 are worse off by around 12.8% and 6.8% respectively in the non-GPU case.

The benchmarks section wraps up with the new Dolphin Emulator (v5) benchmark mode results. This is again a test of the CPU capabilities, but, we don't see much impact on the performance from the patching. The bennchmark consistently took around 325 seconds in all three patching configurations.

Concluding Remarks

While the primary purpose of this exercise was just to update our datasets for future system reviews, it none the less proved to be an enlightening one, and something worth sharing. We already had an idea of what to expect going into refreshing our benchmark data for Meltdown and Spectre, and in some ways we still managed to find a surprise or two while looking at Intel's NUC7i7BNH NUC. The table below summarizes the extent of performance loss in various benchmarks.

Looking at the NUC – and really this should be on the mark for most SSD-equipped Haswell+ systems – there isn't a significant universal trend. The standard for system tests such as these is +/- 3% performance variability, which covers a good chunk of the sub-benchmarks. What's left then are more meaningful performance impacts in select workloads of the BAPCo SYSmark 2014 SE and Futuremark PCMark 10 benchmarks, particularly storage-centric benchmarks. Other than those, we see certain compute workloads (such as the 2nd stage of the Agisoft Photoscan benchmark) experience a loss in performance of more than 10%.

On the whole, we see that the patches for Meltdown and Spectre affect real-world application benchmarks, but, synthetic ones are largely unaffected. The common factor among most of these benchmarks in turn is storage and I/O; the greater the number of operations, the more likely a program will feel the impact of the patches. Conversely, a compute-intensive workload that does little in the way of I/O is more or less unfazed by the changes. Though there is a certain irony to the fact that taken to its logical conclusion, patching a CPU instead renders storage performance slower, with the most impacted systems having the fastest storage.

As for what this means for future system reviews, the studies done as part of this article give us a way forward without completely invalidating all the benchmarks that we have processed in the last few years. While we can't reevaluate every last system – and so old data will need to stick around for a while longer still – these results mean that the data from unimpacted benchmarks is still valid and relevant even after the release of the Meltdown and Spectre patches. To be sure, we will be marking these results with an asterisk to denote this, but ultimately this will allow us to continue comparing new systems to older systems in at least a subset of our traditional benchmarks. Which combined with back-filling benchmarks for those older systems that we do have, lets us retain a good degree of review and benchmark continuity going forward.