Last week, we published our AMD 2nd Gen Ryzen Deep Dive, covering our testing and analysis of the latest generation of processors to come out from AMD. Highlights of the new products included better cache latencies, faster memory support, an increase in IPC, an overall performance gain over the first generation products, new power management methods for turbo frequencies, and very competitive pricing.

In our review, we had a change in some of the testing. The big differences in our testing for this review was two-fold: the jump from Windows 10 Pro RS2 to Windows 10 Pro RS3, and the inclusion of the Spectre and Meltdown patches to mitigate the potential security issues. These patches are still being rolled out by motherboard manufacturers, with the latest platforms being first in that queue. For our review, we tested the new processors with the latest OS updates and microcode updates, as well as re-testing the Intel Coffee Lake processors as well. Due to time restrictions, the older Ryzen 1000-series results were used.

Due to the tight deadline of our testing and results, we pushed both our CPU and gaming tests live without as much formal analysis as we typically like to do. All the parts were competitive, however it quickly became clear that some of our results were not aligned with those from other media. Initially we were under the impression that this was as a result of the Spectre and Meltdown (or Smeltdown) updates, as we were one of the few media outlets to go back and perform retesting under the new standard.

Nonetheless, we decided to take an extensive internal audit of our testing to ensure that our results were accurate and completely reproducible. Or, failing that, understanding why our results differed. No stone was left un-turned: hardware, software, firmware, tweaks, and code. As a result of that process we believe we have found the reason for our testing being so different from the results of others, and interestingly it opened a sizable can of worms we were not expecting.

An extract from our Power testing script

What our testing identified is that the source of the issue is actually down to timers. Windows uses timers for many things, such as synchronization or ensuring linearity, and there are sets of software relating to monitoring and overclocking that require the timer with the most granularity - specifically they often require the High Precision Event Timer (HPET). HPET is very important, especially when it comes to determining if 'one second' of PC time is the equivalent to 'one second' of real-world time - the way that Windows 8 and Windows 10 implements their timing strategy, compared to Windows 7, means that in rare circumstances the system time can be liable to clock shift over time. This is often highly dependent on how the motherboard manufacturer implements certain settings. HPET is a motherboard-level timer that, as the name implies, offers a very high level of timer precision beyond what other PC timers can provide, and can mitigate this issue. This timer has been shipping in PCs for over a decade, and under normal circumstances it should not be anything but a boon to Windows.

However, it sadly appears that reality diverges from theory – sometimes extensively so – and that our CPU benchmarks for the Ryzen 2000-series review were caught in the middle. Instead of being a benefit to testing, what our investigation found is that when HPET is forced as the sole system timer, it can  sometimes a hindrance to system performance, particularly gaming performance. Worse, because HPET is implemented differently on different platforms, the actual impact of enabling it isn't even consistent across vendors. Meaning that the effects of using HPET can vary from system to system, as well as the implementation.

And that brings us to the state HPET, our Ryzen 2000-series review, and CPU benchmarking in general. As we'll cover in the next few pages, HPET plays a very necessary and often very beneficial role in system timer accuracy; a role important enough that it's not desirable to completely disable HPET – and indeed in many systems this isn't even possible – all the while certain classes of software such as overclocking & monitoring software may even require it. However for a few different reasons it can also be a drain on system performance, and as a result HPET shouldn't always be used. So let's dive into the subject of hardware timers, precision, Smeltdown, and how it all came together to make a perfect storm of volatility for our Ryzen 2000-series review.

A Timely Re-Discovery

Most users have no need to worry about the internals of a computer: point, click, run, play games, and spend money if they want something faster. However one of the important features in a system relates to how they measure time. A modern system relies on a series of both hardware and software timers, both internal and external, in order to maintain a linear relation between requests, commands, execution, and interrupts.

The timers have different users, such as following instructions, maintaining video coherency, tracking real time, or managing the flow of data. Timers can (but not always) use external references to ensure their own consistency – damage, unexpected behavior, and even thermal environments can cause timers to lose their accuracy.

Timers are highly relevant for benchmarking. Most benchmark results are a measure of work performed per unit time, or in a given time. This means that both the numerator and the denominator need to be accurate: the system has to be able to measure what amount of work has been processed, and how long it took to do it in. Ideally there is no uncertainty in either of those values, giving an accurate result.

With the advent of Windows 8, between Intel and Microsoft, the way that the timers were used in the OS were changed. Windows 8 had the mantra that it had to ‘support all devices’, all the way from the high-cost systems down to the embedded platforms. Most of these platforms use what is called an RTC, a ‘real time clock’, to maintain the real-world time – this is typically a hardware circuit found in almost all devices that need to keep track of time and the processing of data. However, compared to previous versions of Windows, Microsoft changed the way it uses timers, such that it was compatible with systems that did not have a hardware-based RTC, such as low-cost and embedded devices. The RTC was an extra cost that could be saved if the software was built to do so.

Ultimately, any benchmark software in play has to probe the OS to determine the current time during the benchmark to then at the end give an accurate result. However the concept of time, without an external verifying source, is an arbitrarily defined constant – without external synchronization, there is no guarantee that ‘one second’ on the system equals ‘one second’ in the real world. For the most part, all of us rely on the reporting from the OS and the hardware that this equality is true, and there are a lot of hardware/software engineers ensuring that this is the case.

However, back in 2013, it was discovered that it was fairly easy to 'distort time' on a Windows 8 machine. After loading into the operating system, any adjustment in the base frequency of the processor, which is usually 100 MHz, can cause the ‘system time’ to desynchronise with ‘real time’. This was a serious issue in the extreme overclocking community, where world records require the best system tuning: when comparing two systems at the same frequency but with different base clock adjustments, up to a 7% difference in results were observed when there should have been a sub-1% difference. This was down to how Windows was managing its timers, and was observed on most modern systems.

For home users, most would suspect that this is not an issue. Most users tend not to adjust the base frequencies of their systems manually. For the most part that is true. However, as shown in some of our motherboard testing over the years, frequency response due to default BIOS settings can provide an observable clock drift around a specified value, something which can be exacerbated by the thermal performance. Having a system with observable clock drift, and subsequent timing drift, is not a good thing. It relies on the accuracy and quality of the motherboard components, as well as the state of the firmware. This issue has formally been classified as ‘RTC Bias’.

The extreme overclocking community, after analysing the issue, found a solution: forcing the High Performance Event Timer, known as HPET, found in the chipset. Some of our readers will have heard of HPET before, however our analysis is more interesting than it first appears.

Why A PC Has Multiple Timers

Aside from the RTC, a modern system makes use of many timers. All modern x86 processors have a Time Stamp Counter (TSC) for example, that counts the number of cycles from a given core, which was seen back in the day as a high-resolution, low-overhead way to get CPU timing information. There is also a Query Performance Counter (QPC), a Windows implementation that relies on the processor performance metrics to get a better resolution version of the TSC, which was developed in the advent of multi-core systems where the TSC was not applicable. There is also a timer function provided by the Advanced Configuration and Power Interface (ACPI), which is typically used for power management (which means turbo related functionality). Legacy timing methodologies, such as the Programmable Interval Timer (PIT), are also in use on modern systems. Along with the High Performance Event Timer, depending on the system in play, these timers will run at different frequencies.

Ryzen 7 2700X with HPET Off ASUS ROG Crosshair VII Hero	Core i7-8700K with HPET Off ASRock Z370 Gaming i7
Core i7-6950X with HPET On ASUS X99-E-10G	Core i7-6700K with HPET Off GIGABYTE X170-Extreme ECC
Core i5-5200U with HPET Off GIGABYTE BRIX	Core i7-3960X with HPET Off EVGA X79 SLI

The timers will be used for different parts of the system as described above. Generally, the high performance timers are the ones used for work that is more time sensitive, such as video streaming and playback. HPET, for example, was previously referred to by its old name, the Multimedia Timer. HPET is also the preferred timer for a number of monitoring and overclocking tools, which becomes important in a bit.

With the HPET timer being at least 10 MHz as per the specification, any code that requires it is likely to be more in sync with the real-world time (the ‘one-second in the machine’ actually equals ‘one-second in reality’) than using any other timer.

In a standard Windows installation, the operating system has access to all the timers available. The software used above is a custom tool developed to show if a system has any of those four timers (but the system can have more). For the most part, depending on the software instructions in play, the operating system will determine which timer is to be used – from a software perspective, it is fundamentally difficult to determine which timers will be available, so the software is often timer agnostic. There is not much of a way to force an algorithm to use one timer or another without invoking specific hardware or instructions that rely on a given timer, although the timers can be probed in software like the tool above.

HPET is slightly different, in that it can be forced to be the only timer. This is a two stage process:

The first stage is that it needs to be enabled in the BIOS. Depending on the motherboard and the chipset, there may or may not be an option for this. The options are usually for enable/disable, however this is not a simple on/off switch. When disabled, HPET is truly disabled. However, when enabled, this only means that the HPET is added to the pool of potential timers that the OS can use.

The second stage is in the operating system. In order to force HPET as the only timer to be used for the OS, it has to be explicitly mentioned in the system Boot Configuration Data (BCD). In standard operation, HPET is not in the BCD, so it remains in the pool of timers for the OS to use. However, for software to guarantee that the HPET is the only timer running, the software will typically request to make a change and make an accompanying system reboot to ensure the software works as planned. Ever wondered why some overclocking software requests a reboot *before* starting the overclock? One of the reasons is sometimes to force HPET to be enabled.

This leads to four potential configuration implementations:

1. BIOS enabled, OS default: HPET is in list of potential timers
2. BIOS enabled, OS forced: HPET is used in all situations
3. BIOS disabled, OS default: HPET is not available
4. BIOS disabled, OS forced: HPET is not available

Again, for extreme overclockers relying on benchmark results to be equal on Windows 8/10, HPET has to be forced to ensure benchmark consistency. Without it, the results are invalid.

The Effect of a High Performance Timer

With a high performance timer, the system is able to accurately determine clock speeds for monitoring software, or video streaming processing to ensure everything hits in the right order for audio and video. It can also come into play when gaming, especially when overclocking, ensuring data and frames are delivered in an orderly fashion, and has been shown to reduce stutter on overclocked systems. And perhaps most importantly, it avoids any timing issues caused by clock drift.

However, there are issues fundamental to the HPET design which means that it is not always the best timer to use. HPET is a continually upward counting timer, which relies on register recall or comparison metrics rather than a ‘set at x and count-down’ type of timer. The speed of the timer can, at times, cause a comparison to fail, depending on the time to write the compared value to the register and that time already passing. Using HPET for very granular timing requires a lot of register reads/writes, adding to the system load and power draw, and in a workload that requires explicit linearity, can actually introduce additional latency. Usually one of the biggest benefits to disabling HPET on some systems is the reduction in DPC Latency, for example.

AMD and Intel Have Different HPET Guidance

A standard modern machine, with a default BIOS and a fresh Windows operating system, will sit on the first situation in the table listed above: the BIOS has HPET enabled, however it is not explicitly forced in the operating system. If a user sets up their machine with no overclocking or monitoring software, which is the majority case, then this is the implementation you would expect for a desktop.

AMD

We reached out to AMD and Intel about their guidance on HPET, because in the past it has both been unclear as well as it has been changed. We also reached out to motherboard manufacturers for their input.

For those that remember the Ryzen 7 1000-series launch, about a year ago from now, one point that was lightly mentioned among the media was that in AMD’s press decks, it was recommended that for best performance, HPET should be disabled in the BIOS. Specifically it was stated that:

Make sure the system has Windows High Precision Event Timer (HPET) disabled. HPET can often be disabled in the BIOS. [T]his can improve performance by 5-8%.

The reasons at the time were unclear as to why, but it was a minor part in the big story of the Zen launch so it was not discussed in detail. However, by the Ryzen 5 1000-series launch, that suggestion was no longer part of the reviewer guide. By the time we hit the Ryzen-2000 series launched last week, the option to adjust HPET in the BIOS was not even in the motherboards we were testing. We cycled back to AMD about this, and they gave the following:

The short of it is that we resolved the issues that caused a performance difference between on/off. Now that there is no need to disable HPET, there is no need for a toggle [in the BIOS].

Interestingly enough, with our ASUS X470 motherboard, we did eventually find the setting for HPET – it was not in any of the drop down menus, but it could be found using their rather nice ‘search’ function. I probed ASUS about whether the option was enabled in the BIOS by default, given that these options were not immediately visible, and was told:

It's enabled and never disabled, since the OS will ignore it by default. But if you enable it, then the OS will use it – it’s always enabled, that way if its needed it is there, as there would be no point in pulling it otherwise.

So from an AMD/ASUS perspective, the BIOS is now going to always be enabled, and it needs to be forced in the OS to be used, however the previous guidance about disabling it in the BIOS has now gone, as AMD expects performance parity.

It is worth noting that AMD’s tool, Ryzen Master, requires a system restart when the user first loads it up. This is because Ryzen Master, the overclocking and monitoring tool, requires HPET to be forced in order to do what it needs to do. In fact, back at the Ryzen 7 launch in 2017, we were told:

AMD Ryzen Master’s accurate measurements present require HPET. Therefore it is important to disable HPET if you already installed and used Ryzen Master prior to game benchmarking.

Ultimately if any AMD user has Ryzen Master installed and has been run at any point, HPET is enabled, even if the software is not running or uninstalled. The only way to stop it being forced in the OS is with a command to chance the value in the BCD, as noted above.

For the Ryzen 2000-series launch last week, Ryzen Master still requires HPET to be enabled to run as intended. So with the new guidance that HPET should have minimal effect on benchmarks, the previous guidance no longer applies.

Ryzen Master is not the only piece of software that requires HPET to be forced in order to do what it needs to do. For any of our readers that have used overclocking software and tools before, or even monitoring tools such as fan speed adjusters – if those tools have requested a restart before being used properly, there is a good chance that in that reboot the command has been run to enable HPET. Unfortunately it is not easy to generate a list, as commands and methods may change from version to version, but it can apply to CPU and GPU overclocking.

Intel

The response we had from Intel was a little cryptic:

[The engineers recommend that] as far as benchmarking is concerned, it should not matter whether or not HPET is enabled or not. There may be some applications that may not function as advertised if HPET is disabled, so to be safe, keep it enabled, across all platforms. Whatever you decide, be consistent across platforms.

A cold reading of this reply would seem to suggest that Intel is recommended HPET to be forced and enabled, however my gut told me that Intel might have confused ‘on’ in the BIOS with ‘forced’ through the OS, and I have asked them to confirm.

Looking back at our coverage of Intel platforms overall, HPET has not been mentioned to any sizeable degree. I had two emails back in 2013 from a single motherboard manufacturer stating that disabling HPET in the BIOS can minimise DPC latency on their motherboard, however no comment was made about general performance. I cannot find anything explicitly from Intel though.

Forcing HPET On, Plus Spectre and Meltdown Patches

Based on my extreme overclocking roots back in the day, my automated benchmark scripts for the past year or so have forced HPET through the OS. Given that AMD’s guidance is now that it doesn’t matter for performance, and Intel hasn’t even mentioned the issue relating to a CPU review, having HPET enabled was the immediate way to ensure that every benchmark result was consistent, and would not be interfered with by clock drift on special motherboard manufacturer in-OS tweaks. This was a fundamental part of my overclocking roots – if I want to test a CPU, I want to make certainly sure that the motherboard is not causing any issues. It really gets up my nose when after a series of CPU testing, it turns out that the motherboard had an issue – keeping HPET on was designed to stop any timing issues should they arise.

From our results over that time, if HPET was having any effect, it was unnoticed: our results were broadly similar to others, and each of the products fell in line with where they were expected. Over the several review cycles we had, there were a couple of issues that cropped up that we couldn’t explain, such as our Skylake-X gaming numbers that were low, or the first batch of Ryzen gaming tests, where the data was thrown out for being obviously wrong however we never managed to narrow down the issue.

Enter our Ryzen 2000 series numbers in the review last week, and what had changed was the order of results. The way that forcing HPET was affecting results was seemingly adjusted when we bundle in the Spectre and Meltdown patches that also come with their own performance decrease on some systems. Pulling one set of results down further than expected started some alarm bells and needed closer examination.

HPET, by the way it is invoked, is programmed by a memory mapped IO window through the ACPI into the circuit found on the chipset. Accessing it is very much an IO command, and one of the types of commands that fall under the realm of those affected by the Spectre and Meltdown patches. This would imply that any software that required HPET access (or all timing software if HPET is forced) would have the performance reduced even further when these patches are applied, further compounding the issue.

It Affects AMD and Intel Differently: Productivity

So far we have done some quick initial re-testing on the two key processors in this debate, the Ryzen 7 2700X and Intel Core i7-8700K. These are the two most talked about processors at this time, due to the fact that they are closely matched in performance and price, with each one having benefits in certain areas over the other. For our new tests, we have enabled the Spectre/Meltdown patches on both systems – HPET is ‘on’ in the BIOS, but left as ‘default’ in the operating system.

For our productivity tests, on the Intel system, there was an overall +3.3% gain when un-forcing HPET in the OS:

The biggest gains here were in the web tests, a couple of the renderers, WinRAR (memory bound), and PCMark 10. Everything else was pretty much identical. Our compile tests gave us three very odd consecutive numbers, so we are looking at those results separately.

On the AMD system, the productivity tests difference was an overall +0.3% gain when un-forcing HPET in the OS:

This is a lower gain, with the biggest rise coming from PCMark10’s video conference test to the tune of +16%. The compile test results were identical, and a lot of tests were with 1-2%.

If Affects AMD and Intel Differently: Gaming

The bigger changes happen with the gaming results, which is the reason why we embarked on this audit to decipher our initial results. Games rely on timers to ensure data and pacing and tick rates are all sufficient for frames to be delivered in the correct manner – the balance here is between waiting on timers to make sure everything is correct, or merely processing the data and hoping it comes out in more or less the right order: having too fine a control might cause performance delays. In fact, this is what we observe.

With our GTX 1080 and AMD’s Ryzen 7 2700X, we saw minor gains across the board, however it was clear that 1080p was the main beneficiary over 4K. The 10%+ adjustments came in only Civilization 6 and Rise of the Tomb Raider.

Including the 99^th percentile data, removing HPET gave an overall boost of around 4%, however the most gains were limited to specific titles at the smaller resolutions, which would be important for any user relying on fast frame rates at lower resolutions.

The Intel side of the equation is where it gets particularly messy. We rechecked these results several times, but the data was quite clear.

As with the AMD results, the biggest beneficiaries of disabling HPET were the 1080p tests. Civilization 6 and Rise of the Tomb Raider had substantial performance boosts (also in 4K testing), with Grand Theft Auto observing an additional +27%. By comparison, Shadow of Morder was ‘only’ +6%.

Given that the difference between the two sets of data is related to the timer, one could postulate that the more granular the timer, the more the effect it can have: on both of our systems, the QPC timer is set for 3.61 MHz as a baseline, but the HPET frequencies are quite different. The AMD system has a HPET timer at 14.32 MHz (~4x), while the Intel system has a HPET timer at 24.00 MHz (~6.6x). It is clear that the higher granularity of the Intel timer is causing substantially more pipeline delays – moving from a tick-to-tick delay of 277 nanoseconds to 70 nanoseconds to 41.7 nanoseconds is crossing the boundary from being slower than a CPU-to-DRAM access to almost encroaching on a CPU-to-L3 cache access, which could be one of the reasons for the results we are seeing, along with the nature of how the HPET timer works.

There is also another aspect to gaming that does not appear with standard CPU tests: depending on how the engine is programmed, some game developers like to keep track of a lot of the functions in flight in order to either adjust features on the fly, or for internal metrics. For anyone that has worked extensively on a debug mode and had to churn through the output, it is basically this. If a title had shipped with a number of those internal metrics still running in the background, this is exactly the sort of issue that having HPET enabled could stumble upon - if there is a timing mismatch (based on the way HPET works) and delays are introduced due to these mismatches, it could easily slow down the system and reduce the frame rate.

Conclusion: It Changes Our Results

When we first published our Ryzen-2000 series review, with HPET forced as the timer in the operating system, our results were broadly showing that the new processors leading the pack. In light of the audit, especially with the way that the Intel gaming results have changed, paint a different picture.

At 1080p, the Core i7-8700K has a clear lead in most titles, although that lead does somewhat vanish moving to 4K, except with Civilization. Ultimately for any user pushing the pixel count, in our tests for the most part, the chips retain parity performance. AMD’s claims for the Ryzen-2000 launch were more focused on the 1440p gaming, however it is clear that there is still margin that benefits Intel at the most popular resolutions, such as 1080p.

Why This Matters, and How AnandTech is Set to Test in the Future

The interesting thing to come out of both Intel and AMD is that they seem to not worry if HPET is enabled or not. Regardless of the advice in the past, both companies seem to be satisfied when HPET is enabled in the BIOS and irreverent when HPET is forced in the OS. For AMD, the result change was slight but notable. For Intel we saw substantial drops in performance when HPET was the forced timer, and removing that lock improved performance.

It would be odd to hear if Intel are not seeing these results internally, and I would expect them to be including an explicit mention of ensuring the HPET settings in the operating system in every testing guide. Or Intel's thinking could be that because HPET not being forced is the default OS position, they might not see it as a setting they need to explicitly mention in their reviewer guides. Unfortunately, this opens up possibilities when it comes to overclocking software interfering with how the timers are being used.

As noted above, overclocking and monitoring tools like Ryzen Master request a restart when used for the first time in order to make sufficient changes to the system to run correctly. Some of this software will be forcing HPET in the BCD in order to enable what it needs to do, and the adjustment is unlikely to be explicitly mentioned in the request to restart. In a standard review, it is typically expected that each system will have a fresh OS and fresh software install, such that systems are tested as if it were new. For any user looking to tune the system, this is the point where any potential software issues could occur. Now should a reviewer decide to first analyze the software bundled with the system before testing or after testing could have significantly different results. It can create a conundrum, as has clearly been the case for us.

Moving forward, the immediate goal here at AnandTech is to ensure that our readers have the most up-to-date and correct results, particularly for our Ryzen 2000-series review. As a result, we are taking a few steps both immediately and in the future to correct our data, update our Ryzen 2000-series review, and to prevent this issue going forward.

First and foremost, we have decided that force-enabling HPET is not how we want to test systems, as this is non-default behavior. While it has an important role in extreme overclocking, to verify accurate timing, ultimately it was akin to taking a sledgehammer to cracking an egg for our testing - we need to be testing systems at stock. So all further CPU testing going forward will be using HPET's default behavior, and we have even put checks in our scripts to ensure this is now the case.

As a result we are retracting our existing results for all of the processors we used in the Ryzen 2000-series review. This goes for both the review and for Bench. All of these products will be updated with revised results using the default HPET behavior just as soon as the updated data is available over the course of the next week. In fact we're already the process of running this updated testing, which we've used for this article and uploaded to Bench.

The end goal here is to cover most of the popular processors from the previous few generations on our existing 2017 benchmark suite in order to fully update and republish our Ryzen 2000-series review. Meanwhile, because the results in that review are still being updated, the conclusion for that review is also being retracted. We don't anticipate updated results meaningfully changing that conclusion, but it is inappropriate to have a conclusion remain published until we have all of the data we need.

Longer-term, because this issue goes back further than just the Ryzen 2000-series review and we are already on the cusp of organizing our 2018 CPU benchmarking suite, we're also accelerating our rollout of that suite. After replacing the data for key hardware on that 2017 test suite, we will be rolling out the 2018 update in earnest. The 2018 CPU benchmarking suite will upgrade to the latest software, drivers, and a change-up on games (F1 2017, Shadow of War, Far Cry 5; also had requests for Deus Ex). Our 2018 suite will require that Spectre and Meltdown patches are in place for the systems we test, to ensure that everyone has access to the latest data.

(ed: It should be noted that this only affects Ian's CPU review data; Brett and Nate run different tools in their laptop and GPU reviews respectively)

Overall we expect to be done collecting data to finish and update the Ryzen 2000-series review next week. After that, it will take some time to roll out the 2018 CPU benchmarking suite data, but that should only be on the order of weeks assuming that there are no further surprises (ed: knock on wood).

We also would like to give all of our readers and colleagues a sincere thank you for assisting with this analysis. We continually strive to publish the best possible data, so your input is and always will be invaluable for finding patterns and oddities we may have missed.

Finally, while we're on the subject of timers, we'd like to throw out an open-ended question to everyone: given what we've found, should the use/requirement of HPET in software be made clearer? Or is there a risk that information being more confusing than helpful? One of the issues we grappled with in writing this article is that while HPET can have a performance impact, it's also not necessarily wrong to use it given its unique accuracy. So we're interested in hearing from all of you on how you think the use of HPET should be documented, so that users aren't caught off-guard by the potential performance impact..

 

Update: 04/26

HPET and Invariant TSC

Since publishing this follow up, several readers have reached out about their experiences with timers, as well as offering deeper explanations of some of the key points in this article. I will attempt to cover some of them here.

The main on-die CPU timer is the Time Stamp Counter (TSC), which was one of the main timers in single core systems. With the movement to multi-core, HPET became the new more accurate timer that as described can protect against clock drift. HPET was preferred to TSC, but can take 10-100x longer to be probed, due to its location on the chipset. The industry however is moving back towards TSC through an Invariant TSC (ITSC), which is a version of TSC that is stable through CPU frequency changes and C-state changes. The ITSC is accessed through the RDTSC instruction, which can be used simultaneously by both the kernel and user code if permitted (unlike HPET, which is a locked timer), and is sufficient for multi-core systems. And although this method still has the RTC bias issue, the lower latency is favoured by all, except overclockers adjusting the platform's 100 MHz base frequency.

TL;DR: HPET can take 1000s of cycles to read, and reading it with multiple cores compounds the issue. Invariant TSC, as a core instruction, is a potential solution with lower latency already in use.

“There is a HPET Bug, No Intel is Not Cheating” and TimerBench

Matthias from Overclockers.at reached out to me and linked me to his article on how they have previously encountered the issue. The article is a nice read, and well worth clicking through:

The HPET bug: What it is and what it isn't

Matthais explains how during their X299 testing, they were experiencing slowdown in their game benchmarks, and pin-pointing the problem with HPET. (We also had similar issues, and didn’t post results, but never got to the bottom of the issue.) As a result, the team over at Overclockers.at developed a tool called TimerBench in order to determine the effect of HPET. As noted, HPET has a much longer latency, but is more accurate.

In the results from overclockers.at one metric stood out: moving from Broadwell-E to Skylake-X meant that the number of theoretical peak HPET calls per second reduced from 1.4 million to 0.2 million – the latency to make a HPET call suddenly became 7x longer with Skylake-X. TimerBench, the tool developed, provides an Unreal 4.7.2 scene and measures timer calls between a system running a game, and one without.

For our results, we used TimerBench on each system with a GTX 1080 in 1920x1080 mode, running fullscreen.

With the HPET timer, the i7-8700K system went from 214k timer calls per second outside of a game down to 144k timer calls per second, which is about the same fraction as with the ITSC timer. The big difference however is the frame rate, decreasing from 289 FPS with ITSC to 238 FPS with HPET, as well as the average GPU load, down from 97.6% to 78.1%. This is shown in the maximum frame time as well.

TimerBench 1.3: GTX 1080 at 1920x1080p
	ITSC		HPET		Frames Per Second		Average GPU Load
	Calls OS	Calls Game	Calls OS	Calls Game	ITSC	HPET	ITSC	HPET
Desktop: GTX 1080 at 1920x1080
Ryzen 7 1800X	27.7m	2.0m	0.4m	0.3m	283	279	96%	95%
Core i7-8700K	40.3m	2.7m	0.2m	0.1m	289	238	98%	78%
Core i7-7820X	35.5m	2.4m	0.2m	0.1m	285	252	95%	83%
Core i7-6700K	36.1m	2.3m	0.2m	0.1m	286	258	96%	85%
Core i7-6950X*	91.8m	1.3m	1.1m	0.6m	285	262	98%	96%
Mobile: MX 150 at 800x600
Core i7-8550U	34.3m	0.9m	0.2m	0.06m	148	137	-	-

* No Spectre/Meltdown Patches

When I correlate this data with the systems I have currently running, we see that the AMD Ryzen 7 1800X system is not particularly affected, but all of our Intel systems are: Skylake-S, Skylake-X, Coffee Lake, and even our mobile device. What is clear that the HPET timer is causing performance degredation by virtue of having a lower average GPU load. If the GPU is waiting on the same timing delays caused by HPET, this would lover the overall GPU load.

So this interesting correlation leads me to think that maybe this issue, aside from potential Spectre/Meltdown related points, is related to the chipset. HPET circuits are normally found on the chipset/southbridge, and in this case Intel has a wide HSIO chipset design in all the systems tested. As the chipset is, among other things, a PCIe switch, then it has various buffers to deal with the data coming in and out. The effect of these wide chipset and buffers might be part of the HPET issue. I need to go dig out an older system.