One of the longest running gaming franchises around, the Civilization series goes all the way back to my high school years. There have been many changes along the way, but the core gameplay has remained. Civilization is a turn-based strategy game where you guide your fledgling empire from the early stages of a single settlement up through global endgame battles for domination. The games have all been digital crack as far as I'm concerned, and I know at least a few people that failed – or very nearly failed – out of college thanks to time spent playing various incarnations of Civilization. The latest update, as the name implies, takes the series outside the confines of Earth once more – a place Civilization hasn't really been since Alpha Centauri back in 1999. If you happen to enjoy turn-based strategy like Ryan or me, say goodbye to your loved ones and prepare for a trip down the rabbit hole of Civilization once more….

One of the interesting aspects with Civilization: Beyond Earth is that the developers, Firaxis, have opted to include support for AMD's Mantle API. We've been over this subject quite a few times, but the core idea is that Mantle provides low level access to the graphics hardware, potentially allowing for improved performance or other improvements. We've seen other games use Mantle (Battlefield 4, Thief, Plants vs. Zombies: Garden Warfare, and Sniper Elite 3), but this is the first game to really do something more with Mantle than reducing the cost of draw calls. That's still a benefit that Civilization: Beyond Earth enjoys, but Firaxis has gone a step further for owners of CrossFire setups and has enabled a custom Split-Frame Rendering (SFR) approach for multiple GPUs instead of the typical Alternate-Frame Rendering (AFR) that most games use with CrossFire and SLI.

I like the idea of SFR, as it means adding multiple GPUs won't increase input latency, and in fact from a theoretical perspective SFR could actually reduce latency. Consider one GPU that takes 20ms to render a frame, and if you have two GPUs each taking 20ms you can get a new frame every 10ms, but buffering means you're still seeing latency of around 20ms (and in worst-case scenarios up to 40ms). With SFR, the two GPUs could potentially work together to render a single frame in 10ms, cutting input latency in half. That's the potential upside, but of course it can be far more complex to implement SFR (which is why most games choose to go with AFR).

I don't have multiple NVIDIA GPUs to test SLI performance, and normally I wouldn't worry too much about CrossFire, but the Mantle SFR support does make for a potentially interesting outcome so I'll at least test the two CrossFire configurations I have on hand (R9 290X and R9 280). I should note that the preview release of Civilization: Beyond Earth that we're testing does have a specific limitation that prevents it from using more than two GPUs; in fact, if you have more than two GPUs, you apparently need to disable the extra GPUs in the device manager or the game won't launch, though a 0-day patch is in the works to address this. Anyway, we'll look at CrossFire performance with and without Mantle later on.

Test Settings

As our testing environments at AnandTech tend to vary a bit by editor, let me again reiterate that I'm not using the same setup as Ryan uses for our GPU reviews; for gaming benchmarks like this, I have the following hardware:

Basically, my hardware is more of a "mainstream" gaming setup rather than an extreme configuration (e.g. LGA2011), though I did overclock the CPU to 4.1GHz to remove as much of the CPU bottleneck as possible. For anyone running a single GPU – and in most cases even with two GPUs – this sort of build is more than fast enough to keep the GPUs happy. I've also upgraded to a 4K display, Acer's XB280HK G-SYNC panel, which will help tax the faster GPUs. And it's a good thing too, as Civilization: Beyond Earth can hit the CPU pretty hard, with ultra-high resolutions being one of the few things that will help differentiate between the various high-end GPUs.

For testing Civilization: Beyond Earth, I've selected four settings: 3840x2160, 2560x1440, and 1920x1080 with Ultra quality and 4xMSAA, and then as a slightly less strenuous setting I've also tested 1920x1080 High without any anti-aliasing. I'm using the latest AMD 14.9 Beta 2 Catalyst drivers, which should be publicly available by the time you read this; for NVIDIA GPUs I've updated to the recently released 344.48 WHQL driver that's "Game Ready" for Civilization: Beyond Earth. Of course this is an AMD Gaming Evolved title, and with Mantle support I've also tested the AMD GPUs using both the DX11 rendering path as well as the Mantle rendering path. There doesn't appear to be any difference in terms of image quality, so Mantle is being used purely as an option to increase performance.

One final item to note is that the built-in benchmark basically represents something of a worst-case scenario for performance. Civilization: Beyond Earth has a grid of hexes over the terrain, and early on there aren't too many units on screen. Near the end of a game however, it's not unusual to have most hexes occupied – or at least a lot of them – and that can cause performance to tank. As you can see in the above screenshot taken from the benchmark, Civilization: Beyond Earth really can't get much worse than what we're showing here; very nearly every hex on the map is occupied. And with that said, let's see how things play out….

Civilization: Beyond Earth Performance

I wouldn't expect many games to run well at a 4K resolution with Ultra quality settings and 4xMSAA, especially when running a single GPU, but Civilization: Beyond Earth proves to be relatively forgiving. The GeForce GTX 980 tops our charts for performance (AMD says their R9 290X 8GB card should be slightly faster – though that claim was made before NVIDIA released their Game Ready 344.48 drivers), with the R9 290X coming in second place. In terms of pure bang for the buck, the R9 290X is of course the winner, since it can now be had for well under $400.

Meanwhile the GTX 970 and GTX 780 are basically in a dead heat for third place (the R9 290, which I don't have, would likely come in slightly ahead of the GTX 970). Somewhat surprisingly, even the venerable R9 280X is able to break the 30FPS mark, at least with the slight boost it receives courtesy of Mantle.

What the performance figures don't necessarily show – and what is immediately obvious when you're swapping between GPUs – is just how loud the R9 280/280X/290X blower GPUs can get! It's seriously almost laughable how hard the fans are working on my AMD GPUs. There are definitely quieter Radeon R9 GPUs, but even then they're going to be hard pressed to match the GTX 970 or 980.

Dropping to 2560x1440, nearly all of our tested GPUs break the 30FPS mark with room to spare, though minimum frame rates can at times be in the sub-20 range. However, since Civilization isn't exactly the sort of game where fast reflexes are required, the occasional stutter won't be too much of a problem. Our overall rankings remain basically unchanged – and that holds for testing at 1920x1080 as well (for both Ultra + 4xMSAA and High + No AA settings).

Lest anyone get the idea that Civilization: Beyond Earth isn't a taxing game, do pay attention to the R7 250X results. Even at 1080p High, it's still languishing in the sub-30 FPS realm; you'll need to drop the quality and/or resolution another notch to get above 30FPS with that level of hardware. (Note that the MSI GE60 GPU is basically the same as a GTX 750 Ti, so you can use that as a reference point for $120-$150 GPUs.)

Notebook Performance

Moving over to the notebook side of things, first I need to point out a problem that came up in testing: the MSI GS60 basically wouldn't work properly at higher resolutions, even with an external display. Civilization: Beyond Earth would simply crash during the loading process, possibly due to problems with video decoding. I don't know if the issue is with the Intel GPU at High DPI resolutions or an Optimus problem or something else, but if you load the game with a lower resolution (e.g. 1920x1080), everything is okay.

As for the GE60, it only has an HDMI output, so I couldn't test at anything above 1920x1080 regardless. The MSI GT70 with 880M on the other hand doesn't use the Intel GPU for the DisplayPort connection, effectively bypassing Optimus it seems; I need to go back and retest at some point, but it will probably be a while.

Even the MSI GS60 with its 3K panel wouldn't work at the native resolution, so that's at least one bug to fix (though whether it's in the game, the Intel drivers, or the NVIDIA drivers I couldn't say). Ultimately, I was only able to test most of the notebooks at 1080p Ultra and High settings. The exception is the GT72 with GTX 980M, which worked fine with the Acer XB280HK.

As far as performance goes, all of the notebooks are able to handle Civilization: Beyond Earth at 1080p, and the 980M has plenty of juice left in the tank for even higher resolutions. The GTX 860M on the other hand struggles a bit to stay above 30FPS in the most demanding scenarios (basically zoomed out all the way with tons of units on most of the hexes), but overall it's still very playable – and on smaller maps as well as earlier in the game, performance will be much higher than the built-in benchmark results. We're also likely hitting a bit of CPU bottlenecks on the faster laptops, though it's probably not enough to really matter. Even with a slightly slower CPU, the GTX 980M is able to consistently run faster than the GTX 880M.

AMD Mantle vs. DX11

Flipping over to the AMD Mantle vs. DX11 side of things, Mantle tends to provide a consistent though small improvement in performance. Even the lowly R7 250X sees marginal improvements in performance at settings that are otherwise too high for the GPU, to the tune of 3-5% increases in performance at 1080p. You would probably think that Mantle will benefit the fastest GPUs the most, but that doesn't appear to be the case either; the R9 280 and R9 280X see the biggest improvements (8-10% at our Ultra settings), while the R9 290X gets a slightly less impressive bump (4-6% faster). Also somewhat unusual compared to other Mantle games we've looked at is that our less demanding High quality settings see a smaller improvement than the Ultra settings.

While Mantle gives developers more control over the hardware, unfortunately that means developers also have to do a bit more optimization work to wring the most performance out of each different GPU. The R9 280/280X have been very popular cards so they're in widespread use, while the newer R9 290X may not be reaching its full potential yet. But of course there's another topic to discuss with Mantle….

Civilization: Beyond Earth Minimum Frame Rates

One of the requests I've received since initially posting this overview of performance was to investigate minimum frame rates. All of the necessary data was already available, but I just hadn't spent much time investigating the subject. It turns out the minimum frame rates end up being a bit more interesting when we get to the subject of Mantle.

Before we get to the charts, let me also explain one detail that can be important: I don't really care much about the pure minimum frame rate. If a game has to load a texture or some other hiccup causes a single frame dip to 10 FPS, but all of the remaining frames render at 90FPS over a two minute period, does that one dip matter much? I don't think so. To better show the impact of minimum frame rates, I've averaged the bottom 1% of all frames over the benchmark sequence. This makes the measurement much more significant, because if you're routinely seeing downward spikes in frame rates it can be distracting.

In the case of Civilization: Beyond Earth, I didn't notice any issues with inconsistent frame rates, so this "Lowest 1% FPS" measurement isn't quite so important, but in other games it can mean the difference between reporting 30FPS as the minimum and reporting 130FPS as the first percentile of frame rates. Basically, Civilization: Beyond Earth hits minimum FPS when you zoom all the way out, particularly when there are a lot of units on the screen, and that's exactly what happens in the benchmark at about the halfway point.

With that explanation out of the way, let's look at the charts:

One thing that immediately jumps out is that the improvements in minimum frame rates with Mantle tend to be even higher than the improvements in average frame rates. This is actually more or less what you would expect, particularly in this case where the minimum frame rates appear to be caused by CPU bottlenecks on the faster GPUs like the R9 290X. Lighten the CPU load with Mantle and that allows for frames to be generated more quickly in such cases.

Where average FPS only improved about 5-10% with Mantle, the improvements in minimum frame rates are much more noticeable. The lowly R7 250X sees the least benefit, which makes sense as it's almost totally GPU limited. The R7 250X still improves by 3-5% at our highest resolutions, and a more significant 9% increase at 1080p Ultra; but look at the 1080p High result: it's 19% faster on minimum frame rates. What's more, that 19% mark ends up being pretty tame in comparison to the improvements on the faster GPUs.

The R9 280 improves by 12-25%, with the biggest jump coming at 1080p Ultra. The R9 280X sees 17-24% improvements in minimum frame rates, again with the largest increase coming at 1080p Ultra. And then there's the R9 290X. Interestingly, it only sees a small 5% improvement at 4K Ultra, but at QHD it gets a 16% bump; at 1080p Ultra and High settings, Mantle really frees up some additional performance and the minimum frame rates increase by 39% and 53%!

Moving to the AMD vs. NVIDIA side of things, the R9 290X actually manages to come out ahead of the GTX 980 (though just barely) at 4K and QHD resolutions. The margin of victory for AMD is also quite pronounced at QHD, where it leads by a not insignificant 19% with Mantle enabled (but only 3% with DX11). The R9 280X tells a similar story, where it loses to the GTX 770 by a sometimes large margin using the DX11 path, but Mantle closes the gap and often gives the 280X a lead over the 770.

Looking quickly at the laptops, the GTX 980M continues to be a force to be reckoned with, delivering performance generally slightly above that of the GTX 770 and R9 280X, particularly at the GT72's native 1080p resolution. The GTX 880M also manages to stay above 30FPS in our testing, but the 870M and 860M drop into the low to mid 20s. That said, Civilization: Beyond Earth remains playable even at slightly lower frame rates, as the turn-based nature doesn't generally require fast responses and the mouse input is already decoupled from the rendering pipeline.

AMD CrossFire Performance

Update, 10/24: Turns out there's a "hidden" setting in the configuration file that needs to be manually enabled for Mantle SFR rendering on CrossFire setups to work. Go to your configuration file (%homepath%\Documents\my games\Sid Meier's Civilization Beyond Earth\GraphicsSettings.ini) and change the one line to say "Enable MGPU=1". This will allow Mantle SFR to work properly. I had to retest CrossFire performance, and the updated results are below, with a more detailed explanation posted in a separate article.

Last but not least we have some early results for CrossFire performance. With the appropriate tweak to the configuration file, Mantle CrossFire support adds an interesting wrinkle to the mix. Regular CrossFire still works as usual, but minimum frame rates take a hit relative to single GPUs even while average framerates improve. Here are a couple graphs showing scaling of performance as we move through the various settings. I've included both single GPU and CrossFire results for the R9 280 and R9 290X, both with and without Mantle:

CrossFire 290X ends up pushing past the 60FPS barrier, even at 4K with 4xMSAA, so that's quite a feat. Except, minimum frame rates take a hit and go from 32FPS with a single 290X to 20FPS with CrossFire 290X. Enabling Mantle changes the story quite a bit; the CrossFire Mantle configuration can't quite push past 60FPS, but at 56FPS it's at least close and the minimum frame rate of 37FPS is better than both the single GPU and single GPU with Mantle figures.

We can also see quite nicely in this graph that the CPU bottleneck at High and Ultra settings is around 110-120FPS, even with a 4.1GHz Core i7-4770K, while our minimum FPS bottleneck is even lower: 65FPS for CrossFire Mantle and 49FPS with a single 290X (without Mantle).

As for the R9 280 CF configuration, the two Tahiti GPUs (without Mantle) come just short of matching the performance of a single R9 290X Hawaii card. Mantle again improves minimum frame rates, but the gap between R9 290X Mantle and R9 280 CF Mantle grows. This is one more reason why we generally prefer single GPU configurations, at least up to the point where you have the fastest GPU you can buy.

Civilization: Beyond Earth Testing Notes

Along with discussions of the performance, there are also a few bugs and other quirks in many games that could use fixing. In the case of Civilization: Beyond Earth, my testing has uncovered a few issues. First and foremost, the problem with the MSI GS60 running at higher than 1080p definitely needs a fix (most likely from NVIDIA's or Intel's drivers, though possibly it's something specific to the laptop that MSI would need to fix, or perhaps it's something with the game itself).

Another issue I encountered (which might be hardware specific) is that the game always drops to a 24Hz refresh rate on my Acer XB280HK LCD; the monitor supports 3840x2160 at 60Hz, and if I change the refresh rate after the game has started everything is fine. When I exit and restart, unfortunately, the game is back to 24Hz even though the settings claim it's at 59Hz.

Of course I'd also love to see the support for Mantle SFR "just work" without requiring any manual tweaking of configuration files. Fixing Mantle support for users with more than two GPUs is a known issue, though it only likely affects a handful of users. Another known issue is that if Windows is set to anything other than 100% scaling, the game basically breaks right now – your mouse cursor doesn't map properly to the screen coordinates so you can't click on buttons or units. And let me just say that a 28" display running at 3840x2160 does not result in particularly legible text (in Windows) for my tired old eyes at 100% scaling.

While I'm here talking about fixes, it would also be great if Firaxis improved the usefulness of the benchmark results. Right now, all it generates is a CSV file with one long line of comma separated values containing the frame rendering times (in milliseconds). That's certainly better than nothing, but I had to create a macro to convert the row into a column, calculate the total time and number of frames (and thereby the average FPS), as well as determining the instantaneous frame times. In some ways it's easier to just use FRAPS to log the performance during the benchmark, but of course FRAPS only works with Direct3D and OpenGL and not Mantle.

Being able to launch the benchmark from a command prompt without Steam complaining about the extra command line parameters (see above dialog) and being able to launch the Mantle version of the game from the command prompt would also be appreciated. The latter is a Firaxis issue, while Steam now appears to complain about any command line parameters on the games I've tested. Basically, if you're going to include a built-in benchmark, generating data in a more useful format and providing tools to automate testing multiple settings helps the people that will actually use the benchmark (i.e. hardware reviewers like me).

Closing Thoughts

We're still early in the release cycle for Civilization: Beyond Earth – in fact, our testing was done with pre-launch code, and there's already talk of a day zero patch to fix a few glitches (e.g. with multi-GPU configurations that have more than two GPUs) – so by no means is this the final word in performance. It's not too surprising to see NVIDIA's GTX 980 taking the single GPU performance crown for Beyond Earth, but additional driver tuning may change things a bit. AMD also had a slide showing the R9 290X 8GB edition leading the GTX 980 in performance (with Mantle at least). Hmmm....

The good news is that if you have any reasonably modern GPU – from the GTX 770 and R9 280 (nee HD 7950) – you should be able to run at 1080p and High or even Ultra quality settings. Lesser GPUs can still handle the game as well, and if you're curious it's possible to get frame rates well over 100 FPS even at 1080p if you drop to lower quality settings.

As for Mantle, it's an interesting option if you have an AMD card. While we're not seeing any huge benefits on our test system, it does typically run 5-10% faster than the DX11 path; this is good but in most cases it's not enough to really make a palpable difference to the end user. The word is that lower spec CPUs like the Intel Core i3 and Pentium Anniversary Edition along with AMD APUs can benefit even more. I'm not sure how many people are actually pairing up slower CPUs with high-end GPUs these days, as $200 CPUs are pretty common for gaming systems, but additional options are never a bad thing.

Mantle does have a much more tangible impact on minimum frame rates, and this is always beneficial, particularly once a game is averaging more than 60FPS. In some cases Mantle was able to improve the minimum frame rates (e.g. on R9 290X) by 40-50%, though that was only at less strenuous settings. Still, even the 5-15% increases in minimum frame rates at higher resolutions are welcome. In some cases, Mantle is the difference between AMD's "equivalent" GPUs trailing versus leading NVIDIA's offerings.

Then there's the subject of Mantle and CrossFire performance, which I've now finished retesting. Mantle generally means more work for the developers when it comes to multi-GPU configurations, but with the appropriate effort the results can be quite interesting. Firaxis has chosen to implement a custom SFR mode for CrossFire on Mantle, though at present the only way to enable SFR is to manually edit your configuration files. (Why!?) The result of CrossFire and SFR in our benchmarks is substantially higher minimum frame rates than the D3D11 AFR rendering mode, and average frame rates are also improved relative to a single GPU.

As far as the game itself is concerned Civilization: Beyond Earth looks like it will prove every bit as addictive as the earlier releases. Now that I've done a ton of benchmarking with Civilization: Beyond Earth, I need to find some time to actually play the game. My birthday is coming up next week, so maybe I can get some free time as a present…. Then again, birthdays only last 24 hours and if past experience is any indication, I might need more like a month to get Civilization Fever out of my system. ;-)