Rage Against the (Benchmark) Machine

Rage came out a little over a week ago, and in the aftermath we’ve discovered some interesting pieces of information. I thought I’d chime in with some thoughts on the game itself, a look at various image quality settings, and a discussion of benchmarks with the title and why they’re virtually meaningless. First, let’s start with the game itself.

I know a lot of people have had issues getting Rage to run well, and that has certainly colored impressions of the game. Lucky for me, I’m not one of those unfortunate souls: with the NVIDIA 285.38 beta driver released at nearly the same time as the game, I updated drivers and never encountered any severe issues with stability or playability. I also tried Rage on an AMD HD 6950 system, which generally worked okay, but I did notice some texture flickering/corruption going on. I’m certainly running higher spec hardware than most people (i7-965 Extreme running at 3.65GHz with 12GB DDR3-1333 and a GTX 580/HD 6950 2GB), but with my gaming systems the experience has been remarkably stable and playable. What about the game play?

Here’s where things get a little shaky. First impressions are okay, but by the time you’re running around on an ATV in a wasteland environment 5 minutes into the game, it’s impossible to ignore comparisons with Borderlands. Both games are FPS titles with vehicular elements, set in a predominantly desert environment. Borderlands takes place on a different planet with some muddled background information and Rage is set in the not-too-distant future after an asteroid smashes into the Earth and scatters some new mineral that appears to cause mutations everywhere. Whatever the back story, however, both settings feel a lot like Mad Max’s post-apocalypse world of gangs, bandits, and vehicular mayhem. Then again, I loved the original Wasteland as an early teen (on my Commodore 64 no less!), all the Fallout games (including the oft-panned Fallout: Tactics), and the Road Warrior movies, so I’m okay with revisiting the wasteland.

Delving deeper, Borderlands had a nice cell-shaded aesthetic with a rudimentary storyline, and most interactions with “NPCs” as such came in the way of talking robots, job boards, and vending machines. This may come as a bit of a shock, but I actually found the Rage storyline and NPCs to be superior to Borderlands. That’s right: the same people that brought us Wolfenstein, Doom, and Quake actually put some effort into the story this time! That doesn’t mean the story isn’t a bit cliché, but at least there’s something here other than a pure shooter. I think the best comparison would be that the story is presented much like the original Diablo (or maybe Diablo 2) in that you have NPCs in towns who serve as little more than quest givers and background providers, but they don’t go anywhere and your actions typically don’t affect the world around you other than ridding areas of bad guys. The story is also strictly linear with no chance for role playing; you’re here to save the wasteland from the power-mongering Authority, helping others along the way. Some of the side quests (e.g. from the job boards) are optional, but you either do the job(s) or you don’t.

So the story is okay if not great, but what about the game? I wasted more than a few days (weeks) playing Quake and similar titles in my college days, but I’ll be honest: I’m pretty much done with multiplayer gaming now. I haven’t even tried it in Rage, but unless it’s quite different from the single-player experience it’s not going to be the hyper-speed twitch shooter that Quake was. If you’re looking for a multiplayer title to test your mettle, I’ll defer to other opinions; as a single-player experience, though, this is a pretty major departure from previous id titles.

Your character moves at moderate speed with the option to sprint for a limited time, at which point you start panting and resort to regular speed. You have an (unlimited as far as I can tell) inventory along with various items to use, and you can sell and buy ammo, items, and other stuff at several shops scattered around the game world. There are likewise various components and ingredients strewn throughout the game that can be combined (engineered) into useful items. Yes, there were similar pseudo-crafting elements to Borderlands as well, though here most of the items are either ammo modifications or one-use items instead of permanent character/weapon mods. It’s a strange mix, really, where id Software has created a title that feels more like Deus Ex or System Shock in how you manage your equipment than Quake or Doom—but don’t let that comparison make you think the story or freedom to play as you want is up to the level of the DX/SS games.

The driving sections work well enough, providing a nice change of pace from walking around, but the races and vehicles don’t really do much other than provide you with something to do other than walking. They give you cash to upgrade your weapons/vehicles as well. Elsewhere, Rage is pretty standard shooter fare: there are ten weapons comprising the usual assortment of fists, pistol, shotguns, machine guns, sniper rifle, crossbow, and rocket launcher, and a “futuristic” weapon at the very end (a plasma gun/BFG). Most weapons also have a variety of ammunition available; typically these just do more damage, but one causes electrocution (shoot at water to kill multiple enemies), another lets you mind-control the target for a bit before they detonate/die, and others add explosion/penetration to your ammo. You can also lay out turrets and mobile turrets, though it’s hardly necessary (at least on normal difficulty). Overall, the variety of options for weapons and ammo is good, though my go-to weapons for most of the game were the sniper rifle and shotgun, with the pistol working well for the first half or so. I also hoarded my special ammo for far too long, thinking I’d need it later; hint: you won’t (again on normal difficulty).

I’m going to keep this short (too late?) and just give my overall impression of the game before we get to some talk about the technical aspects of the game. Rage is a fun distraction, and it looks quite nice overall. I grew up in the deserts of Utah and spent plenty of time out near the Grand Canyon, and Rage does an excellent job of capturing the feel of the place. That also means that it’s a very brown game, and the game itself even acknowledges this in their Quayola Quaons easter egg (a “very rare collection of Quayola’s Brown Spectrum Quayons”). The texturing may not always be as crisp as I’d like, but if there’s any straight repeating of textures in the environments it’s very hard to spot. Shadows on the other hand are practically absent; the environments are all pre-calculated lighting, with only characters and vehicles casting 1-light-source shadows. Basically, id traded dynamic lighting for performance, and it definitely shows on high-end PCs where numerous games look better.

My take is that Rage looks good if not exceptional, and at least in my testing it runs well. If you’re looking for a new shooter to while away ~15 hours, this will suffice, but be prepared for the equivalent of a summer action blockbuster. As a movie-type experience, I’d rate Rage at 2.5 stars; it’s fun and exciting, but you’re not going to have an emotional reaction or learn something new and insightful. If we’re going with letter grades, it’s somewhere in the B or B- range. Fans of Borderlands will also likely find something to enjoy here, though they might also be struck with a sense of déjà vu—did the two developers branch off from the same design document a couple years back? Rage makes me want to go back and replay Borderlands (a game I never did quite finish, despite playing for over 20 hours), just to see if it’s any better; right now, they’re basically a tie in my book. Also, the ending of Rage felt very anti-climactic; there were a couple big boss battles earlier and I expected one at the end, but it never came; weird. Anyway, that’s one man’s opinion; take it for what it’s worth.

Technical Discussion

The bigger news with Rage is that this is id’s launch title to demonstrate what their id Tech 5 engine can do. It’s also the first major engine in a long while to use OpenGL as the core rendering API, which makes it doubly interesting for us to investigate as a benchmark. And here’s where things get really weird, as id and John Carmack have basically turned the whole gaming performance question on its head. Instead of fixed quality and variable performance, Rage shoots for fixed performance and variable quality. This is perhaps the biggest issue people are going to have with the game, especially if they’re hoping to be blown away by id’s latest graphical tour de force.

Running on my gaming system (if you missed it earlier, it’s an i7-965X @ 3.6GHz, 12GB RAM, GTX 580 graphics), I get a near-constant 60FPS, even at 2560x1600 with 8xAA. But there’s the rub: I don’t ever get more than 60FPS, and certain areas look pretty blurry no matter what I do. The original version of the game offered almost no options other than resolution and antialiasing, while the latest patch has opened things up a bit by adding texture cache and anisotropic filtering settings—these can be set to either Small/Low (default pre-patch) or Large/High. If you were hoping for a major change in image quality, however, post-patch there’s still plenty going on that limits the overall quality. For one, even with 21GB of disk space, id’s megatexturing may provide near-unique textures for the game world but many of the textures are still low resolution. Antialiasing is also a bit odd, as it appears have very little effect on performance (up to a certain point); the most demanding games choke at 2560x1600 4xAA, even with a GTX 580, but Rage chugs along happily with 8xAA. (16xAA on the other hand cuts frame rates almost in half.)

The net result is that both before and after the latest patch, people have been searching for ways to make Rage look better/sharper, with marginal success. I grabbed one of the custom configurations listed on the Steam forums to see if that helped at all. There appears to be a slight tweak in anisotropic filtering, but that’s about it. [Edit: removed link as the custom config appears mostly worthless—see updates.] I put together a gallery of several game locations using my native 2560x1600 resolution with 8xAA, at the default Small/Low settings (for texturing/filtering), at Large/High, and using the custom configuration (Large/High with additional tweaks). These are high quality JPEG files that are each ~1.5MB, but I have the original 5MB PNG files available if anyone wants them.

You can see that post-patch, the difference between the custom configuration and the in-game Large/High settings is negligible at best, while the pre-patch (default) Small/Low settings have some obvious blurriness in some locations. Dead City in particular looked horribly blurred before the patch; I started playing Rage last week, and I didn’t notice much in the way of texture blurriness until I hit Dead City, at which point I started looking for tweaks to improve quality. It looks better now, but there are still a lot of textures that feel like they need to be higher resolution/quality.

Something else worth discussing while we’re on the subject is Rage’s texture compression format. S3TC (also called DXTC) is the standard compressed texture format, first introduced in the late 90s.  S3TC/DXTC achieves a constant 4:1 or 6:1 compression ratio of textures. John Carmack has stated that all of the uncompressed textures in Rage occupy around 1TB of space, so obviously that’s not something they could ship/stream to customers, as even with a 6:1 compression ratio they’d still be looking at 170GB of textures. In order to get the final texture content down to a manageable 16GB or so, Rage uses the HD Photo/JPEG XR format to store their textures. The JPEG XR content then gets transcoded on-the-fly into DXTC, which is used for texturing the game world.

The transcoding process is one area where NVIDIA gets to play their CUDA card once more. When Anand benchmarked the new AMD FX-8150, he ran the CPU transcoding routine in Rage as one of numerous tests. I tried the same command post-patch, and with or without CUDA transcoding my system reported a time of 0.00 seconds (even with one thread), so that appears to be broken now as well. Anyway, I’d assume that a GTX 580 will transcode textures faster than any current CPU, but just how much faster I can’t say. AMD graphics on the other hand will currently have to rely on the CPU for transcoding.

Update: Sorry, I didn't realize that you had to have a game running rather than just using vt_benchmark at the main menu. Bear in mind that I'm using a different location than Anand used in his FX-8150 review; my save is in Dead City, which tends to be one of the more taxing areas. I'm using two different machines as a point of reference, one a quad-core (plus Hyper-Threading) 3.65GHz i7-965 and the other a quad-core i7-2630QM. I've also got results with and without CUDA, since both systems are equipped with NVIDIA GPUs. Here's the result, which admittedly isn't much:

This is using "vt_benchmark 8" and reporting the best score, but regardless of the number of threads it's pretty clear that CUDA is able to help speed up the image transcoding process. How much this actually affects gameplay isn't so clear, as new textures are likely transcoded in small bursts once the initial level load is complete. It's also worth pointing out that the GPU transcoding looks like it would be of more benefit with slower CPUs, as my desktop realized a 41% improvement while the lower clocked notebook (even with a slower GPU) realized a 52% improvement. I also tested the GTX 580 and GTX 560M with and without CUDA transcoding and didn’t notice a difference in perforamnce, but I don’t have any empirical data. That brings us to the final topic.

Initial Rage Performance Investigation

All of the technical information is fine, but we still come back to the question of performance. Earlier we showed the Low/High/Custom screenshots, and while the Low mode is definitely blurrier, it should at least come with higher performance, right? Well, perhaps on some hardware configurations that would be true, but on my particular system the benchmark results are really easy: put 60 FPS across the chart and you’re done. Yes, even at 2560x1600 with 8xAA, Rage hits the 60FPS barrier and refuses to go any faster. Not so useful, but I’m running a GTX 580 and that’s currently the fastest single-GPU solution on the market, right? I decided to test a much lower end setup, so I grabbed the ASUS G74SX I recently reviewed to give that a look and ran it through a similar set of tests, and I used a second system similar to my gaming rig but with an HD 6950 2GB.

Performance on the G74SX does fluctuate a bit (particularly if I enable VSYNC), but there’s still some obvious image quality scaling going on. (You see textures popping in at times, but mostly it only happens when you first enter a level.) At the Low settings and 1080p, performance checks in at 58FPS (just shy of the 60FPS that VSYNC would want), and bumping up the in-game settings to Large/High didn’t change that much—it dropped to 57FPS, which is margin of error for FRAPS. Using the custom configuration likewise didn’t affect performance. In short, the hardware in the G74SX is running at very close to the target 60FPS regardless of what settings I select—even enabling 8xAA doesn’t change the performance much, although at 4xAA the notebook does lock in at a solid 60FPS. The only way to get performance to tank is to enable 16xAA, which drops frame rates to 24FPS. All of this is at 1080p, however, as I don’t have a way to test higher resolutions with a notebook.

So two down and no interesting information unearthed in regards to performance. What about AMD? I’ve got an HD 6950 2GB with an overclocked i7-920 (3.3GHz) and 12GB RAM, very similar to my main gaming system. I connected that to a 30” display and tested at 1080p and 2560x1600. AMD’s hardware is apparently limited to 8xAA, and performance is lower than on the GTX 580 at these punishing settings. Low/High settings however still fail to make a difference, with average frame rate with 8xAA being around 42 FPS; dropping to 4xAA brings the HD 6950 back to the 60FPS cap, and likewise 1080p at 8xAA hits 60FPS.

The initial release of Rage had a configuration option “com_syncToTime” that you could set to -1 to remove the frame rate cap, but doing so would apparently speed up everything (e.g. so at 120FPS everything would move twice as fast). I never tested this before the Steam update came out, and post-patch I can’t get com_syncToTime to work. So far, I’ve tried “cvaradd com_synctotime -1;” in my rageConfig.cfg file, which did nothing. I’ve also tried using the same command from the in-game console, and I’ve tried it as a command-line argument. If anyone has input, though, send it along and I’ll be happy to try again. Until we can get rid of the 60FPS cap and dynamic image quality, however, benchmarking Rage is essentially meaningless.

Update #2: One reader suggested that our custom settings were mostly useless (ignored by the game), which is very likely, so we removed the link. He also suggested we force 16k textures as that would improve both quality and potentially make the game into a more useful benchmark. Unfortunately, neither actually worked out. 16k texturing did not noticeably improve image quality or reduce performance in most areas; the only way it improves quality is if you're in a scene where the total number of textures in use exceeds the 8k limit, which only happens in large outdoor vistas. As a downside, setting the game to 16k texture cache did make it more unstable, particularly at certain resolution/AA combinations. 2560x1600 8xAA with 16k textures on a GTX 580 crashes, for example, and 2560x1600 4xAA with 16k textures can take 5-20 seconds to load level transitions (when the game doesn't crash). 2GB cards might (emphasis: might) fare better, but that doesn't change the fact that most areas show no difference between the 8k and 16k texture cache.

A second suggestion was that we force 16xAA to improve quality and put more of a load on the GPU, thus making the game run below 60FPS and allowing us to do more of a benchmark comparison. This misses the point that a game benchmark needs to be at meaningful settings; otherwise we can just go back to using 3DMarks. The other problem is that once we move beyond 8xAA, antialiasing starts to degrade image quality by making everything blurry. AMD cards limit you to 8xAA, but you can force NVIDIA up to 32xAA through the config file. Here's another gallery of comparison shots, showing both forced texture size and antialiasing comparisons, all on the GTX 580. (The image corruption at 16xAA appears to be a problem with FRAPS, as in-game we didn't notice the rendering errors.)

AMD vs. NVIDIA Image Quality Comparison

There’s one final item we can quickly discuss, and that’s image quality comparisons between different hardware. I grabbed screenshots from five locations on the same three test systems, with all of them locking in at 60FPS at 1080p 4xAA. I skipped the custom configuration file and just went with the in-game settings, testing each system at Small/Low and Large/High for Texture Cache and Anisotropic Filtering. Here’s a second gallery, this time with 1080p files. As before, I’ve saved all of the images as high quality JPEGs, as uploading (and downloading) 3MB images takes quite a while, but if there’s a desire for the original PNGs I’ll see about uploading them as well.

Looking at the still images, there’s very little if any difference between most of the screenshots. In motion, however, right now I notice some quality differences on the AMD hardware, with a few areas that have flickering textures and some other minor distortions. There’s nothing bad enough to make the game unplayable, but browsing the forums it does sound as though more users are having issues with Rage on AMD hardware right now. Long-term, I expect all of this will get worked out, but at least for the next month or two you may encounter some driver bugs or other issues with Rage. Caveat emptor.

Wrap Up

I’m not going to pretend to be a professional game reviewer, just like I’m not a movie critic. I’ve enjoyed both movies and games that have been panned by the critics; other games and movies that rated highly have been seriously dull for me. What I can say is that I enjoyed Rage enough to finish the single-player campaign, and while the story isn’t great, the graphics are still quite impressive. When I look at the steady 60FPS, I can certainly understand id’s rationale: take the hardware configuration confusion out of the equation and just let people play your game at a silky smooth frame rate. The megatexturing is also quite nice, as the lack of repeated textures is definitely noticeable. What really impresses me, however, is id's ability to have a game look this good and still pull 60FPS at maximum quality settings. As good as that might be for gamers, however, in the current state it makes Rage essentially useless as a benchmark.

We tried to do a few comparisons between NVIDIA and AMD graphics solutions, but the game engine (and drivers) feel a bit too raw for that right now. In another month or two, AMD, NVIDIA, and id Software will likely have all the bugs worked out and everyone can get around to playing the game as intended. Perhaps we’ll also get a useable benchmark mode where we can at least see how the graphics solutions compare. Until then, Rage remains an impressive technology demonstration and a game that some people will certainly enjoy; it’s just not a good benchmark for us right now.