We previewed AMD's CrossFireX technology a couple weeks ago, but today a WHQL drier is available that supports more than two AMD GPUs. In order to do this, AMD employs Vista's Linked Display Adapter (LDA) technology to make the collective GPUs appear to the system as a single virtual device. While this allows for more flexibility in GPU configurations, there are some drawbacks that we will talk about shortly.

The new Catalyst 8.3 is a milestone release for AMD that brings, in addition to Vista LDA support for more than two GPUs, a number of new features. These are (from AMD's documentation):

ATI Hybrid Graphics Support: The ability to use integrated and discrete graphics in hybrid mode to support either power saving (disabling the discrete graphics card) or to enable the integrated and discrete graphics hardware to share the graphics load. This is only available with lower end graphics cards and integrated chipsets using the 3200/3300 series hardware.

Anti-Aliasing enhancements: AA support for all UE3 titles (Unreal Tournament 3, Gears of War, Rainbow Six, etc.) through the control panel. Support for edge detect filters plus Super AA was slated to be in this release but it was pushed back. Tent filters can be enabled with Super AA, but edge detect will be coming soon.

Digital Panel GPU image scaling: The option to enable image scaling for maintaining aspect ratio of a display has been added. Additionally, the GPU itself is now used to scale the image if the check box is ticked. NVIDIA has had this for a while and it's good to see AMD adding support for this.

Advanced Video Quality controls: An edge enhancement slider has been added to adjust the sharpness of the video playback, along with a noise reduction slider. Having variable adjustment for these kinds of features is a very welcome addition and something we've been wanting ever since noise reduction became a hot topic in video decoding.

HydraVision for Vista: Multiple monitor and virtual desktop manager with support for hotkeys and tools to help keep things organized.

Let's take another look at CrossFireX now that we are allowed to test it on Intel hardware and compare it with competing solutions.

Setting it Up …

We Had Issues

Our experience with the new 8.3 drivers and CrossFireX were not seamless. AMD has been working very to try and reproduce the install issues we had with no success so far. Multiple other AnandTech editors had no installation problems, but we feel that it's at least worth mentioning. We can also advise some steps to follow to avoid the issue. This may be an isolated incident, but here are some things you can do to never find out.

It does need to be noted that we only experienced major issues on the Skulltrail platform that we've been using for our graphics benchmarking lately. We've had multiple hardware and drivers installed and uninstalled over the past month or so. These factors certainly could have contributed to the problem and would also make finding the issue tough for AMD. It's always easier to start from a fresh install, but our readers (especially the ones interested in actually running CrossFireX and Skulltrail) will very likely be the ones who install and uninstall hardware and drivers most frequently. We think our usage model for this system could be fairly representative in this case.

The specific problem we ran into was a timeout during attempts to install the driver either through the Catalyst Install Manager or manually. Additionally, once we got the driver installed on one 3870X2, plugging in the second would cause a hard lock when windows tried to automatically find the driver for the second card. We suspect that this occurred because we had previously installed two 3870X2 cards and then removed one before installing the 8.3 catalyst driver. We aren't exactly sure of the cause and neither is AMD, but we can say that the fix for us was removing some non-present including display adapters (along with driver files) in safe mode. After rebooting a few times to let windows do its thing, we installed the driver and all was well.

How to Not Have Issues

To make sure this doesn't happen, when installing CrossFireX, here is what we suggest:

1. Install Vista SP1 (this is easier/better than installing the required hotfixes)
2. Uninstall old drivers first
3. Reboot and let windows install a standard VGA adapter (or cancel if prompted for drivers)
4. Power down and install the second (or additional two or three) cards
5. Boot and let windows finish finding new hardware
6. Reboot and let windows finish finding new hardware (if it needs to)
7. Install Catalyst 8.3 or higher

That should keep you safe, in our experience. It's a serious headache to fix the problems once they start, as simply uninstalling doesn't fix the issue.

How to Fix It (Spring Cleaning)

Just in case, here's what you'd need to do if you find the driver times out on install. Please be cautious with this procedure and do not remove any software devices, as they can be difficult to reinstall without reinstalling the OS.

1. Start up with only one card in the system
2. Uninstall ATI drivers through the control panel -> add/remove programs
3. Reboot into safe mode (press F8 at boot)
4. Open a command window with admin privileges
5. Run these commands from a command prompt:
   • set devmgr_show_nonpresent_devices=1
   • start devmgmt.msc
6. Enable show hidden devices
7. Uninstall all display adapters listed (present or non-present/hidden - they appear slightly faded); make sure to check the box to remove driver files if possible
8. Uninstall any non-present/hidden "PCI standard PCI-to-PCI bridge" devices from system devices. It may also help to uninstall other non-present hardware, but be careful about software and non plug-and-play devices (don't remove them).
9. Reboot normally
10. Let windows finish finding new hardware
11. Power down and plug in all new hardware
12. Boot and let windows finish finding new hardware
13. Reboot and let windows finish finding new hardware (if it needs to)
14. Install Catalyst 8.3 or higher

As we said, it's best to follow the first set of directions as it's quite a bit less involved. If all goes wrong, however, this lengthier repair/install process should help you get yourself out of a hole.

… and Tearing it Down

Whether the problems we saw end up being very isolated problems or not, it's always good to be cautious. Now, let's move on to the part that will absolutely affect everyone who runs CrossFireX at some point: uninstalling the hardware. Until now, if we needed to step down to one card all we needed to do was remove the hardware and everything would be fine. AMD tells us that because CrossFireX uses Windows Vista's Linked Display Adapter (LDA) technique for combining multiple physical graphics cards into one virtual device, this is no longer as simple.

Before removing a card from the system, you MUST disable CrossFireX or uninstall the driver. If you do not disable CrossFireX, the driver will fail to load the next time the system boots. There will be no way (from what we can tell) to disable CrossFireX if the driver doesn't load. Thus, the driver will need to be uninstalled anyway. If you want to keep one card in the system, be sure to disable CrossFireX first or you will have to uninstall and reinstall your driver.

We've asked for more details about why this isn't something AMD can handle in the driver, and this is what we understand so far. After LDA mode is set up, a specific number of physical devices are expected when the driver tries to load. If there is only one card present, this looks the same as if the link failed and thus the driver won't load. AMD has said that this is the expected behavior based on how Vista handles LDA. Our position is that if this is the case, it is a design flaw in Vista that Microsoft needs to address.

After speaking with AMD on this, they have said they would try to make the documentation about how to handle uninstalling hardware properly as prominent as possible. While it's not ideal, thorough documentation of the issues is definitely a good thing to have when potential issues are very likely to arise.

The Test

This time around, we had the opportunity to compare games of our choosing. We did check to see if there were any real performance improvements up from our preview, but we haven't seen any huge changes. Crysis still only shows a performance improvement with two GPUs, and adding a third or fourth can sometimes even degrade performance. That's unfortunate, considering Crysis remains one of the games best able to bring any current GPU setup to its knees.

We decided to tackle this article from the perspective of two-card performance that enables three or four GPUs. This means using the 3870X2 in combination with the 3870 or another 3870X2. There are quite a few permutations that AMD's CrossFireX makes possible. With the way the industry is moving, it seems we won't be able to avoid further investigation into more-than-two GPU configurations. We will definitely continue to investigate this and will report on any interesting findings, but for now we will use two-card setups in order to avoid any concerns about PCI-E slot bandwidth playing a part in our initial performance numbers.

Our goal is to do two things. First, as in our preview, we want to assess the additional benefit gained from going to three or four GPUs. This is obviously relevant, as we want our readers to know what they are getting for their investment. Second, we want to compare the performance of the 3870 in its multiple configurations to other competing solutions from NVIDIA. This will help us to put the performance in a context of relative value to other single and dual card options.

Our test system is Intel's Skulltrail, as it has been in the recent past for graphics articles. We have moved off the AMD Phenom system and will now be able to better compare relative performance values on the best platform to do so. Here is our test setup:

Test Setup
CPU	2x Intel Core 2 Extreme QX9775 @ 3.20GHz
Motherboard	Intel D5400XS (Skulltrail)
Video Cards	ATI Radeon HD 3870 x2 ATI Radeon HD 3870 NVIDIA GeForce 9600 GT 512MB NVIDIA GeForce 8800 Ultra
Video Drivers	Catalyst 8.3 ForceWare 169.28 ForceWare 174.12 (9600 GT only)
Hard Drive	Seagate 7200.9 120GB 8MB 7200RPM
RAM	2xMicron 2GB FB-DIMM DDR2-8800
Operating System	Windows Vista Ultimate 64-bit

Crysis Performance

As one of the most demanding titles currently available, Crysis seems like a great showcase for multi-GPU technologies. We did test performance, but with the current drivers we saw no improvement in performance. However, we also tested prior to the 1.2 patch that just came out on Thursday. We will look at Crysis performance in a future update once we have conducted additional testing.

The Elder Scrolls IV: Oblivion Performance

Version: 1.2.0416 Shivering Isles

Settings: Ultra High Quality settings defaults with Vsync disabled.

Our Oblivion test takes place in the south of the Shivering Isles, running through woods over rolling hills toward a lake. This is a straight-line run that lasts around 20 seconds and uses FRAPS to record framerate. This benchmark is very repeatable, but the first run is the most consistent between cards, so we only run the benchmark once and take that number.

Our tests show that moving beyond two GPUs in a DX9 game that already benefits well from multi-GPU has the potential to make use of three or four GPUs. As resolution goes up, the usefulness of more than two GPUs goes up. Clearly, there is potential for scaling with games that don't make extensive use of features that get in the way of multi-GPU performance. It should also go without saying that this is a technology that should be reserved for the ultra high-end setups. Hopefully illustrating that with resolution scaling will help bring the point home.

Oblivion Perfomance
	1280x1024	1600x1200	1920x1200	2560x1600
NVIDIA GeForce 9600 GT SLI	89.8	70.4	59	33.7
NVIDIA GeForce 8800 Ultra SLI	113.1	96.1	84.2	57.4
NVIDIA GeForce 8800 Ultra	59.5	54.4	46	31.5
NVIDIA GeForce 9600 GT	49	37.5	30.6	19.8
AMD Radeon HD 3870X2 (x 2)	120.3	114	104.1	82.8
AMD Radeon HD 3870X2 + 3870	122.9	107.6	91.5	64.5
AMD Radeon HD 3870X2	104.5	82	71.1	48.6
AMD Radeon HD 3870	61.8	46.8	41.5	25.4

AMD tends to lead in this benchmark normally, and the 8800 Ultra barely leads the Radeon HD 3870, which is normally in competition with the GeForce 9600 GT. In this case, the 8800 Ultra SLI loses to the three GPU AMD setup (3870X2 + 3870), while four GPUs show an even more significant advantage at higher resolutions. 9600 GT SLI, which would normally compete with the 3870X2, doesn't really come through in Oblivion.

Enabling AA and AF reveals one of the major strengths of multi-GPU acceleration: antialiasing comes at less cost. In this case, even at lower resolutions, all four GPUs are able to contribute and performance stays high. While CrossFire and SLI both enable huge levels of AA, at high resolutions a little bit goes a long way.

Oblivion 4xAA/16xAF Performance
	1024x768	1280x1024	1600x1200	1920x1200
NVIDIA GeForce 9600 GT SLI	72.7	55.4	45.3	26.1
NVIDIA GeForce 8800 Ultra SLI	98.1	78.5	68.3	43.3
NVIDIA GeForce 8800 Ultra	53.8	42.4	36.1	23.1
NVIDIA GeForce 9600 GT	36.9	29.2	25	15.6
AMD Radeon HD 3870X2 (x 2)	106.1	93.3	78.1	54.5
AMD Radeon HD 3870X2 + 3870	87.6	73.6	62.9	42.3
AMD Radeon HD 3870X2	64.4	53.6	46.6	31.7
AMD Radeon HD 3870	36.4	30.9	24.4	16.7

While two 3870X2 cards do lead overall, 8800 Ultra SLI does best the three GPU combination of the 3870X2 and the 3870. Oblivion certainly shows that more than two GPUs can contribute to performance, and that AMD was right when they said DX9 games running under Vista would show the most potential for improvement with four GPU systems.

Call of Duty 4: Modern Warfare Performance

Version: 1.4

Settings: All Highest Quality

For this benchmark, we use FRAPS to measure average frame rate during the opening cut-scene of the game. We start FRAPS as soon as the screen clears in the helicopter and we stop it right as the captain grabs his head gear. As we saw in the preview, this game does scale beyond two GPUs, and our tests here show some very interesting results.

Unlike in Oblivion, Call of Duty 4 scales well with three or four GPUs no matter what resolution we are running. We did enable the option to support dual graphics cards, and it is clear that when developers put some effort into explicitly supporting multi-GPU configurations good results can be achieved. This is also interesting in light of the fact that this game is a little more flat when it comes to resolution scaling than other titles.

Call of Duty 4 Performance
	1280x1024	1600x1200	1920x1200	2560x1600
NVIDIA GeForce 9600 GT SLI	127.7	116	104.3	78.8
NVIDIA GeForce 8800 Ultra SLI	145.8	134.1	127.4	102.8
NVIDIA GeForce 8800 Ultra	78.3	75.7	70	57.5
NVIDIA GeForce 9600 GT	66.5	59.4	55	40.8
AMD Radeon HD 3870x2 (x 2)	89.6	86.3	82.6	74.8
AMD Radeon HD 3870x2 + 3870	78.2	74.8	72.9	64.5
AMD Radeon HD 3870 X2	61.5	56	53.8	47.5
AMD Radeon HD 3870	46.4	41.3	38.2	29.6

In spite of the fact that four GPUs scales well on AMD hardware, NVIDIA's 8800 Ultra GPUs in SLI handily outperform the quad solution, as does 9600 GT SLI up to 2560x1600. Call of Duty 4 has certainly favored NVIDIA hardware, as is shown by the fact that a single 8800 Ultra can keep up with three 3870 cards in CrossFireX, and a single 9600 GT performs on par with the 3870X2.

S.T.A.L.K.E.R. Performance

Version: 1.0005

Settings: full dynamic lighting, everything maxed without AA and no grass shadows.

For this test, we walk in a straight line for about 30 seconds and use FRAPS to measure performance. We use the same save game every time and the path doesn't change. Our performance measurements are very consistent between runs. We do two runs and take the second.

S.T.A.L.K.E.R. is a strange one when it comes to performance. At some resolutions, quad-GPUs helps somewhat, but generally speaking the sweet spot is three GPUs. What's really perplexing is that we see quad help more at 1280x1024 than at 1600x1200 and 1920x1200. The performance benefits of three GPUs is also a bit erratic; though clearly at 2560x1600 the difference is noticeable, that resolution also demonstrates better scaling from two to three cards than from one to two cards - not something we would normally expect to see.

What we can say for certain is that the latest version of STALKER is not performing as well as we would expect in a variety of ways. It's not unusual to see multi-GPU technologies run into CPU limitations at lower resolutions and offer better scaling at higher resolutions, but that's not what we're seeing. Instead, we have our lowest and highest resolutions benefiting more from CrossFire (and CrossFireX) than our middle resolutions. We are looking into the matter more to see if we can determine what is actually happening; this could be something caused by the Skulltrail platform, the 1.0005 patch, the AMD drivers… or more likely than not all of these things working together.

CrossFireX has the potential to add value at all resolutions in S.T.A.L.K.E.R., but at present there are some anomalies (Ed: the kind they pay good money for around Chernobyl, we hear). Due to the nature of performance, gameplay and location in the game could make for huge variations in performance. Looking at the sky will push frame rate way up into the hundreds, so keeping more sky in your view than ground will improve your framerate. Thus, people who normally look higher while running around will experience better performance than those who look lower.

It's very hard with data like this to accurately assess the value of CrossFireX for this particular game. Based on the data we do have, it again seems that three GPUs is the sweet spot rather than four, in spite of the fact that four GPUs can help at both higher and lower resolutions.

S.T.A.L.K.E.R. Performance
	1280x1024	1600x1200	1920x1200	2560x1600
NVIDIA GeForce 9600 GT SLI	94.3	72.8	62.6	39.9
NVIDIA GeForce 8800 Ultra SLI	110.3	94	85.2	61.6
NVIDIA GeForce 8800 Ultra	62.3	53.5	48.8	33.3
NVIDIA GeForce 9600 GT	51	38.7	33.3	21
AMD Radeon HD 3870X2 (x 2)	111	84.3	67.3	39.2
AMD Radeon HD 3870X2 + 3870	103.5	83.4	68.2	36.9
AMD Radeon HD 3870X2	86.3	68.5	59.6	24.1
AMD Radeon HD 3870	43.7	34.3	29.5	17.5

Performance makes sense in general, but it is important to note the sharp relative drop of the 3870X2 at 2560x1600. This is a case where the 3870X2 doesn't appear to have the power to reach the next performance plateau, and adding in another graphics card really made the difference. That, or it's just another driver glitch that needs some ironing.

World in Conflict Performance

Version: 1.005

Settings: Medium quality plus Heat Haze, Debris Physics, and DX10

We tested this game using the built-in benchmark feature of the game. In our experience, this does a good job of testing the different graphical scenarios that can be encountered in the game.

World in Conflict, from the data, really looks like we had Vsync enabled, the system was severely CPU limited, or the framerate cap was on. However, this was not the case. These data points had minimum and maximum framerates extending from ~30 to ~160 fps, and there appears to be another factor in the resulting data looking so flat between resolutions.

The 9600 GT SLI was able to break past this barrier and post average framerates higher than 60fps. The only major difference is that that we had to use a different driver just for the 9600. Given what we experienced in our recent Dell XPS M1730 article, the 170 series drivers help significantly in World in Conflict and Crysis; unfortunately, no official beta or other driver with 8800 Ultra support is available. We are investigating further and waiting for driver updates from NVIDIA.

It seems clear that if there is some limit on performance scaling here with our test platform. We expect future driver updates to significantly help both SLI and CrossFireX.

Pushing resolution higher is the way to get more value out of multi-GPU here. Increasing settings may help, and we will go back and look at higher settings with these configurations in the future. Running at Very High is still not a viable option, but there is room for customization to end up with a workable stress test for current high-end systems.

World in Conflict Performance
	1280x1024	1600x1200	1920x1200	2560x1600
NVIDIA GeForce 9600 GT SLI	73	68	63	51
NVIDIA GeForce 8800 Ultra SLI	58	56	54	52
NVIDIA GeForce 8800 Ultra	58	56	53	44
NVIDIA GeForce 9600 GT	60	50	44	30
AMD Radeon HD 3870X2 (x 2)	63	60	60	58
AMD Radeon HD 3870X2 + 3870	63	60	60	57
AMD Radeon HD 3870X2	63	60	59	52
AMD Radeon HD 3870	57	51	47	34

With all the data compressed under 60fps, it is hard to get a clear understanding of what's going on. In the interest of reporting what we actually saw, the above chart shows our results.

Final Words

Is multi-GPU really the future? Maybe. Advanced rendering techniques will increasingly require the use of persistent dynamic objects created on the fly, however, and this will likely continue to get in the way of scaling beyond two GPUs unless something fundamental in GPU design changes or graphics memory architecture is expanded to allow for easier sharing of the workload. We can also take away from all this that it is a much better idea to have a smaller number of high powered GPUs doing the work, as the return on investment for dual GPUs will be higher than adding more of a lesser card to a system.

Then of course there's the ubiquitous driver questions. It is one thing to make a driver that works with the current graphics libraries; it's another to make it work optimally in all situations. Taking a (nearly) fully optimized driver and extending support to two GPUs adds another level of complexity, and we're still seeing numerous titles released that don't have properly working SLI and/or CrossFire support until after a driver update (and sometimes a game patch as well). The situation with two GPUs is actually quite good these days, particularly if you're willing to search forums and other technical sites for information on tweaks to help performance prior to the inevitable release of new drivers. Unfortunately, CrossFireX takes a step back to the earlier days of CrossFire, and there are numerous titles where scaling is either nonexistent or at least much lower than we would expect. As we've seen in the results today, drivers matter - there's no other aspect of graphics as likely to help (or hurt) performance.

Drivers aren't the only concern at present, of course. The installation process for CrossFireX is far more involved - with more potential conflicts - than any other graphics solution we've used recently. Some of the issues we experienced may be related to the Skulltrail platform rather than specifically stemming from CrossFireX. That's something we will only be able to understand better with additional time. If you want our honest opinion, right now CrossFireX is sitting way out on the razor-sharp bleeding edge. When it works, the results can be very impressive. When it doesn't, the headaches, BSODs, uninstalling, reinstalling, and hacking that may be required is enough to make the best geeks cower in fear.

For now, the real benefit of multi-GPU is still going to be in achieving higher levels of AA with little to no performance decrease. We've already looked at ATI's Super AA, and there was a feature that didn't quite make it into Catalyst 8.3 that we'll want to look at when it arrives. In an upcoming driver, we will be able to enable edge detection with Super AA, which should lead to some insanely high antialiasing. While we tend to lean towards higher resolutions as a better option than higher AA modes (and suggest 4xAA with transparency AA as a good target), we will certainly look at this when it becomes available.

We aren't done with our investigation of multi-GPU technology and the impact of more than two GPUs on a system. There are issues we definitely need to go back and investigate more. For instance, does the ability to render four frames ahead significantly affect input lag? If multi-GPU technology really is the future of high-end graphics, we need to assess the current shortcomings so that we can understand what direction to take to get us there.