"I have no idea; it is an industry standard server after all." That is what we heard from two PR people, each representing a different large server manufacturer at CeBIT, when we asked them to elaborate on the differences between their servers and the competition. There is no better way to get sales people annoyed than asking them to differentiate their x86 servers on technical merits. A very popular philosophy among sales and even IT people is that the services and support make the difference; the hardware itself is merely a commodity - it does not matter which server you chose.

Seasoned system administrators know that the last sentence is an exaggeration. While there is little doubt that fast and capable support is important, it should not be the main selling point. For fragile devices such as notebooks that break down easily, yes, support can be critical. However, for sturdy servers, one of the selling points should be that you hardly ever need to call up the support lines. In addition, while the software services running on those servers are indeed the ultimate goal, making sure your software project starts with the best hardware foundation is critical.

The x86 server market really needs some fresh new ideas, and especially the four socket market could use a wind of change. Granted, IBM's System x3950 M2 and x3850 M2 are very innovative thanks to their ScaleXpander technology that allows you to combine several servers, but that is a solution for the ultra high-end of the x86 market. No, somebody needs to stir up this market - which is almost completely under the control of HP and IBM - with fresh ideas that are less costly. Number one and two in the server market are well entrenched: they control more than 70% of this market. Attacking those two leaders is therefore a daunting task. Dell believes in all-out assault on the HP/IBM double bastion with aggressive pricing. Sun on the other hand decided to undermine the ramparts with "think different" engineering.

The Sun Fire x4500 is one example, which can stack up to 24 TB of storage space in a 4U rack server. The future Sun Fire x4600 is the next (but again expensive) example, delivering 32 cores in a 4U rack space. However, the X4450 is in our humble opinion the best example: it has the promise of saving a lot of costs and is affordable for many enterprises. And the signs are good: the strategy of "Think Different" servers seems to have paid off so far: Sun has gone from the number 9 in the x86 server market in 2005 to the number 5 spot in 2007. In 2007, x86 was about 11% of Sun's business and 32% of Sun server shipments.

In this article we will look at one of Sun's most remarkable recent products, the Sun Fire X4450. The X4450 is a 2U server that promises to offer the same features and performance as the 4U servers from Dell, IBM, and HP. That sounds too good to be true, so we searched for the catch and compared with Intel's S7000FC4UR and Dell's PowerEdge R900. Especially the comparison with the PowerEdge R900 proved to be very interesting: both servers are perfect examples of the fact that Dell and Sun have a fundamentally different philosophy.

The "4 Socket Reference Design"

The 2U Sun Fire X4450 is the only 2U server on the market that features the Quad Socket Intel Xeon MP 73xx platform. Even four mighty 130W Xeon MP "Tigerton" X7350 2.93GHz can find a home in this slick Sun server. In case you don't remember, "Tigerton" is basically the quad socket capable version of the 65 nm "Clovertown".

Only the Supermicro 8015C-TB does better and fits the entire Xeon MP platform in a 1U server. However, this server clearly targets a completely different market - the HPC clustering market. The Supermicro only has one PCIe slot, 24 DIMM slots, and features one PSU; the Sun Fire X4450 has two fully redundant PSUs, six PCIe slots, and 32 DIMM slots. It is unlikely that the Supermicro server would be on the shopping list of someone searching for a virtualized consolidation server or heavy database server, since PSU redundancy is usually a requirement for those markets.

To fully appreciate the work that the Sun engineers have done, it is interesting to look at the Intel "Xeon Tigerton MP Reference Server", also called the Intel S7000FC4UR. It is a massive 4U server, with two 1570W 1+1 hot pluggable power supplies. In the front we find two pairs of 120mm fans that draw the cold air in. Each pair of fans starts to spin much faster if you remove the other pair.

In the back four 80mm fans try to blow the hot air out of this sixteen core beast.

Seven total PCI Express slots are available: four x8 (two slots hot-plug) and three x4.

Notice that the 7300 Northbridge needs quite a bit of cooling. The 7300 chipset can dissipate between 33 (idle) and 47W^[1] and therefore needs a large heatsink. At the left and the right, you notice two black memory boards. Each of them can contain eight DIMMs, as you can see below.

Let us compare this Intel server with the Sun server.

The X4450 Up Close and Personal

When we look at the Sun Fire X4450, we find a few differences from the "reference" design. Six pairs of fans are easily accessible at the front of the server. If one fails, you just have to open the first part of the top cover door and replace the faulty fan. This is an advantage compared to other similar servers and the Intel S7000FC4UR for two reasons. The first reason is that all the fans are in one place - no searching is necessary. The second reason why Sun's approach is better is that if a rear fan fails on other servers, you will have to pull out the entire server to replace those fans in the back. You can imagine that this might be difficult if your server is located relatively high in a rack. Thumbs up for the Sun machine: it offers better serviceability.

The top cover is split into two parts. If you remove the main cover (the part in the back), the system will automatically enter standby mode. Sun makes use of two 1050W PSUs, which could save power compared to the 1570W power supplies found in the Intel S7000FC4UR. (Efficiency and other factors will of course play a role.)

The most remarkable part of the X4450 design is that the engineers managed to put a memory board on top of the CPUs and the Northbridge heatsink. The complete memory basket is easy to remove and does not require any tools as you can see below.

All servers claim to be ready for virtualization, but Sun and Dell have to have thought this through just a little better. Integrating four Gigabit Ethernet ports instead of two is a very smart move. For a typical ESX server you need:

• two ports (so you can offer your virtual servers a total of 2Gb/s) for your virtual switch/network
  
• one port for the console
  
• at least one port to connect to your NAS or iSCSI SAN

In some virtualization scenarios, you'll use up to eight (!) gigabit ports:

• Two ports in fail-over for the console
  
• Two ports in fail-over for Vmotion, giving you the ability to easily move your virtual machines to other physical servers
  
• Two teamed ports to give decent bandwidth (2Gb/s) to all your VMs
  
• Two teamed ports for access to the iSCSI SAN

We admit eight ports might be a bit luxurious, but frankly there is little you can do with only two Ethernet ports these days.

Three risers allow you to plug in six different PCIe cards, which should be more than enough upgrade space for most applications. Let's see how Sun compares to its competitors:

The HP server is the most expandable, but still forces almost every customer to buy an additional NIC. It's not a big deal, but it's still a small disadvantage. IBM is clearly counting on the fact that you'll use a SAN storage with this server. This is also only a small disadvantage as most users will combine this kind of server with a SAN. However, if you are looking for one big database server, the IBM will not fit the bill. Notice how the Sun Fire server has the PSU with the lowest spec, a first indicator of power consumption.

PowerEdge R900 and S7000FC4UR: birds of a feather

The Dell PowerEdge got our attention, as the configuration specifications were remarkably similar to the Intel S7000FC4UR server we had in the lab. The power supplies seem to have the same electrical characteristics as the ones found in the S7000FC4UR. When we started to compare, we found more and more similarities. Just look at the rear of each server.

When you look inside the Dell PowerEdge, it is clear that the Dell PowerEdge R900 and Intel S7000FC4UR are siblings. Just compare the image below with the photo that we showed you of the inside of the Intel server a few pages back.

The way the CPUs are cooled, the layout of the board, the two pairs of rear fans, the memory modules... everything looks exactly the same. Only the arrangement at the front of the server is slightly different: hard disks are placed vertically instead of horizontally and the RAID card is positioned differently. The Dell PowerEdge is a slightly improved version of the slightly older Intel S7000FC4UR, so we are 99% sure that performance and power consumption will be very similar too.

That is not really a surprise, as this approach to server design is typical for Dell. Dell tries to keep costs as low as possible and reusing (part of) Intel's engineering is a smart way to do that. Dell's bread and butter is the mass market of dual socket servers, which is good for about 75% of x86 server revenue. To Dell it makes less sense to invest heavy engineering efforts in the much smaller four socket market (about 14% of the revenue of the server market).

Power Consumption

Power consumption is one of the factors that will determine the cost of operation. To measure power consumption, we made sure that both systems use the same configuration:

• Four Xeon MP E7340 (quad-core, 2.4GHz, 2x 4MB L2)
• 32GB (8x 4GB) of 667MHz FBDIMM
• Two 2.5 inch 73GB 10.000 RPM SAS hard disks
• A 1+1 Redundant PSU configuration

As we did not have the HP ProLiant 585 G5 in the labs, we also use the power calculators of each manufacturer (Sun, Dell and HP) to get an idea of the power consumption. We didn't find a similar power calculator from IBM.

Power calculator numbers are of course not very accurate. Some manufacturers may use a large safety margin, other might publish low numbers as a way of convincing customers. It gives us a rough idea.

Let us see how the "power calculator" numbers compare with the reality in our lab. We measured peak power consumption running Specjbb2005 when the CPUs were at 85% and 100%.

Sun’s power consumption is pretty low with these CPUs, but the Dell numbers we got from Dell’s capacity planner seem to be very high if you consider that they should be close to the Intel server. Sun's power calculator numbers seem to be accurate, while Dell's numbers have a high safety margin.

The fans seem to be one reason the difference between the 2U Sun machine and 4U Intel machine is higher at idle. Once the CPUs are working at full load, the fans inside the Sun have to work harder to get the heat out of the cramped 2U box. We measured up to 715W at peak if you use four Xeon MP x7350 (TDP: 130W) instead of four Xeon MP E7340 (TDP:80W). The fans were screaming quite loud, so our advice is to use 80W TDP CPUs in this server - your fans will last a lot longer.

Performance

Extensive performance measurements are only interesting if they really test the platform and not just the CPUs. Sun, HP, IBM, and Dell all use the same Intel CPUs and the only way they can influence the performance of the platform is by designing either the motherboard or tuning it as well as possible. IBM is the only one of these four that takes it further: it designs the chipset too. With the exception of the IBM servers, we shouldn't expect big differences in real world performance except for applications that really stress the memory subsystem. To magnify the small expected performance differences, we used our multi-threaded, 64-bit Linux Stream binary (courtesy of Alf Birger Rustad) based on v2.4 of Pathscale's C Compiler, compiled with the following switches:

-Ofast -lm -static -mp

We tested with 16 threads on a SUSE Linux SLES 10 SP1, 64 bit.

Both Sun and Intel offer a "high bandwidth" option in the BIOS. However, it seems to decrease bandwidth instead of raising it in our testing, so we left it off (default setting).

Despite the fact that the BIOS settings were identical (prefetcher off, high bandwidth off, adjacent Cache line prefetch off) and the fact that we are using exactly the same software configuration, the Sun x4450 outperforms the Intel board by more than 10% in this memory bandwidth test. The question is if this is reflected in a more real world test which is known to be limited by memory bandwidth like SPECjbb2005. We used almost the exact same setup as we described here in great detail. The only changes were:

• We use the BEA JRockit R27.5.0-5 Linux x86_64)
• We do not use the SUN JVM as Sun is about to release a new JVM version which is significantly faster than the current one. This new JVM (1.6.0_06-prelease) allows Sun to claim the crown in SPECjbb2005 as you can see here.
• We changed one of our java parameters: -Xms1800m -Xmx1800m -Xns1300m -XXaggressive -XXlazyunlocking -Xgc:genpar -XXtlasize:min=4k,preferred=512k -XXcallprofiling

We use Xns = 1.3 GB instead of 1.5 GB (used in previous reviews) as this leaves more room for the old space (Xms minus Xns) which avoid excessive Garbage Collecting. Our thanks goes to Henrik Stahl for pointing this out.

To our surprise, Sun’s bandwidth advantage was also reflected in our SPECjbb2005 test: the Sun machine is a full 10% faster. We will explore the performance capabilities of this kind of server in the next article and try to find out more. But this performance advantage was quite unexpected.

The Matter of TCO

TCO is probably the most talked about three-letter word in the IT industry. It is important we get an idea how much impact the hardware decision makes in the grand scheme of Total Cost of Operation/Ownership. In the first years of IT, the focus was mostly on TCA (Total Cost of Acquisition). However, the past years we have seen reports that are downplaying the initial buying costs. Since such papers were mostly sponsored by vendors of very expensive servers, we don't trust them either.

We decided to use numbers from two projects we have been involved with at our lab. In the first project, we had to implement a high availability solution with several servers (obviously) and shared storage. The direct impact of the hardware decision is the cost of buying the hardware and facilities (Cooling, Electricity and rack space). Training is usually a result of the software choices, but administration can be reduced by good serviceability. However, it is difficult to translate serviceability in hard currency, so we will not go into that detail.

According to our calculations, the impact of the hardware choices is at least 43%. This gives us a first data point. The next project is very different; it is a consolidation of a number of physical servers on two VMware ESX servers with shared storage.

You can see that - as in many cases - the software costs are taking the biggest bite out of your budget. In this case, the hardware choice has an impact of at least 26%. We believe it's reasonable to conclude that the hardware choice has an impact of about 30% to 45% of the costs of your IT services (not including development costs). That is quite important as we try to estimate how the Sun X4450 compares to the competition when it comes to TCO.

TCO comparison: Sun X4450 versus the competition

We started our TCO comparison by determining the price of the following configuration:

• Four Xeon MP 7340 (quad-core, 2.4GHz, 2x 4MB L2)
  
• 32GB (8x 4GB) of 667MHz FBDIMM
  
• Two 2.5" 73GB 10,000 RPM SAS hard disks
  
• A simple RAID controller that allows those hard disks to run in RAID-1
  
• At least four Gigabit NICs
  
• A 1+1 redundant PSU configuration
  
• All the necessary cable arms and rack mount kits to get the server in the rack
  
• A simple hardware only support warranty

We start as an SMB (Small or Medium Business) buying a server via the Internet. While Sun's, Dell's, and HP's sites were very helpful and give us full control on what kind of configuration we would like to order, IBM's site is hard to use. You do not get any overview of the configuration you will buy, which makes it impossible for us to add IBM to the comparison. Sure, we could a call an IBM representative, but as a typical SMB we quickly just gave up: our time is precious and the last thing we want to do is to go through long conversations on the phone.

Sun ended up as the most costly server, but they give SMBs the opportunity to try out the server for 60 days and then deduct 25% off the initial price. That is a very good offering that makes Sun competitive with Dell, who also gives an $1800 discount. In the case of Sun, there is a snake under the grass: you only get 25% off on a set configuration. From the "Try and Buy" FAQ:

"Q: May I customize the trial configurations offered?

A: No. In order to offer such a wide variety of free trial systems to such a wide audience on a global basis, we need to have set configurations. If you are interested in a free trial for products or configurations not available through this program, please see your local Sun sales representative or Authorized Sun Partner.

Let's now make the TCO calculation for 3 years, which consists of TCA and Facilities: power consumption, cooling and rack space. We calculate the power and rack space for one server, and we assume that you can use 40U of a rack for your servers.

Assumptions

• We assume a $550 collocation cost / rack space cost per month and per rack.
  
• In case of collocation, electricity costs are generally calculated relative to the number of Amperes that you need, but there are so many formulas. The compromise is that we calculate the real electricity costs as if the datacenter would charge the electricity costs directly. All the "based on the number of Ampere or KWh" formulas follow the same philosophy: the more power, the more you pay.
  
• Cooling costs are assumed to be 50% of the power consumption.
  
• We assume that the power consumption of the Dell will be roughly equal to the Intel S7000FC4UR measured in our lab. Considering how similar the servers are, the chances are good that this accurate enough for this estimate.
  
• The 0.15 cent per KWh is an estimate for an average of 3 years. The server runs at 85% for 8 hours, than idles for 16 hours.
  
• We assume that HP's power consumption is a bit higher than Sun's, based on the 1200W PSU specifications and higher "fan power".
  
• We ignore network bandwidth costs, which should be equal for all servers.

After all these assumptions, we must follow up with a disclaimer: the following calculation is a best effort estimate. If we made a large logical error, let us know. We welcome all constructive criticism as this is our first attempt at a TCO calculation for a server.

* if Dell R900 power consumption would equal to our measurements on the Intel S7000FC4UR.

** Based on HP's calculator, probably slightly too high

*** We only deducted 25% on the "try and buy" configuration that was available in the US. For example, 16 GB of RAM had to be added later, so that part of the RAM was calculated with the full price.

So based on our "not accurate, but probably close enough" TCO numbers, Sun manages to save per server:

• Probably about $2000 on Dell, while giving same day support (Dell provides next business day)
  
• More than $5000 on HP

Those are pretty impressive numbers if you ask us. A more detailed analyses would probably further improve the situation for Sun as several details make the X4450 easier to maintain (separate cover for replacing fans, fans all situated in the same place, almost cable-less).

Conclusion

We start with the only problem we encountered. We feel Sun should adopt the more flexible configuration possibilities (with discount!) of the DELL Servers. The “Try and Buy” program is only applicable to about two predefined configurations in most countries, and the 25% discount can make a big difference to the buying decision. Sun should give the "Try and Buy" customers a few more options: especially memory and disks should be chosen freely. But at the end of the day, sacrificing a bit of flexibility for getting this lower power, low rack space and powerful server at a low price is well worth in our opinion.

IBM and HP better take notice: Sun has thrown down the proverbial gauntlet. We would be happy to review HP and IBM systems, but we think it will be very difficult to beat the Sun Fire X4450 unless the servers are also 2U designs. There is life outside the blade world, where IBM and HP have demonstrated so much innovation. It is time to return to the drawing board and bring that kind of innovation to the four socket servers.

Unless HP, Dell, or IBM comes out quickly with an appropriate reply, we feel Sun has a winner here. Sun's aggressive pricing helps, but it is the exceptional hardware design that makes the difference. This 2U server is able to deliver the same performance and 95% of the expandability of the 4U competition, and it is easier to maintain than any of those bulky boxes are. We experienced it first hand, as we are used to working with Dell boxes. Lower power, a pretty face, but especially the "try and buy program" top it off: the Sun X4450 is ahead of the pack. Here's to hoping this will create a new wave of innovation throughout the x86 server world.

References

[1] Intel, "Intel 7300 Chipset Memory Controller Hub (MCH)" ,Thermal/Mechanical Design Guide, September 2007