In August of 2000 we documented the largest server upgrade in the history of AnandTech.  We added to our previous set of four dual Xeon boxes an additional four web servers and a new mail/images server.  The significance of this particular server upgrade was that the five systems we sent up to our host were all using AMD Athlon processors. 

For the longest time AMD could not be taken seriously as a manufacturer of server class processors.  The introduction of the Athlon changed all that; with the release of the Thunderbird core the Athlon was finally ready for primetime. 

If you remember back to our original article on the upgrade, the reason for pursuing an AMD based solution was because, at the time, there were no Intel solutions that could fit our needs.  We required a solution that supported at least 768MB of PC133 SDRAM, this ruled out the i815E as an option. 

We didn't have enough confidence in the Apollo Pro 133A chipset at the time and it was clear that the majority of the 133A-based motherboards out there weren't designed with server setups in mind.  Xeons weren't really an option since the higher speed solutions required a 133MHz FSB and the only Slot-2 chipset that did so was part of the ServerWorks line which we had no experience with at the time either.  Plus, for web serving, Xeons were overkill.  We did not require a large L2 cache; we required a fast L2 cache and a high clock speed. 

The Athlon delivered on both of those requirements making it perfect for our needs.  And over the past few months we have definitely put the setup to the test.  In fact, four of the 1GHz Thunderbirds were more than enough to handle the load of AnandTech's ColdFusion based front-end.  They did such a good job that we devoted our remaining dual Xeon based webserver (formerly www5.anandtech.com) to hosting the AnandTech Forums which have gone through an extreme growth-spurt of their own. 

So as it stands now, we have two database servers, both dual Xeon systems:  one hosting the AnandTech Database, the other hosting the AnandTech Forums Database.  On the web side of things, AnandTech currently employs the same four 1GHz Thunderbird systems we introduced in August while the Forums are still run off of two of our original dual Xeon boxes. 

Just recently, AnandTech Forums users began noticing high traffic errors and general sluggishness while surfing through the Forums.  However on our end, the performance of the web servers seemed to be just fine leaving the database server as the culprit of the recently degraded performance.  This didn't make much sense, because the Forum's database server was configured almost identically to the AnandTech database server and AnandTech is definitely under a heavier load than the forums.  Or is it?

Finding the Bottleneck

Remembering back to our first article on how AnandTech's server backbone works, all content (news, articles, and forum posts) are stored in a SQL database on a database server.  The main site has a database server as do the Forums.  The web servers handle presenting this content to you all whenever a request is made as they pull the content off of the database server and feed it into a template stored on the web servers.  When you hit www.anandtech.com or forums.anandtech.com the web servers are serving you templates stored on those servers filled with content pulled from the database servers.

The AnandTech web servers were definitely under a heavier load than the Forums web servers, simply because there are more people on the main site than there are on the Forums.  But comparing the load on AnandTech database server to the load on the Forums db server is like comparing apples to oranges; you can't take the load percentages for their face value.

The dynamics of the two entities are completely different.  The AnandTech main site is, for the most part, updated daily with the exception of the Web News section that features multiple updates throughout the day.  The beauty of this is that the main site can be served almost entirely out of memory, with the cache only having to be flushed once a day.  This makes the job of AnandTech's database server very easy, however the same can't be said about the AnandTech Forums.

Unlike the main site, the Forums are being constantly updated.  You can consider the Forums to be an example of what the main site would be like if we posted a new article every minute.  The nature of the Forums prevents us from employing much caching at all since it would be pretty useless if new posts only appeared once every 12 hours.  Compared to the workout that the main site gives its database server, the Forums put their database server through Olympic training, in particular, the server's disk I/O subsystem.

Initially, the Forums Database server was setup identically to the AnandTech database server, with three IBM Ultrastar 9LZX U2SCSI drives setup in a RAID 5 configuration.  For those of you that aren't familiar with RAID 5, it is a storage configuration that offers two benefits: Striping and Parity. 

Striping, a benefit gained from RAID 0, is the result of combining two or more hard disks in such a way that your read/write performance is increased by a factor of the number of drives you have present in the array.  The reason behind this is simple, instead of reading/writing directly off of a single disk, two drives combined in a RAID 0 fashion will allow data to be split into "stripes" and read from/written to each drive in the array.  The downside here is that if a single drive fails the entire array goes down. 

Parity is a feature that is provided for by RAID 5 when dealing with storage arrays.  In this type of an array, parity data is stored on all of the drives in the array (a minimum of 3).  In the event that a single drive out of the array fails, the data can be reconstructed from the parity data that exists on all of the remaining drives.  Fault tolerance does not exist if more than one drive fails. 

Two drives in a RAID 5 array can offer read performance similar to that of drives in a RAID 0 array (potentially even greater than that because you have one more physical drive with RAID 5).  As we stated before, this is approximately equal to or greater than twice the read performance of a single drive.  The real downfall of RAID 5 however is that its write performance is significantly lower than RAID 0.  The drives in a RAID 5 array must be essentially written to twice, one for the data and one for the parity information.  In a write-intensive situation such as the picture we painted for you all with the AnandTech Forums, RAID 5 offers more of a performance bottleneck than an advantage. 

The Solution

For database servers under load similar to that of the AnandTech main site a RAID 5 array makes sense because it offers a good balance between performance, fault tolerance and array capacity.  However for a server such as the AnandTech Forums database server, a RAID 5 array would definitely be a limiting factor in the server's performance.

The next step was to move away from a parity solution to another fault tolerance mechanism known as mirroring.  The most basic form of mirroring in a RAID setup comes with what is known as RAID 1.  A mirrored array, as the name implies, is a two-drive setup where the larger of the two drives is used to mirror the data on the remaining drive.  If the drives are of equal size, the mirror is obviously the same size as the data drive. 

The read performance of a RAID 1 array is approximately equal to that of a similar RAID 0 setup, while the write performance is about the speed of a single drive depending on the type of writes being performed. 

If you combine the fault tolerance of a RAID 1 solution with the performance of a RAID 0 solution you get two new configurations:  RAID 0+1 and RAID 10.  RAID 0+1 being a RAID 1 array of two RAID 0 arrays (four drives) and RAID 10 being a RAID 0 array of multiple RAID 1 arrays (four or more drives).  The beauty of this setup is that you get read performance that is very close to a four drive RAID 0 setup with write performance that resembles a two drive RAID 0 setup and fault tolerance courtesy of the RAID 1 component. 

For database servers such as the AT Forums database server, this setup would make much more sense although it is a more expensive solution since a minimum of four drives is required. 

Since we were building a new database server for the Forums we figured that we might as well pick out a RAID adapter that was properly supported under Windows 2000.  The original problem we had with upgrading the Forums database server to Windows 2000 was that the Adaptec AAA-133U2 RAID adapter it used did not have reliable Windows 2000 drivers.  Adaptec's only solution to this was a discount program in which they would give us a discounted price on a new, Windows 2000 compatible adapter. 

The controller we ended up going with was the Adaptec 2100S.  It is a single channel Ultra160 SCSI RAID adapter that comes with 16MB of SDRAM standard.  The controller is a 32-bit PCI solution meaning it is limited by the PCI bus' 132MB/s bandwidth limitation.  We'll explain why we didn't go for a 64-bit PCI solution later on, but for the most part this 132MB/s limitation wasn't too big of a deal; it just meant that our storage array would never be able to burst at greater than that speed, we didn't have to worry about sustained transfer rates exceeding that figure.

The array itself was composed of four 9GB Quantum Atlas 10K II drives.  A RAID 0+1/10 array of four identical drives offers a storage capacity equal to that of two of the drives.  For the purposes of our Forums db server, 18GB of storage was more than enough making these four drives a great buy at less than $300 a piece. 

The Atlas 10K II is a very high performing Ultra160 SCSI solution.  Its 10,000-RPM rotational speed and large 8MB buffer help it perform quite well.  The fact that these drives are Ultra160 solutions and are being used on an 32-bit Ultra160 RAID controller help to ensure that the drive interface won't be a limiting factor before the bus interface limits performance. 

Database servers love multiple processors

Although our last server upgrade consisted of a large migration to Socket-A platforms using AMD Athlon processors, a need for multiprocessor support for our single Forums database server kept the Athlon out of the running for this upgrade.  With the AMD 760MP chipset still four or five months away from being readily available, the only remaining option was to pursue an Intel based solution.

Instead of pursuing higher clocked Pentium III Xeon processors to replace our Pentium III Xeon 500s that were currently running the Forums database we decided to move to higher clocked Pentium III processors.  The 133MHz FSB Pentium III Xeon processors are currently no different than the 133MHz FSB Pentium III processors that have been in the desktop market since late 1999.  The Pentium III Xeons simply plug into a Slot-2 interface while the desktop Pentium IIIs are Slot-1/Socket-370 solutions; they both have 256KB of on-die L2 cache.

The sweet spot for Pentium III processors is around 800 - 866MHz since they currently offer the best price to performance ratio out of all of the Pentium III processors.  At the time we decided to go with dual Pentium III 800s, these being the 133MHz FSB versions. 

DB Servers also love memory

With the requirements for our new Forums box growing we tacked on another stipulation, the dual processor setup must have at least 1GB of SDRAM, preferably more.  Our previous Forums DB box had 1GB of PC100 SDRAM and it was already running low, so ideally 2 - 4GB was what we wanted.  With no Socket-370 boards featuring more than 4 SDRAM sockets we had to find 512MB or 1GB PC133 SDRAM modules.  But even before that, the quest for the perfect motherboard was under way. 

With the i815E chipset limited to single processors and no more than 512MB of PC133 SDRAM it was immediately thrown out as an option.  The BX/GX chipsets could support more than 512MB of SDRAM however they were both limited to a 100MHz FSB and PC100 SDRAM, which would be sufficient however it wasn't ideal. 

Luckily there was another option available from a company that's rarely talked about in our community, ServerWorks (just recently acquired by Broadcom).  The ServerWorks line consists of three chipsets, the ServerSet III LE, WS and HE.  For our needs, the ServerSet III LE chipset was perfect.  While the III LE does not support AGP, for a server that won't be running any 3D applications there isn't really a need for it; the WS and HE versions of the chipset do support AGP if you are interested. 

Click to Enlarge

Other features of the ServerSet III LE chipset are its 64-bit PCI bus and ATA/66 support.  The chipset itself is composed of a very large 576-pin Northbridge and a sizeable 352-pin Southbridge.

The ServerSet III LE's dual processor support and ability to accept as much as 4GB of PC133 SDRAM made it the most well rounded solution we could find.  The only stipulation to that 4GB SDRAM limit is that the chipset only supports registered DIMMs.  Mushkin was able to provide us with a total of 2GB of PC133 SDRAM in the form of 4 registered DIMMs.  We did encounter an interesting issue with one of the four modules, which we will talk about later. 

Click to Enlarge

There are quite a few motherboards available that make use of the ServerSet III LE chipset and the first one we decided to use was the Intel STL2.  But as we soon discovered, the STL2 would have to be replaced as our motherboard of choice.

Conserving Space: a 2U Database Server

Continuing the assembly process we needed to find a case to house all of these components. 

Our drive bay requirements were actually one of the most difficult needs to tend to when looking for a chassis.  Not only did we have the four Quantum Atlas 10K II drives that would handle the Forums database, the new server would have two more hard drives.  The first of the two, a Western Digital Ultra ATA 66 boot drive and the second an Ultra SCSI Seagate Barracuda 18XL we had lying around in the lab.  The Barracuda was going to be used for storing the full-text catalog files that allow us to implement a full-text search on the Forums.  By keeping these files on a separate drive and on a separate controller (the on-board SCSI controller) the I/O load is spread out even more, helping performance as well. 

We wanted to find the smallest chassis that could house a dual Socket-370 motherboard and at least 6 drives.  After searching around and turning up units that could house 4 or 5 drives we found just what we were looking for in a company called BoomRack. 

BoomRack had a 2U chassis (3.5" high) that featured a total of 8 drive bays (6 x 3.5" and 2 x 5.25").  The BOOM2U300XA was also a very cost effective solution as it was priced just under $400.  While the screws that came with the chassis weren't the strongest we had encountered, the chassis was the fairly well made. 

Click to Enlarge

With anything as small as a 2U chassis, there isn't enough headroom to mount add-in cards vertically.  Instead you must use riser adapters that basically allow you to install the cards horizontally.  The chassis we picked up from BoomRack came with two 32-bit PCI riser adapters to go along with the two horizontal slot cutouts on the back of the case. 

Click to Enlarge

Unfortunately the two slot cutouts on the case lined up with the two 64-bit PCI slots on the Intel STL2 board (you can't really see it from the above picture but the riser instlaled in the first PCI slot does not match up with the two cutouts on the back).  The reason this posed a problem was because the risers we had were 5V PCI risers meaning that they could not plug into the 3.3V 64-bit PCI slots that were on the Intel board.  This presented us with a pretty big problem.  The Intel board already had video, SCSI and 10/100 Ethernet on-board, however we needed to be able to install our RAID card at least and it would be ideal if we could install another network card for our private LAN.  The two options we had were to either find a new chassis or find a new motherboard.

A New Motherboard

Luckily we were able to find another ServerWorks board that would work with the BoomRack chassis and the two riser cards.  The ASUS CUR-DLS, a board we happened to be reviewing in the lab at the same time fit perfectly, so we went ahead and ordered another one right away. 

We did get one benefit out of the Intel STL2 board however.  During our preliminary testing before the chassis arrived the STL2 was able to detect a bad memory module we had and disable it.  This feature is not present on the ASUS CUR-DLS and is very useful since it would otherwise be very difficult to diagnose a bad memory module.  The memory would count up properly and Windows 2000 would run just fine for a little while before popping up the dreaded blue screen of death and list the error as a "Hardware Malfunction." 

This is a server-level feature that we would actually like to see on more motherboards since it can be a very useful function.  Upon a reboot if the STL2 detected a memory error it would not only complain about it but it would also disable that memory bank so that the system can start normally until that bank can be replaced. 

Due to time constraints we just sent the server up to our host in Pittsburgh with "only" 1.5GB of SDRAM rather than delay the upgrade another week to get a fixed memory module. 

Completing the Upgrade

We managed to snap a few pictures of the case while the Intel STL2 board was installed before we removed it:

Click to Enlarge

Click to Enlarge

Click to Enlarge

With our new Forums database box up and running the question of what to do with the old database server came up.  The original plan was to bring it down to the lab and rebuild it as a web server, but with the Forums growing so quickly another week or two didn't seem worth it.  Instead, we left the server up in its original configuration and simply turned it into a web server -- one with a killer I/O setup that is.  This brings our total server count up to 10: 4 AnandTech web boxes, 3 Forums web boxes, 1 AT DB server, 1 Forums DB server and 1 mail box. 

At the current rate of growth, as well as factoring in some definite spikes in traffic over the next few months, it wouldn't be too surprising if we were at close to 20 servers by the end of 2001. 

Hopefully by then we will have more to chose from in terms of multiprocessor server solutions and we are sure to have enough interesting experiences to write about our future upgrades as well. 

Behind AnandTech - The Server Pictures

You all requested it, so here are pictures of some of our boxes in their home. The new Database box was added after these pictures were taken so it isn't shown here:

Here we have the front of a rack that holds 9 of our boxes:

At the top of the rack is an exhaust fan that helps to keep the rack cool. You can see the top most switch that handles all outbound traffic and our SMC switch underneath it that handles all traffic on our private LAN.

Click to Enlarge

Organization is key to keeping airflow unobstructed throughout the rack. All of the CAT5 cables are tied to the walls of the rack and all of the power cables are pushed out of the way.

Click to Enlarge

Behind AnandTech - The Server Configs

Here is the updated server configs list of the 10 boxes behind AnandTech: