Ever since GIGABYTE’s Server team and I first started discussing reviews, it was interesting to see what a purely B2B (business to business) unit could do. Since then, GIGABYTE Server has expanded, catering to both the B2B and B2C (customer) markets and selling direct to end users. With the release of Haswell-EP we reported on their large launch at the time and they sent us the MD60-SC0 for review.

GIGABYTE Server MD60-SC0 Overview

For those not versed in 2P workstation and server culture, the MD60-SC0 looks a bit different to consumer line products. The DRAM and CPU sockets are aligned for airflow, first coming across the power delivery then the socket, and out the rear panel all in one straight line with the dram providing a baffle effect to channel the air. This means that the PCIe slots are in a somewhat awkward position in the middle of the motherboard, limiting the length of PCIe devices, and making the platform focus more on CPU power and storage rather than a GPU powerhouse without riser cables.

The sockets might also confuse some users. Unlike the standard square LGA2011-3 sockets we see on most Haswell-E or Haswell-EP motherboards, the MD60-SC0 uses the narrow socket configuration. This requires different coolers as well due to the different screw hole placement, and these coolers are typically not sold at retail and thus come OEM only.  Gigabyte also supplied us with a pair of Dynatron R14 heatsinks for our review which certainly looked the part, although the noise at high loading is something I wouldn't wish on anyone. This system combination belongs in a server room for sure.

The system is based on the C612 chipset, which is similar to the consumer based X99 but with 2P related features, such as MTCP over PCIe. We were also supplied with a type-T LSI RAID mezzanine card to enable some on-board SAS ports. So while the ports are part of the motherboard, the raid card is a separate purchase and leverages the T slot configuration. It also allows a series of RAID card potential upgrades over time if needed.

As with most server motherboards, the MD60-SC0 does not have any form of onboard audio but does offer two gigabit network ports alongside an Aspeed management interface. The board also comes with a QSFP+ port for added network connectivity.

Benchmark wise, as this is the first Haswell-EP motherboard we have tested, it is a little hard to place. As usual with management-esque type motherboards, POST times are long and power consumption is in the upper echelons. DPC Latency with 2697 V3 CPUs was quite reasonable though, despite the other two CPU combinations giving much higher peaks.

One of the main server design crux points is the orientation and the size of the board, so the MD60-SC0 has to fit into your design paradigm to make the short list. With narrow sockets and the PCIe orientation it clearly aims itself at the OEM and server builders more than consumers.

Visual Inspection

The first thing that strikes as the motherboard is taken out of the box is two things – the size of the sockets, and the fact that the motherboard is not a simple square but with a cut out. The size of the sockets is merely from the point of view that I have not encountered the Narrow ILM LGA2011 based socket before, let alone the LGA2011-3 iteration. Using the socket is easy enough, although instead of the hooks we get with the larger socket the narrow socket has a flattened lever on one side and a raised lever the other. The usage is exactly the same.

Similar to the narrow socket, Type-T mezzanine connectors are a new concept for this reviewer. The gap in the PCB is such that the add-in card can fit, as the card is built within a certain 1U/2U standard. I would assume that non-perfect rectangular PCBs are harder to create, requiring a cutout, but then again all the motherboards we have reviewed at AnandTech all have cutouts for screw holes, and as such is probably not the oddest thing we have encountered.

Each socket uses all four memory channels at two DIMMs per channel. Combine this with the narrow sockets and there is space left for SATA ports on the edge of the board. Either way you cut it, it is a very tight squeeze and as such the narrow CPU coolers have to conform to Intel specifications exactly. The 16 total DRAM slots will accept 128GB of UDIMM memory, up to 512GB of RDIMMs and 1024GB of LRDIMMs. The thought of 1TB of DRAM in a single system is mindboggling.

Each socket is supplied with six phase power and an applicable heatsink. For those more accustomed to the mid-to-high end consumer market, this combination might not look like enough, especially when the motherboard has to deal with 160W CPUs. One thing C612 motherboards have in their favor is a lack of overclocking, meaning that power draw is a known quantity. Also, with this motherboard being oriented with sockets and DRAM aligned with the rear panel, the focus will be in systems with high pressure fans blowing in a single direction across the motherboard. This will aid the cooling, especially when at full tilt.

The PCIe mezzanine type-T arrangement is a full PCIe 3.0 x8 affair, with each side of the PCIe arrangement dealing with power and data. One important thing to note with type-T is the ability to cope with height restrictions. PCIe devices are notoriously tall, and are rarely given upright in server systems that are not 4U or above. Type-T allows smaller height arrangements, and as this motherboard is geared more towards storage with SATA breakout connectors and the eight SAS RAID ports, Type-T is a good fit for smaller height systems. It is worth nothing that the red SATA ports on the motherboard are the basic SATA 6 Gbps ports for testing the system or OS/storage when the breakout cables are not in use.

With our memory right up against the SAS ports, there might be a slight conflict if locking cables are used here especially at the edges with the DRAM latches. Though one would imagine that in a server, the cables are fixed and only the drives are moved if they need replacing.

In this area of the motherboard is also where we see the fan header arrangement. Beside each socket is a four-pin fan header, although these are SYS headers. There are five SYS headers on board – four on the right hand side of the board and one at the top. The two CPU fan headers are found to the left of both the sockets.

The PCIe arrangement affords two possibilities: either an x8/x16/x8/x8 arrangement with the type-T at PCIe 3.0 x8, or an x8/x16/-/x16, again with the type-T at PCIe 3.0 x8.  Either way, due to the location of the sockets, large PCIe co-processors can only be used with a riser card or cable. For our testing, we typically equip the system with a GTX 770 Lightning. This was not possible with this system, and as such we used an R7 240 instead and we were unable to perform our normal GPU based testing.

Above the PCIe slots is the meat of the IO and control, with the Aspeed management engine chip paired with 256MB of Samsung flash and also the Intel 82599ES controller in quick succession.

Also in this area of the motherboard is a USB 3.0 header, a TPM header, a COM header and a Thunderbolt header (for use with a TB card). The QSFP Ethernet controller requires its own heatsink, and the port extends some way onto the motherboard:

Perhaps a little surprising is the power connectors. Bonus points for their location on the edge of the motherboard, although typically we see them a lot near the CPUs, especially the 8-pin connectors. This might have implications for power arrangement and delivery though the PCB, although as the board squeezes two sockets with 2DPC DRAM support in the way that it does it seems to be a reasonable compromise.

The rear panel is networking focused, giving the QSFP+ port alongside two Intel I350 ports and the server management port. All six USB ports on the rear are USB 3.0 standard, with a combination PS/2 port, a COM port and a VGA port (from the Aspeed) also in the mix.

Board Features

GIGABYTE MD60-SC0
Price	US (Newegg)
Size	SSI EEB
CPU Interface	LGA2011-3, Narrow ILM
Chipset	Intel C612
Memory Slots	Sixteen DDR4 DIMM slots Up to 128 GB UDIMM, 512GB RDIMM, 1024GB LRDIMM Up to Quad Channel, 2133 MHz
Video Outputs	VGA (via Aspeed)
Network Connectivity	Intel 82599ES (QSFP+) 2 x Intel I350 10/100 Management Port
Onboard Audio	None
Expansion Slots	2 x PCIe 3.0 x16 2 x PCIe 3.0 x8 1 x PCIe 3.0 x8 Type-T
Onboard Storage	2 x SATA 6 Gbps, RAID 0/1/5/10 4 x SATA 6 Gbps via mini-SAS 4 x S_SATA 6 Gbps, no RAID, via mini-SAS 8 x SAS/SATA via Type-T RAID card
USB 3.0	6 x USB 3.0 via Rear Panel 2 x USB 3.0 via Onboard Header
Onboard	2 x SATA 6 Gbps 2 x mini-SAS Breakout connectors 8 x SAS RAID Ports 1 x USB 3.0 Header 1 x COM Header 1 x TPM Header 7 x Fan Headers 1 x Thunderbolt Header Front Panel Server Header
Power Connectors	1 x 24-pin ATX 2 x 8-pin CPU
Fan Headers	2 x CPU (4-pin) 5 x SYS (4-pin)
IO Panel	1 x Combination PS/2 Port 6 x USB 3.0 Ports 1 x COM Port 1 x VGA Port 1 x QSFP+ Port (via Intel 82599ES) 2 x 1Gbit RJ-45 Ports (via Intel I350) 1 x 10/100 Network Management Port (via Aspeed)
Warranty Period	3 Years
Product Page	Link

The big cost here will be that QSFP+ port, although we are not sure on exact cost between manufacturer and end-user – it could be in the region of $50 to $300. The narrow LGA2011-3 slots will also require different CPU coolers to normal as well. The mezzanine Type-T arrangement and RAID slots will need an added purchase to get these working too.

GIGABYTE Server MD60-SC0 BIOS

In most server environments, the look of the BIOS is superfluous to the actual functionality, although any attempt by the manufacturer to improve the look and make it easier to use is welcome. Our recent GIGABYTE Server based reviews have looked at the base UEFI mode, similar to what used to be normal and still is when you strip away the graphical interfaces of mainstream motherboards, except there is one difference with the MD60-SC0: we have a screenshot option! Luckily I found this by accident, as it is not mentioned in the list of options on the right hand side:

Despite being an 800x600 resolution list of options, they are easier to read than some of the more consumer oriented motherboard options (!). Here on this first page we get the motherboard model, BIOS version, processors installed, CPU speed, memory installed, the memory frequency and MAC addresses of the LAN ports. While this is perhaps expecting a lot from a server grade BIOS, perhaps some CPU temperatures might be handy here as well.

The bulk of the options are in the Advanced menu, involving PCIe control, LAN control, COM configuration, fast boot/CSM and USB configuration. Users looking for Above 4G Decoding can find it in the PC Subsystem menu where it is set to disabled by default:

CSM is enabled by default, but the fan controls are a little obfuscated and hidden inside the Chipset Configuration option:

Here we get several modes for all the fan headers, including Performance, Silent, Turbo and Full Speed. On a normal consumer motherboard, we would expect a per-header breakdown with an interface to determine the fan gradients, however as this motherboard is headed for servers, ‘Performance’ fan speed will most likely be the scenario. We left our fans on performance during testing.

For a bigger breakdown of CPU configurations, the IntelRCSetup tab has the following options:

Users wanting to adjust core counts, HyperThreading and C-states can find options here, involving the Processor Configuration menu:

As well as the Power Management Configuration menu:

Here we have the BIOS by default in Energy Efficient mode, with options for Performance mode as well. This adjusts the power states in the other menus below.

For DRAM options, the Integrated Memory Controller offers users to enforce ECC, select the memory frequency or adjust the RAS Configuration:

VT-d options are via the IIO Configuration option in IntelRCSetup:

Here VT-d is enabled by default.

Error logging also has its own sub-menu:

The Server Management menu contains information regarding the Aspeed chip on board, as well as options to fix the IP address and configuration:

Secure Boot mode has options for Key Management:

Boot options are standard, with the final Save and Exit menu allowing for Boot Override.

One request to the GIGABYTE Server team would be to give the user an indication of what menu they are in. If a user ends up going through six menus to find an option, in order to help other users find the same option, by having the screenshot say where they are it might help. MSI did this on their consumer range recently at my request, and it went down well. Apart from this, some finer control over the fans would be good.

GIGABYTE Server MD60-SC0 Software

As with GIGABYTE Server’s other motherboards, OS software is non-existent. The only software to mention is that provided by the Aspeed management chip, accessed through the web interface on another machine. This software is based on the MergePoint Management interface, integrated with GIGABYTE’s logos, and offers the typical server management options we have seen previously.

The latest firmware was pre-installed on our sample, dated for August. The Aspeed 2400 is recognized here as well to differentiate itself from the Aspeed 2300 setups we have encountered previously.

Nothing in the MergePoint system is new from our perspective, and starting from the top we have the Network and Network Security:

The management system allows a maximum of 16 named users. The default user is ‘admin’.

Web server details can also be defined:

An administrator can update the BIOS/ME of the motherboard through this interface as well with a pre-downloaded file from GIGABYTE’s website. This is particularly helpful to update the system without entering the BIOS or updating through the OS. It will however terminate all other sessions.

As with other server motherboards, there are sensors all over the place. The management interface has an adequate list of everything a user might need:

One missing feature is the ability to set the non-critical and critical boundaries for each of the sensors separately.

Power control is an important part of server management, and here is our list of options.

The System Event Log captures all boots, reboots or sensor trips. I have used this several times in previous reviews when unable to get an image to display on a monitor.

The management software also allows direct Java-based KVM control of the system as well.

GIGABYTE Server MD60-SC0 In The Box

Server motherboards are typically quite spartan, giving only a couple of SATA cables in the box to help facilitate non-standard use. Due to the orientation of the motherboard and the layout of the dedicated SATA ports, GIGABYTE Server steps a little further in this regard by providing two mini-SAS to SATA cables for the two headers on board. This allows the 8 of the 10 SATA ports provided by the chipset to be used without extra hardware needing to be purchased. Also in the box is a rear IO shield, manuals and driver disks.

Many thanks to...

We must thank the following companies for kindly providing hardware for our test bed:

Thank you to OCZ for providing us with PSUs and SSDs.
Thank you to G.Skill for providing us with memory.
Thank you to Corsair for providing us with an AX1200i PSU and a Corsair H80i CLC.
Thank you to MSI for providing us with the NVIDIA GTX 770 Lightning GPUs.
Thank you to Rosewill for providing us with PSUs and RK-9100 keyboards.
Thank you to ASRock for providing us with some IO testing kit.
Thank you to Cooler Master for providing us with Nepton 140XL CLCs.

Test Setup

Test Setup
Processor	Intel Xeon E5-2697 v3 x2 (2 x 145W),  - 28C/56T, 2.3 GHz (3.6 GHz Turbo) Intel Xeon E5-2650L v3 x2 (2 x 65W),  - 24C/48T, 1.8 GHz (3.6 GHz Turbo) Intel Xeon E5-2687W v3 x2 (2 x 160W),  - 20C/40T, 2.3 GHz (3.6 GHz Turbo)
Motherboards	GIGABTYE MD60-SC0
Cooling	Dynatron R14
Power Supply	OCZ 1250W Gold ZX Series Corsair AX1200i Platinum PSU
Memory	ADATA XPG Z1 DDR4-2400 C16 8x8GB 1.2V
Memory Settings	JEDEC @ 2133
Video Cards	R7 240
Video Drivers	AMD Catalyst 13.12
Hard Drive	OCZ Vertex 3 256GB
Optical Drive	LG GH22NS50
Case	Open Test Bed
Operating System	Windows 7 64-bit SP1
USB 2/3 Testing	OCZ Vertex 3 240GB with SATA->USB Adaptor

System Benchmarks

Power Consumption

Power consumption is normally tested on the system while in a single MSI GTX 770 Lightning GPU configuration with a wall meter connected to the OCZ 1250W power supply, however for this review due to the PCIe arrangement we had an R7 240 equipped. This power supply is Gold rated, and as I am in the UK on a 230-240 V supply, leads to ~75% efficiency > 50W, and 90%+ efficiency at 250W, suitable for both idle and multi-GPU loading. This method of power reading allows us to compare the power management of the UEFI and the board to supply components with power under load, and includes typical PSU losses due to efficiency. These are the real world values that consumers may expect from a typical system (minus the monitor) using this motherboard.

While this method for power measurement may not be ideal, and you feel these numbers are not representative due to the high wattage power supply being used (we use the same PSU to remain consistent over a series of reviews, and the fact that some boards on our test bed get tested with three or four high powered GPUs), the important point to take away is the relationship between the numbers. These boards are all under the same conditions, and thus the differences between them should be easy to spot.

Having two processors installed doesn't take much more power at idle than our i7-5960X X99 counterparts, but when the CPU load starts to flow, the obvious differences arise. Interestingly the dual 65W combination for the E5 2650L v3 CPUs used less power than a single 130W CPU.

Windows 7 POST Time

Different motherboards have different POST sequences before an operating system is initialized. A lot of this is dependent on the board itself, and POST boot time is determined by the controllers on board (and the sequence of how those extras are organized). As part of our testing, we look at the POST Boot Time using a stopwatch. This is the time from pressing the ON button on the computer to when Windows 7 starts loading. (We discount Windows loading as it is highly variable given Windows specific features.) 

As mentioned earlier in the review, POST time on server motherboards is naturally slow due to the server management tools as well as the extra controllers. POST times are not that important for servers anyway, given that they tend to be restarted far less frequently than desktops or workstations.

USB Backup

For this benchmark, we transfer a set size of files from the SSD to the USB drive using DiskBench, which monitors the time taken to transfer. The files transferred are a 1.52 GB set of 2867 files across 320 folders – 95% of these files are small typical website files, and the rest (90% of the size) are small 30 second HD videos. In an update to pre-Z87 testing, we also run MaxCPU to load up one of the threads during the test which improves general performance up to 15% by causing all the internal pathways to run at full speed.

DPC Latency

Deferred Procedure Call latency is a way in which Windows handles interrupt servicing. In order to wait for a processor to acknowledge the request, the system will queue all interrupt requests by priority. Critical interrupts will be handled as soon as possible, whereas lesser priority requests such as audio will be further down the line. If the audio device requires data, it will have to wait until the request is processed before the buffer is filled.

If the device drivers of higher priority components in a system are poorly implemented, this can cause delays in request scheduling and process time.  This can lead to an empty audio buffer and characteristic audible pauses, pops and clicks. The DPC latency checker measures how much time is taken processing DPCs from driver invocation. The lower the value will result in better audio transfer at smaller buffer sizes. Results are measured in microseconds.

The DPC Latency for the dual E5 2697 v3 setup was not bad - our previous barrier to good/bad was 200 microseconds, although Z97 and X99 have both push the average well below 100. The other two CPUs caused large spikes in our DPC testing, giving results of 502 and 714 microseconds.

Professional Performance

Professional Performance: Windows

Agisoft Photoscan – 2D to 3D Image Manipulation: link

Agisoft Photoscan creates 3D models from 2D images, a process which is very computationally expensive. The algorithm is split into four distinct phases, and different phases of the model reconstruction require either fast memory, fast IPC, more cores, or even OpenCL compute devices to hand. Agisoft supplied us with a special version of the software to script the process, where we take 50 images of a stately home and convert it into a medium quality model. This benchmark typically takes around 15-20 minutes on a high end PC on the CPU alone, with GPUs reducing the time.

Cinebench R15

Professional Performance: Linux

Built around several freely available benchmarks for Linux, Linux-Bench is a project spearheaded by Patrick at ServeTheHome to streamline about a dozen of these tests in a single neat package run via a set of three commands using an Ubuntu 14.04 LiveCD. These tests include fluid dynamics used by NASA, ray-tracing, molecular modeling, and a scalable data structure server for web deployments. We run Linux-Bench and have chosen to report a select few of the tests that rely on CPU and DRAM speed.

C-Ray: link

C-Ray is a simple ray-tracing program that focuses almost exclusively on processor performance rather than DRAM access. The test in Linux-Bench renders a heavy complex scene offering a large scalable scenario.

NAMD, Scalable Molecular Dynamics: link

Developed by the Theoretical and Computational Biophysics Group at the University of Illinois at Urbana-Champaign, NAMD is a set of parallel molecular dynamics codes for extreme parallelization up to and beyond 200,000 cores. The reference paper detailing NAMD has over 4000 citations, and our testing runs a small simulation where the calculation steps per unit time is the output vector.

NPB, Fluid Dynamics: link

Aside from LINPACK, there are many other ways to benchmark supercomputers in terms of how effective they are for various types of mathematical processes. The NAS Parallel Benchmarks (NPB) are a set of small programs originally designed for NASA to test their supercomputers in terms of fluid dynamics simulations, useful for airflow reactions and design.

Redis: link

Many of the online applications rely on key-value caches and data structure servers to operate. Redis is an open-source, scalable web technology with a b developer base, but also relies heavily on memory bandwidth as well as CPU performance.

CPU Benchmarks

Readers of our motherboard review section will have noted the trend in modern motherboards to implement a form of MultiCore Enhancement / Acceleration / Turbo (read our report here) on their motherboards. This does several things, including better benchmark results at stock settings (not entirely needed if overclocking is an end-user goal) at the expense of heat and temperature. It also gives in essence an automatic overclock which may be against what the user wants. Our testing methodology is ‘out-of-the-box’, with the latest public BIOS installed and XMP enabled, and thus subject to the whims of this feature. It is ultimately up to the motherboard manufacturer to take this risk – and manufacturers taking risks in the setup is something they do on every product (think C-state settings, USB priority, DPC Latency / monitoring priority, memory subtimings at JEDEC). Processor speed change is part of that risk, and ultimately if no overclocking is planned, some motherboards will affect how fast that shiny new processor goes and can be an important factor in the system build.

Point Calculations – 3D Movement Algorithm Test: link

3DPM is a self-penned benchmark, taking basic 3D movement algorithms used in Brownian Motion simulations and testing them for speed. High floating point performance, MHz and IPC wins in the single thread version, whereas the multithread version has to handle the threads and loves more cores.

Compression – WinRAR 5.0.1: link

Our WinRAR test from 2013 is updated to the latest version of WinRAR at the start of 2014. We compress a set of 2867 files across 320 folders totaling 1.52 GB in size – 95% of these files are small typical website files, and the rest (90% of the size) are small 30 second 720p videos.

Image Manipulation – FastStone Image Viewer 4.9: link

Similarly to WinRAR, the FastStone test us updated for 2014 to the latest version. FastStone is the program I use to perform quick or bulk actions on images, such as resizing, adjusting for color and cropping. In our test we take a series of 170 images in various sizes and formats and convert them all into 640x480 .gif files, maintaining the aspect ratio. FastStone does not use multithreading for this test, and thus single threaded performance is often the winner.

Video Conversion – Handbrake v0.9.9: link

Handbrake is a media conversion tool that was initially designed to help DVD ISOs and Video CDs into more common video formats. The principle today is still the same, primarily as an output for H.264 + AAC/MP3 audio within an MKV container. In our test we use the same videos as in the Xilisoft test, and results are given in frames per second.

Rendering – PovRay 3.7: link

The Persistence of Vision RayTracer, or PovRay, is a freeware package for as the name suggests, ray tracing. It is a pure renderer, rather than modeling software, but the latest beta version contains a handy benchmark for stressing all processing threads on a platform. We have been using this test in motherboard reviews to test memory stability at various CPU speeds to good effect – if it passes the test, the IMC in the CPU is stable for a given CPU speed. As a CPU test, it runs for approximately 2-3 minutes on high end platforms.

Synthetic – 7-Zip 9.2: link

As an open source compression tool, 7-Zip is a popular tool for making sets of files easier to handle and transfer. The software offers up its own benchmark, to which we report the result.

GIGABYTE Server MD60-SC0 Conclusion

The MD60-SC0 has a clear market - air directed servers that prioritize CPU performance and storage over serious GPU compute, as the latter is only possible with PCIe riser configurations. Due to the orientation for the sockets, the use of narrow sockets and the SSI EEB form factor, it's use as a workstation board for OEMs or as a DIY board for home use is most likely limited unless they are already geared up for this scenario.

As always with GIGABYTE Server motherboards, most of the extra controls in place are via the Aspeed management console with access via the network interface. The BIOS and software bundled with the system is basic at best, especially when or if a user wants to control the fan speeds depending on workload. That being said, the management console will be familiar to those with experience in this field.

For those more used to consumer type products, it might sound odd for a motherboard manufacturer to include ports on the board that are only enabled when an add-in card is used. The MD60-SC0 does this by supporting a raid card via the Type-T mezzanine connector to power the SAS ports beside it. This does have some usefulness by allowing users to decide which RAID card is relevant, allowing an upgrade without much fuss, although the card GIGABYTE recommends revolves around the LSI SAS 2308 chip. It is worth noting that this card gets very hot to touch without airflow.

With respect to the system benchmarks, as per usual with management controlled server motherboards, the initial power on to turn on time is over 30 seconds. POST times were significantly cut if there is no use for some of the extra controllers or the raid card connectors. DPC latency was a little obscure, with the motherboard preferring the 2697 V3 processors over the other models we had at hand.

As this is our first LGA2011-3 dual socket motherboard review, it is a little difficult to place the benefits against the competition beyond the big obvious checkbox of QSFP. GIGABYTEServer's range of motherboards at launch afforded several models, a few focused on workstation but mostly for rack servers. The MD60-SC0 falls into the latter category, especially with this network configuration, and it ran most of our benchmark suite without issue. The main selling point of the motherboard will be that QSFP port.