When decent, somewhat affordable, client focused solid state drives first came on the scene in 2008 the technology was magical. I called the original X25-M the best upgrade you could do for your system (admittedly I threw in the caveat that I’d like to see > 100GB and at a better price than $600). Although NAND and SSD pricing have both matured handsomely over time, there’s still the fact that mechanical storage is an order of magnitude cheaper.

The solution I’ve always advocated was a manual combination of SSD and HDD technologies. Buy a big enough SSD to house your OS, applications and maybe even a game or two, and put everything else on a RAID-1 array of hard drives. This approach works quite well in a desktop, but you have to be ok with manually managing where your files go.

I was always curious about how OEMs would handle this problem, since educating the masses on having to only put large, infrequently used files on one drive with everything else on another didn’t seem like a good idea. With its 6-series chipsets Intel introduced its Smart Response Technology, along with a special 20GB SLC SSD designed to act as a cache for a single hard drive or a bunch in an array.

Since then we’ve seen other SSD caching solutions come forward that didn’t have Intel’s chipset requirements. However most of these solutions were paired with really cheap, really small and really bad mSATA SSDs. More recently, OEMs have been partnering with SSD caching vendors to barely meet the minimum requirements for Ultrabook certification. In general, the experience is pretty bad.

Hard drive makers are working on the same problem, but are trying to fix it by adding a (very) small amount of NAND onto their mechanical drives. Once again this usually results in a faster hard drive experience, rather than an approximation of the SSD experience.

Typically this is the way to deal with hiding latency the lower you get in the memory hierarchy. Toss a small amount of faster memory between two levels and call it a day. Unlike adding a level 3 cache to a CPU however, NAND storage devices already exist in sizes large enough to house all of your data. It’s the equivalent of having to stick with an 8MB L3 cache when for a few hundred dollars more you could have 16GB. Once you’ve tasted the latter, the former seems like a pointless compromise.

Apple was among the first OEMs to realize the futility of the tradeoff. All of its mainstream mobile devices are NAND-only (iPhone, iPad and MacBook Air). More recently, Apple started migrating even its professional notebooks over to an SSD-only setup (MacBook Pro with Retina Display). Apple does have the luxury of not competing at lower price points for its Macs, which definitely makes dropping hard drives an easier thing to accomplish. Even so, out of the 6 distinct Macs that Apple ships today (MBA, rMBP, MBP, Mac mini, iMac and Mac Pro), only two of them ship without any hard drive option by default. The rest come with good old fashioned mechanical storage.

Moving something like the iMac to a solid state configuration is a bit tougher to pull off. While notebook users (especially anyone using an ultraportable) are already used to not having multiple terabytes of storage at their disposal, someone replacing a desktop isn’t necessarily well suited for the same.

Apple’s solution to the problem is, at a high level, no different than all of the PC OEMs who have tried hybrid SSD/HDD solutions in the past. The difference is in the size of the SSD component of the solution, and the software layer.

Meet Fusion Drive

Available as a build-to-order option on both the new Mac mini and the new iMac is Apple’s own take on SSD caching, Fusion Drive. In true Apple fashion there are only two Fusion Drive configurations available: 1TB and 3TB. The 1TB option is only available on the upgraded Mac mini ($799) or any of the iMacs, while the 3TB Fusion Drive is a 27-inch iMac exclusive.

In all of these cases, the Fusion Drive is a combination of a 1TB or 3TB hard drive (2.5” or 3.5”) and a 128GB Samsung PM830 based SSD. In the Mac minis this SSD is a 2.5” drive, while in the iMacs it’s the same custom interface that’s used in the MacBook Air and MacBook Pro with Retina Display. For my testing I used a 1TB Fusion Drive in a 27-inch iMac.

The size of the SSD used in Apple’s Fusion Drive is much larger than what we usually find in a caching setup. Most OEMs ship with 8 - 24GB of NAND, and even then the drives rarely use a good controller. In the case of Apple’s Fusion Drive, Samsung’s PM830 continues to be one of the best combinations of performance and reliability we’ve ever tested. While I would’ve personally picked something like the Link A Media or Intel S3700 controller due to their excellent performance consistency, the PM830 is probably a more proven and/or affordable option for Apple.

Right off the bat Fusion Drive is different than most of the hybrid/caching solutions we’ve seen, but where it really diverges from the norm is in the software component. This isn’t simply Intel’s Smart Response Technology running under an Apple brand, instead we’re looking at virtualized storage courtesy of OS X’s Core Storage. First introduced in Lion, Core Storage is a logical volume manager that allows the OS to treat multiple physical disks as a single volume.

Apple originally used Core Storage to enable full disk encryption in Lion, but its use has been expanded to Fusion Drive in Mountain Lion. The creation of a Fusion Drive is simple. If you have multiple drives you can create a Fusion Drive yourself using some simple Terminal commands. When you buy a Fusion Drive equipped Mac, Apple does everything for you. Subsequent system and backup restores on your Mac with FD will maintain the Fusion Drive facade, even if you’ve purposefully destroyed the array.

Unlike traditional SSD caching architectures, Fusion Drive isn’t actually a cache. Instead, Fusion Drive will move data between the SSD and HDD (and vice versa) depending on access frequency and free space on the drives. The capacity of a single Fusion Drive is actually the sum of its parts. A 1TB Fusion Drive is actually 1TB + 128GB (or 3TB + 128GB for a 3TB FD).

The latest version of Disk Utility will present a Fusion Drive as a single drive, labeled Macintosh HD from the factory. Apple doesn’t attempt to hide the FD underpinnings however, looking at System Report or using a third party utility like iStat Menus you’ll get statistics on both drives:

If you’ll notice, the 128GB SSD is reported as having a 121.33GB capacity. Since OS X 10.6, Apple has reported capacities in base 10 but if you do the math based on the capacity in bytes you’ll get an idea of how much space is set aside as spare area:

Approximately 11.7% of the 128GiB of NAND is set aside as spare area, which is no different than what you get with a 128GiB SSD in a standard Mac, but a bit higher than the usual 6.7% spare area you get with most of these drives. The added spare area will help improve performance consistency, but it’s still a bit shy of what I like to see on Samsung SSDs (~25%).

You can create Boot Camp or other additional partitions on a Fusion Drive, however these partitions will reside on the HDD portion exclusively.

Fusion Drive: Under the Hood

I took the 27-inch iMac out of the box and immediately went to work on Fusion Drive testing. I started filling the drive with a 128KB sequential write pass (queue depth of 1). Using iStat Menus 4 to actively monitor the state of both drives I noticed that only the SSD was receiving this initial write pass. The SSD was being written to at 322MB/s with no activity on the HDD.

After 117GB of writes the HDD took over, at speeds of roughly 133 - 175MB/s to begin with.

The initial test just confirmed that Fusion Drive is indeed spanning the capacity of both drives. The first 117GB ended up on the SSD and the remaining 1TB of writes went directly to the HDD. It also gave me the first indication of priority: Fusion Drive will try to write to the SSD first, assuming there's sufficient free space (more on this later).

Next up, I wanted to test random IO as this is ultimately where SSDs trump hard drives in performance and typically where SSD caching or hybrid hard drives fall short. I first tried the worst case scenario, a random write test that would span all logical block addresses. Given that the total capacity of the Fusion Drive is 1.1TB, how this test was handled would tell me a lot about how Apple maps LBAs (Logical Block Addresses) between the two drives.

The results were interesting and not unexpected. Both the SSD and HDD saw write activity, with more IOs obviously hitting the hard drive (which consumes a larger percentage of all available LBAs). The average 4KB (QD16) random write performance was around 0.51MB/s, it was constrained by the hard drive portion of the Fusion Drive setup.

After stopping the random write task however, there was immediate moving of data between the HDD and SSD. Since the LBAs were chosen at random, it's possible that some (identical or just spatially similar) addresses were picked more than once and those blocks were immediately marked for promotion to the SSD. This was my first experience with the Fusion Drive actively moving data between drives.

A full span random write test is a bit unfair for a consumer SSD, much less a hybrid SSD/HDD setup with roughly an 1:8 ratio of LBAs. To get an idea of how good Fusion Drive is at dealing with random IO I constrained the random write test to the first 8GB of LBAs.

The resulting performance was quite different. For the first pass, average performance was roughly 7 - 9MB/s, with most of the IO hitting the SSD and a smaller portion hitting the hard drive. After the 3 minute test, I waited while the Fusion Drive moved data around, then repeated it. For the second run, total performance jumped up to 21.9MB/s with more of the IO being moved to the SSD although the hard drive was still seeing writes.

In the shot to the left, most random writes are hitting the SSD but some are still going to the HDD, after some moving of data and remapping of LBAs nearly all random writes go to the SSD and performance is much higher

On the third attempt, nearly all random writes went to the SSD with performance peaking at 98MB/s and dropping to a minimum of 35MB/s as the SSD got more fragmented. This told me that Apple seems to dynamically map LBAs to the SSD based on frequency of access, a very pro-active approach to ensuring high performance. Ultimately this is a big difference between standard SSD caches and what Fusion Drive appears to be doing. Most SSD caches seem to work based on frequency of read access, whereas Fusion Drive appears to (at least partially) take into account what LBAs are frequently targeted for writes and mapping those to the SSD.

Note that subsequent random write tests produced very different results. As I filled up the Fusion Drive with more data and applications (~80% full of real data and applications), I never saw random write performance reach these levels again. After each run I'd see short periods where data would move around, but random IO hit the Fusion Drive in around an 7:1 ratio of HDD to SSD accesses. Given the capacity difference between the drives, this ratio makes a lot of sense. If you have a workload that is composed of a lot of random writes that span all available space, Fusion Drive isn't for you. Given that most such workloads are confined to the enterprise space, that shouldn't really be a concern here.

Management Granularity

Much of Apple’s marketing on Fusion Drive talks about moving data at the file and application level, but in reality data can be moved between the SSD and HDD portions in 128KB blocks.

Ars actually confirmed this a while ago, but I wanted to see for myself. Using fs_usage I got to see the inner workings of Apple's Fusion Drive. Data is moved between drives in 128KB blocks, likely determined by frequency of use of those blocks. Since client workloads tend to be fairly sequential (or pseudo-random at worst) in nature, it's a safe bet that if you're accessing a single LBA within a 128KB block that you're actually going to be accessing more LBAs in the same space. The migration process seems to happen mostly during idle periods, although I have seen some movement between drives during light IO.

What’s very interesting is just how quickly the migration is triggered after a transfer occurs. As soon as file copy/creation, application launch or other IO activity completes, there’s immediate back and forth between the SSD and HDD. As you fill up the Fusion Drive, the amount of data moved between the SSD and HDD shrinks considerably. Over time I suspect this is what should happen. Infrequently accessed data should settle on the hard drive and what really matters will stay on the SSD. Apple being less aggressive about evicting data from the SSD as the Fusion Drive fills up makes sense.

The migration process itself is pretty simple with data being marked for promotion/demotion, it being physically copied to the new storage device and only then is it moved. In the event of a power failure during migration there shouldn't be any data loss caused by the Fusion Drive, it looks like only after two copies of the 128KB block are in place is the source block removed. Apple told me as much last year, but it's good to see it for myself.

By moving data in 128KB blocks between the HDD and SSD, Apple enjoys the side benefit of partially defragmenting the SSD with all writes to it. Even though the Fusion Drive will prefer the SSD for all incoming writes (which can include smaller than 128KB, potentially random/pseudo-random writes), any migration from the HDD to the SSD happens as large block sequential writes, which will trigger a garbage collection/block recycling routine in cases of a heavily fragmented drive. Performance of the SSD can definitely degrade over time, but this helps keep it higher than it would otherwise given that the SSD is almost always running at full capacity and the recipient of all sorts of unrelated writes. As I mentioned earlier, I would’ve preferred a controller with more consistent IO latency or for Apple to set aside even more of the PM830’s NAND as spare area. I suspect cost was the deciding factor in sticking with the standard amount of overprovisioning.

The Application Experience

By this point I’ve talked a lot about the synthetic experience with Apple’s Fusion Drive, but what about the real world user experience? In short, it’s surprisingly good. While I would describe most SSD caching implementations I’ve used as being more HDD-like than SSD-like, Apple’s Fusion Drive ends up almost half way between a HDD experience and an SSD experience.

Installing anything of reasonable size almost always goes to the SSD first, which really goes a long way towards making Fusion Drive feel SSD-like. This isn’t just true of application installs, but copying anything in general hits the SSD first. The magic number appears to be 4GB, although with a little effort you can get the Fusion Drive to start writing to the HDD after only 1 - 2GB. I used Iometer to create a sequential test file on the Fusion Drive, monitored when the file stopped writing to the SSD, stopped the process, renamed the file and started the file creation again. The screenshot below gives you a good idea of the minimum amount of space Apple will keep on the SSD for incoming writes:

You can see that if you’re quick enough you can easily drop below 2GB of writes to the SSD before the HDD takes over. I don’t know for a fact that this is the amount of free space on the SSD, but that’s likely what it is since there’s no sense in exposing a 121GB SSD and not using it all.

In most real world scenarios where you’re not aggressively trying to fill the SSD, Fusion Drive will keep at least 4GB of the SSD free. Note that when you first use a mostly empty Fusion Drive almost anything you write to the drive, of any size, will go straight to the SSD. As capacity pressure increases however, Apple’s policy shifts towards writing up to 4GB of any given file to the SSD and the remainder onto the hard drive.

I confirmed this by installing Apple's OS X developer tools as well as Xcode itself. The latter is closer to the magic 4GB crossover point, but the bulk of the application ended up on the SSD by default.

The same is true for data generated by an application. I used Xcode to build Adium, a 682MB project, and the entire compile process hit the SSD - the mechanical side of the Fusion Drive never lifted a finger. I tried building a larger project, nearly 2GB of Firefox. In this case, I did see a very short period of HDD activity but the vast majority was confined to the SSD.

I grabbed a large video file (> 10GB) I cloned over when I migrated my personal machine to the iMac and paid attention to its behavior as I copied the file to a new location. For the first 2GB of the transfer, the file streamed from the SSD and went back to the SSD. For the next 2GB of the transfer, the file was being read off of the HDD and written to the SSD. After copying around 4GB, both the source and target became the HDD instead. Fusion Drive actually ended up caching way more of that large video than I thought it would. In my opinion the right move here would be to force all large files onto the hard drive by default unless they were heavily accessed. Apple's approach does seem to be a reasonable compromise, but it's still way more aggressive at putting blocks on the SSD than I thought it would be.

I repeated the test with a different video file that I had never accessed and got a completely different result. The entire file was stored on the hard drive portion of the Fusion Drive. I repeated the test once more with my iPhoto library, which I had been accessing a bunch. To my surprise, the bulk of my iPhoto Library was on the HDD but there were a few bursts of reads to the SSD while I was copying it. In both cases, the copy target ended up being the SSD of course.

My AnandTech folder is over 32GB in size and it contains text, photos, presentations, benchmark results and pretty much everything associated with every review I’ve put together. Although this folder is very important, the truth is that the bulk of that 32GB is never really accessed all that frequently. I went to duplicate the folder and discovered that almost none of it resided on the SSD. The same was true for my 38GB Documents folder, the bulk of which, again, went unread.

Applications on the other hand were almost always on the SSD.

In general, Apple’s Fusion Drive appears to do a fairly good job of automating what I typically do manually: keeping my OS and applications on the SSD, and big media files on the HDD. About the only difference between how I manually organize my data and how Fusion Drive does it is I put my documents and AnandTech folder on my SSD by default. I don’t do this just for performance, but more for reliability. My HDD is more likely to die than my SSD.

Putting Fusion Drive’s Performance in Perspective

Benchmarking Fusion Drive is a bit of a challenge since it prioritizes the SSD for all incoming writes. If you don’t fill the Fusion Drive up, you can write tons of data to the drive and it’ll all hit the SSD. If you do fill the drive up and test with a dataset < 4GB, then you’ll once again just measure SSD performance.

In trying to come up with a use case that spanned both drives I stumbled upon a relatively simple one. By now my Fusion Drive was over 70% full, which meant the SSD was running as close to capacity as possible (save its 4GB buffer). I took my iPhoto library with 703 photos and simply exported all photos as TIFFs. The resulting files were big enough that by the time I hit photo 297, the 4GB write buffer on the SSD was full and all subsequent exported photos were directed to the HDD instead. I timed the process, then compared it to results from a HDD partition on the iMac as well as compared to a Samsung PM830 SSD connected via USB 3.0 to simulate a pure SSD configuration. The results are a bit biased in favor of the HDD-only configuration since the writes are mostly sequential:

The breakdown accurately sums up my Fusion Drive experience: nearly half-way between a hard drive and a pure SSD configuration. In this particular test the gains don't appear all that dramatic, but again that's mostly because we're looking at relatively low queue depth sequential transfers. The FD/HDD gap would grow for less sequential workloads. Unfortunately, I couldn't find a good application use case to generate 4GB+ of pseudo-random data in a repeatable enough fashion to benchmark.

If I hammered on the Fusion Drive enough, with constant very large sequential writes (up to 260GB for a single file) I could back the drive into a corner where it would no longer migrate data to the SSD without a reboot (woohoo, I sort of broke it!). I suspect this is a bug that isn't triggered through normal automated testing (for obvious reasons), but it did create an interesting situation that I could exploit for testing purposes.

Although launching any of the iMac's pre-installed applications frequently used by me proved that they were still located on the SSD, this wasn't true for some of the late comers. In particular, Photoshop CS6 remained partially on the SSD and partially on the HDD. It ended up being a good benchmark for pseudo-random read performance on Fusion Drive where the workload is too big (or in this case, artificially divided) to fit on the SSD partition alone. I measured Photoshop launch time on the Fusion Drive, a HDD-only partition and on a PM830 connected via USB 3.0. The results, once again, mirrored my experience with the setup:

Fusion Drive delivers a noticeable improvement over the HDD-only configuration, speeding up launch time by around 40%. A SSD-only configuration however cuts launch time in more than half. Note that if Photoshop were among the most frequently used applications, it would get moved over to the SSD exclusively and deliver performance indistinguishable from a pure SSD configuration. In this case, it hadn't because my 1.1TB Fusion Drive was nearly 80% full, which brings me to a point I made earlier:

The Practical Limits of Fusion Drive

Apple's Fusion Drive is very aggressive at writing to the SSD, however the more data you have the more conservative the algorithm seems to become. This isn't really shocking, but it's worth pointing out that at a lower total drive utilization the SSD became home to virtually everything I needed, but as soon as my application needs outgrew what FD could easily accommodate the platform became a lot pickier about what would get moved onto the SSD. This is very important to keep in mind. If 128GB of storage isn’t enough for all of your frequently used applications, data and OS to begin with, you’re going to have a distinctly more HDD-like experience with Fusion Drive. To simulate/prove this I took my 200GB+ MacBook Pro image and moved it over to the iMac. Note that most of this 200GB was applications and data that I actually used regularly.

By the end of my testing experience, I was firmly in the category where I needed more solid state storage. Spotlight searches took longer than on a pure SSD configuration, not all application launches were instant, adding photos to iPhoto from Safari took longer, etc... Fusion Drive may be good, but it's not magic. If you realistically need more than 128GB of solid state storage, Fusion Drive isn't for you.

Final Words

For the first time since late 2008, I went back to using a machine where a hard drive was a part of my primary storage - and I didn’t hate it. Apple’s Fusion Drive is probably the best hybrid SSD/HDD solution I’ve ever used, and it didn’t take rocket science to get here. All it took was combining a good SSD controller (Samsung’s PM830), with a large amount of NAND (128GB) and some very aggressive/intelligent software (Apple’s Core Storage LVM). Fusion Drive may not be fundamentally new, but it’s certainly the right way to do hybrid storage if you’re going to do it.

It seems that Fusion Drive is really made for the user who doesn't necessarily have a ton of applications/data, but does have a reasonable sized media collection. For that user, Fusion Drive should be a reasonable approximation of a well managed SSD/HDD setup with your big media files going to the HDD and everything that you launch frequently living on the SSD. I’m always going to ask for a larger cache, but I do believe that 128GB is a good size for most client workloads and usage models today. For me in particular I’d probably need a 256GB cache for Fusion Drive to win me over, but I understand that I’m not necessarily the target market here.

The real question is whether or not it’s worth it. I’m personally a much bigger fan of going all solid state and manually segmenting your large media files onto HDD arrays, but perhaps that’s me being set in my ways (or just me being right, not sure which one). Fusion Drive doesn’t do anything to mitigate the likelihood that a hard drive will likely fail sooner than a good SSD, whereas if you go with an internal SSD and external (Thunderbolt or USB 3.0) HDD RAID array you can control your destiny a bit better. Unfortunately, in situations where Fusion Drive is a choice, you don’t often have that flexibility.

On the iMac, Apple limits your options quite a bit. You can either buy a hard drive or the Fusion Drive on the 21.5-inch model, there’s no standalone SSD option. There the choice is a no-brainer. If you’re not going to buy your own SSD and replace the internal HDD with it (or try to see if OWC’s rMBP SSD fits), then the Fusion Drive is absolutely right choice. You’re paying handsomely for the right ($250 for 128GB of NAND is very 2011), but if you’re not willing to crack open the iMac case this is really the only way to go.

For the 27-inch iMac the decision is similarly difficult. Apple does offer a standalone SSD option, but it’s for a 768GB model that will set you back $1300. All of the sudden that $250 Fusion Drive upgrade sounds a lot more reasonable.

On the Mac mini side the decision is far simpler. The Fusion Drive is only available on the $799 configuration (for $250) but so is a 256GB SSD upgrade for $300. As long as you’re ok with using an external hard drive for mass storage, here I’d go for the big standalone SSD. The usual caveat applies: this  is only true if you’re not interested in cracking open the mini yourself and using a 3rd party SSD.

To make things simpler, I made bold the options I'd choose given Apple's current lineup in the table below. Note that this is still assuming you're not going down the DIY route (if you do go down that path, buy the biggest SSD you can find and rely on some external mass storage for everything else):

I am curious to see how long of a roadmap Fusion Drive has ahead of it. Will NAND get cheap/large enough that even the iMac can move to it exclusively? Or will we end up with systems that have more than enough NAND to easily store everything but large media files for even the most demanding of power users? In less than a year Apple could double the size of the NAND used in Fusion Drive at no real change to cost. I suspect another doubling beyond that would be necessary to really make Fusion Drive a one size fits all, but then we're talking ~2 years out at this point and I don't know how static everyone's usage models will remain over that period of time. Go out even further in time, to the post-NAND era and there are some really revolutionary things that can happen to the memory hierarchy altogether...