Neat, this is basically EMC's "FAST VP" Auto-tiering mechanism when LUN's are set to "write high then auto-tier" at least in regards to the write cache. I wonder if this functions as a read cache as well or if it is strictly write cache.
Nice to see "smaller guys" introducing Enterprise level features like this any how.
Except that enterprise SANs don't bottleneck performance to keep parity and cache through. They also store data on RAW disks and use off-disk access tables to keep performance extra high. This sounds like a colossal waste of space with no real benefit until you're into large arrays, where 5+0 is probably a better, safer solution for anyone not using a SAN.
Yea, I don't see enough info to tell how good idea their use of RAID levels are, but having one disk for it might be an issue. In an EMC world those +2TB 7200RPM disks should be RAID 6 (typically 6+2) since the long rebuild time increase risks of double drive failures, so I'd be worried about having one parity disk. I also don't see mention of Hot spares.
This use of the Cache SSD's is great for buffering writes to the disks, it also can keep you from shooting your self in the foot when it comes to over-committing the NL-SAS or SATA drives.
One common issue is overloading the SATA drives, this can cause array write cache to fill and eventually make the array pause incoming write IO until the write cache is written to disk. Having a layer of SSD to write to can be useful, though it is dependent on how the SSD tier is used. Does it fully flush overnight? Is it treated strictly as a write cache tier, or is it an active data tier?
The cache drive is strictly a write cache. It is pretty simple. All shares designated to use the cache drive record writes on the cache drive. At a designated time, the cache drive contents are written to the parity protected array. Currently I have my cache drive set to transfer at around 3 am.
As for the parity disk, it is possible to run a raid 1 array for the parity drive. For years Limetech has promised a 2nd parity disk option, but that has yet to materialize. Double drive failures are still incredibly rare. If you browse the forums, you'll only find a handful of cases where this has happened without some special circumstance. Also, the contents of the data drives are completely readable if two drives fail. So if you parity + 1 data drive fail, you only lose one drive worth of data. The only way to lose all your data is for all of your drives to fail.
Thanks, that sounds an aweful lot like a simplified EMC VNX tiering structure; though strictly in a "tier down" scenario where data is auto-tiered at night (and only drops to lower storage).
What kind of what kind of support is there for hot spares? or is RAID 1 for parity drives the "additional protection" offered?
I'd be a bit worried about drive loss while rebuilding is taking place post drive-loss, especially if I was out on vacation or couldn't get a part ordered in a timely manner.
I use a hot spare in the sense that I usually have a spare disk ready to replace a failed drive immediately upon failure. Unraid requires that the drive be written entirely with 0's before being added to the array. They call this preclearing. There is a helpful preclear script available to preclear drives without adding them to the array. The community typically use this script to stress new drives and weed out drives with manufacturer defects. If you do several cycles on a >4TB drive, it will take well over a week of constant sequential read and write operations - so it puts the drive through its paces. Once you've completed that, you can either leave the drive connected, but not added to the array - a so-called hot spare. Rebuilding is pretty fast, I usually rebuild at a rate of 80-90MB/s. You can also use this spare drive as your cache drive, requiring only that you copy the data from the cache before you add it to the array.
Otherwise, a RAID 1 parity drive setup is the only other hot spare scenario I can think of; however, since this only protects the parity drive from failure (and not the 12 other drives the array - in my case), I don't see it as offering significantly improved failure protection. In fact, a RAID 1 would make more sense on the data drives because if two data drives failed you would lose the data on those two drives, but if a data drive and the parity drive fail, you only lose the data on the one drive. This is impractical IMO though.
The Linux kernel has two similar systems built in to do this cache trick already.
I honestly don't see how they can even guarantee the recovery of data without striping. The point of striping is that you have parity and half the data already so calculating the other half is doable.
striping has nothing to do with data recovery. Parity is what is used to do data recovery. The only difference between us an a RAID 5 is that with RAID 5, you stripe both data and parity across all disks in the group. With us, parity is left on ONE disk only and data isn't striped either (whole files are written to individual disks).
It is treated as strictly a write cache tier, but some data can be forced to "live" in the cache. Specifically metadata for applications and virtual disk images for live VMs tend to live in the cache as performance is better.
Flex raid is also a great option for media archives. The snap shot raid is what I use. I store the parity in the disks I chose. I don't need to do anything to the existing data and I can have as many parity disks as I chose.
I'm sure it's not a fair comparison, but I chose to run with StableBit for my WHS a few years ago instead of unRAID, FreeNAS, or FlexRAID and it's been really great. For my home file/media server, I'm just not sure the extra features are worthwhile. I really need to educate myself on VT to see what it can do for me.
I've been doing the same and it's worked great; but if I go with Windows for my next nas (vs FreeNAS, Synology, or QNAP) at the end of the year I'll probably just use the W10 version of the drive pool features added in W8.
Have they fixed the performance problems? The reason I didn't migrate to it was most folks were reporting single-digit reads/writes. As far as I know, Storage Spaces doesn't automatically monitor SMART data, scan disks, or move data off of failing drives like StableBit does. This feature alone saved me a lot of data loss from multiple drive failures.
Not really, their "fix" was to add SSD caching as a feature of Storage Spaces, which makes the whole setup overpriced and still underperforms compared to Linux software RAID. You still can't expand the parity pool in a flexible manner (if you started with 5 drives, you have to increase with 5 drive increments), and the pool still doesn't rebalance when capacity is added. Avoid it at all costs.
I plan on doing the same, but will probably not use Windows Storage Spaces. I don't like the data being unreadable if their is any corruption. There are a lot of nightmare stories out there about people trying to recover data from a failed storage space. At least with stablebit, you can always read the data on the drive no matter what. It's worth the $20.
I'm also running stablebit drivepool in my WHS server, with 9 hard drives. I bought WHS for $59, and run it from an 80GB SSD, and then have a drive pool of 8 drives of various sizes (even a few laptop drives). The best thing I like about Stablebit is that it uses the normal windows file system - you can pull any drive out and insert it into another computer, and read the files as normal. It automatically backs up 8 computers in the house, and runs 2 Minecraft servers, as well as the normal media stuff (Airvideo and Plex server). I'll probably go to Windows 10 for my next NAS, and buy DrivePool again - I don't want to use the Windows built-in drive pool, since the drives are not readable when taken out of the pool. I just have to figure out how to do automatic backups of the other computers int eh house (Ur backup?)
I've been using the built-in WHS backup system to store image backups of other computers on the network. Then I have an external HDD for critical backup data (personal files/pictures). Carbonite keeps a continuous online backup of everything the server, except I don't bother with DVD/Blu-ray MKVs since I the original discs are the backup (and it would take months to upload it all and download it all). Carbonite also backs up all our mobile devices for no additional cost, so that's been a pretty solid purchase ($60/yr).
I am thinking about trying BackBlaze or CrashPlan - one of those two will let you backup offsite to a friend/family's computer and vice versa, but I forget which one. That way I can keep some HDDs in my friend's server as a large off-site backup, but much easier faster/easier than shipping drives or re-downloading everything.
2 GB of the parity drive contains parity data across all 4 data drives, the 3rd contains parity across 3 drives (2x4gb, 1x gb), and the final GB of the parity drive has 2way parity across the 2 4gb data drives.
I've never understood the point of unRAID. Most people I've encountered who were using it - and making lots of noise about only needing / losing 1 drive for parity - are now moving to systems that support ZFS due to greater data protection. I guess it's like a starter NAS?
sounds very insteresting. it's love to put this on my custom linux nas. but they really missed the mark with xfs. abandoned that years ago. today its either zfs, btrfs or the highway...
You have the choice of Reiserfs, XFS, BTRFS for any given disk. You can mix/match drive sizes/speeds. Moving the Data Disks/Array to another hardware platform is VERY easy. The Cache (or Cache pool) can be set to write to the array at any time (or times). If you write over the size of the Cache it will begin writing to the array as if you did not have a cache enabled for any given share.
The system has easy hooks for full blown gaming VMs, VPN services, Database hosting, compilers, linux scripts, notifications, home automation, etc. I have 1 ultra-low power server backing up to a powerful VM host server. The low power server contains all of my personal files in a share and business files in another share – I use Reiserfs (which has some great recovery tools) for these disks. The powerful machine (which uses XFS for the data and BTRFS for the mirrored Cache pool) backs up the business files to another offsite unraid server, it also uses Crashplan to backup my personal files to the code42/crashplan cloud.
I have been running Unraid for many years and many versions and It needs to be mentioned that Unraid is profoundly simple and totally stable, I have never lost a single bit of data on either of my two servers, I have never even had a disk fail. most disks are powered down most of the time. All disks in an Unraid arrays have very low hours because of the selective spin up and down. I have replaced (with rebuild) may disks to increase capacity and i just cant say enough about how simple and reliable the Unraid system is.
We’ve updated our terms. By continuing to use the site and/or by logging into your account, you agree to the Site’s updated Terms of Use and Privacy Policy.
Facebook
Twitter
RSS
25 Comments
Back to Article
trueg50 - Monday, July 27, 2015 - link
Neat, this is basically EMC's "FAST VP" Auto-tiering mechanism when LUN's are set to "write high then auto-tier" at least in regards to the write cache. I wonder if this functions as a read cache as well or if it is strictly write cache.Nice to see "smaller guys" introducing Enterprise level features like this any how.
blaktron - Monday, July 27, 2015 - link
Except that enterprise SANs don't bottleneck performance to keep parity and cache through. They also store data on RAW disks and use off-disk access tables to keep performance extra high. This sounds like a colossal waste of space with no real benefit until you're into large arrays, where 5+0 is probably a better, safer solution for anyone not using a SAN.trueg50 - Monday, July 27, 2015 - link
Yea, I don't see enough info to tell how good idea their use of RAID levels are, but having one disk for it might be an issue. In an EMC world those +2TB 7200RPM disks should be RAID 6 (typically 6+2) since the long rebuild time increase risks of double drive failures, so I'd be worried about having one parity disk. I also don't see mention of Hot spares.This use of the Cache SSD's is great for buffering writes to the disks, it also can keep you from shooting your self in the foot when it comes to over-committing the NL-SAS or SATA drives.
One common issue is overloading the SATA drives, this can cause array write cache to fill and eventually make the array pause incoming write IO until the write cache is written to disk. Having a layer of SSD to write to can be useful, though it is dependent on how the SSD tier is used. Does it fully flush overnight? Is it treated strictly as a write cache tier, or is it an active data tier?
3DoubleD - Monday, July 27, 2015 - link
The cache drive is strictly a write cache. It is pretty simple. All shares designated to use the cache drive record writes on the cache drive. At a designated time, the cache drive contents are written to the parity protected array. Currently I have my cache drive set to transfer at around 3 am.As for the parity disk, it is possible to run a raid 1 array for the parity drive. For years Limetech has promised a 2nd parity disk option, but that has yet to materialize. Double drive failures are still incredibly rare. If you browse the forums, you'll only find a handful of cases where this has happened without some special circumstance. Also, the contents of the data drives are completely readable if two drives fail. So if you parity + 1 data drive fail, you only lose one drive worth of data. The only way to lose all your data is for all of your drives to fail.
trueg50 - Monday, July 27, 2015 - link
Thanks, that sounds an aweful lot like a simplified EMC VNX tiering structure; though strictly in a "tier down" scenario where data is auto-tiered at night (and only drops to lower storage).What kind of what kind of support is there for hot spares? or is RAID 1 for parity drives the "additional protection" offered?
I'd be a bit worried about drive loss while rebuilding is taking place post drive-loss, especially if I was out on vacation or couldn't get a part ordered in a timely manner.
3DoubleD - Tuesday, July 28, 2015 - link
I use a hot spare in the sense that I usually have a spare disk ready to replace a failed drive immediately upon failure. Unraid requires that the drive be written entirely with 0's before being added to the array. They call this preclearing. There is a helpful preclear script available to preclear drives without adding them to the array. The community typically use this script to stress new drives and weed out drives with manufacturer defects. If you do several cycles on a >4TB drive, it will take well over a week of constant sequential read and write operations - so it puts the drive through its paces. Once you've completed that, you can either leave the drive connected, but not added to the array - a so-called hot spare. Rebuilding is pretty fast, I usually rebuild at a rate of 80-90MB/s. You can also use this spare drive as your cache drive, requiring only that you copy the data from the cache before you add it to the array.Otherwise, a RAID 1 parity drive setup is the only other hot spare scenario I can think of; however, since this only protects the parity drive from failure (and not the 12 other drives the array - in my case), I don't see it as offering significantly improved failure protection. In fact, a RAID 1 would make more sense on the data drives because if two data drives failed you would lose the data on those two drives, but if a data drive and the parity drive fail, you only lose the data on the one drive. This is impractical IMO though.
toyotabedzrock - Tuesday, July 28, 2015 - link
The Linux kernel has two similar systems built in to do this cache trick already.I honestly don't see how they can even guarantee the recovery of data without striping. The point of striping is that you have parity and half the data already so calculating the other half is doable.
jonpanozzo - Tuesday, July 28, 2015 - link
striping has nothing to do with data recovery. Parity is what is used to do data recovery. The only difference between us an a RAID 5 is that with RAID 5, you stripe both data and parity across all disks in the group. With us, parity is left on ONE disk only and data isn't striped either (whole files are written to individual disks).jonpanozzo - Tuesday, July 28, 2015 - link
It is treated as strictly a write cache tier, but some data can be forced to "live" in the cache. Specifically metadata for applications and virtual disk images for live VMs tend to live in the cache as performance is better.SlyNine - Monday, July 27, 2015 - link
Flex raid is also a great option for media archives. The snap shot raid is what I use. I store the parity in the disks I chose. I don't need to do anything to the existing data and I can have as many parity disks as I chose.nathanddrews - Monday, July 27, 2015 - link
I'm sure it's not a fair comparison, but I chose to run with StableBit for my WHS a few years ago instead of unRAID, FreeNAS, or FlexRAID and it's been really great. For my home file/media server, I'm just not sure the extra features are worthwhile. I really need to educate myself on VT to see what it can do for me.DanNeely - Monday, July 27, 2015 - link
I've been doing the same and it's worked great; but if I go with Windows for my next nas (vs FreeNAS, Synology, or QNAP) at the end of the year I'll probably just use the W10 version of the drive pool features added in W8.nathanddrews - Monday, July 27, 2015 - link
Have they fixed the performance problems? The reason I didn't migrate to it was most folks were reporting single-digit reads/writes. As far as I know, Storage Spaces doesn't automatically monitor SMART data, scan disks, or move data off of failing drives like StableBit does. This feature alone saved me a lot of data loss from multiple drive failures.Gigaplex - Monday, July 27, 2015 - link
Not really, their "fix" was to add SSD caching as a feature of Storage Spaces, which makes the whole setup overpriced and still underperforms compared to Linux software RAID. You still can't expand the parity pool in a flexible manner (if you started with 5 drives, you have to increase with 5 drive increments), and the pool still doesn't rebalance when capacity is added. Avoid it at all costs.kmmatney - Monday, July 27, 2015 - link
I plan on doing the same, but will probably not use Windows Storage Spaces. I don't like the data being unreadable if their is any corruption. There are a lot of nightmare stories out there about people trying to recover data from a failed storage space. At least with stablebit, you can always read the data on the drive no matter what. It's worth the $20.kmmatney - Monday, July 27, 2015 - link
I'm also running stablebit drivepool in my WHS server, with 9 hard drives. I bought WHS for $59, and run it from an 80GB SSD, and then have a drive pool of 8 drives of various sizes (even a few laptop drives). The best thing I like about Stablebit is that it uses the normal windows file system - you can pull any drive out and insert it into another computer, and read the files as normal. It automatically backs up 8 computers in the house, and runs 2 Minecraft servers, as well as the normal media stuff (Airvideo and Plex server). I'll probably go to Windows 10 for my next NAS, and buy DrivePool again - I don't want to use the Windows built-in drive pool, since the drives are not readable when taken out of the pool. I just have to figure out how to do automatic backups of the other computers int eh house (Ur backup?)nathanddrews - Monday, July 27, 2015 - link
I've been using the built-in WHS backup system to store image backups of other computers on the network. Then I have an external HDD for critical backup data (personal files/pictures). Carbonite keeps a continuous online backup of everything the server, except I don't bother with DVD/Blu-ray MKVs since I the original discs are the backup (and it would take months to upload it all and download it all). Carbonite also backs up all our mobile devices for no additional cost, so that's been a pretty solid purchase ($60/yr).I am thinking about trying BackBlaze or CrashPlan - one of those two will let you backup offsite to a friend/family's computer and vice versa, but I forget which one. That way I can keep some HDDs in my friend's server as a large off-site backup, but much easier faster/easier than shipping drives or re-downloading everything.
Mr Perfect - Monday, July 27, 2015 - link
How does the parity drive work, exactly? In the example shown, there are 13TBs of unique data storage and only 4TB of parity.DanNeely - Monday, July 27, 2015 - link
2 GB of the parity drive contains parity data across all 4 data drives, the 3rd contains parity across 3 drives (2x4gb, 1x gb), and the final GB of the parity drive has 2way parity across the 2 4gb data drives.Navvie - Tuesday, July 28, 2015 - link
I've never understood the point of unRAID. Most people I've encountered who were using it - and making lots of noise about only needing / losing 1 drive for parity - are now moving to systems that support ZFS due to greater data protection. I guess it's like a starter NAS?bernstein - Tuesday, July 28, 2015 - link
sounds very insteresting. it's love to put this on my custom linux nas. but they really missed the mark with xfs. abandoned that years ago. today its either zfs, btrfs or the highway...jonpanozzo - Tuesday, July 28, 2015 - link
As an FYI, dual-parity is being added in an upcoming release, which should address the concerns of anyone worried about multiple disk failures.Landspeeder - Tuesday, July 28, 2015 - link
I actually have 2 unraid servers at home.You have the choice of Reiserfs, XFS, BTRFS for any given disk. You can mix/match drive sizes/speeds. Moving the Data Disks/Array to another hardware platform is VERY easy. The Cache (or Cache pool) can be set to write to the array at any time (or times). If you write over the size of the Cache it will begin writing to the array as if you did not have a cache enabled for any given share.
The system has easy hooks for full blown gaming VMs, VPN services, Database hosting, compilers, linux scripts, notifications, home automation, etc. I have 1 ultra-low power server backing up to a powerful VM host server. The low power server contains all of my personal files in a share and business files in another share – I use Reiserfs (which has some great recovery tools) for these disks. The powerful machine (which uses XFS for the data and BTRFS for the mirrored Cache pool) backs up the business files to another offsite unraid server, it also uses Crashplan to backup my personal files to the code42/crashplan cloud.
Dual parity is being worked on now.
ezorb - Tuesday, July 28, 2015 - link
I have been running Unraid for many years and many versions and It needs to be mentioned that Unraid is profoundly simple and totally stable, I have never lost a single bit of data on either of my two servers, I have never even had a disk fail. most disks are powered down most of the time. All disks in an Unraid arrays have very low hours because of the selective spin up and down. I have replaced (with rebuild) may disks to increase capacity and i just cant say enough about how simple and reliable the Unraid system is.Nehemoth - Thursday, July 30, 2015 - link
I would like to see a guy to build a DIY NAS on the cheap here @Anandtech