The drop down box I'm seeing on WP8 goes off my screen, and changing the selection doesn't change the graph. The usual button type picker you use for SSD reviews and such works fine.
Tons of people still use RAID 5 in the enterprise. Further more lets call it by its real name RAID 10 instead of OBR10. You can get even further redundancy from RAID 50, RAID 60 and RAID 100.
No it doesnt. Not at all. First, you are taking into account only huge arrays used to store data and not to run applications (so basically only mechanical SATA, that is).Second, you are completeley ignoring costs (raid 5 or raid 6 vs raid 10). Third, you are assuming the raid 5 itself is not backed up or with some sort of software\hardware redundancy or tiering at lower levels (see SANs).
So while I can agree that THEORETICALLY having raid 10 everywhere would indeed be safer, the costs (hdds + enclosures + controllers + backplanes) make this, and this time for real, have zero sense.
"Resilvering" is the ZFS term for rebuilding data on a volume. It's very much a current term still, but it does give us an insight into the current bias of the author, who apparently favors ZFS for his storage until something he proposes as better is golden.
How many times have you had to rebuild a RAID5 in your lifetime? I have over 100 times on over 10 major HARDWARE RAID vendors.
"And when you go to rebuild that huge RAID 5 array and another disk fails your screwed."
The other drive failing is a very small possibility in an enterprise environment that I was talking about, because of enterprise grade drives vs consumer. That is why most have either the raid taken offline for a much faster rebuild. Besides during that rebuild the RAID is still functional just degraded.
Also my point is lots of us still have hardware that is 2-5 years old that is still just working. The newest Arrays that I have setup as of late are 20 to 150 TB in size and we went with Freenas with ZFS which puts all other to shame. NetApp Storage appliances rebuild times are quite fast 6-12 hours for 40TB LUNS. It all depends upon the redundancy that you need. Saying that raid 5 needs to die is asinine. What if the data you are storing is all available in the public domain but have a local copy speeds up the data access rates. The rebuild is faster with a degraded LUN vs retrieving all of the data from the public domain again. There are many use cases for each RAID level just because one level does not fit YOUR uses it does not need to die!
So if you were to buy this NAS for a new implementation would you even consider throwing 10-12 disks in it and building a RAID 5 array? just asking. Even in your own post you state how you use Freenas with ZFS for your new arrays. RAID 5 is the dodo here let it go extinct.
For all you know, he's running ZFS using raidz1 (RAID5 essentially). Also, saying RAID5 needs to die, one must then assume you also think RAID0 is beyond worthless, since it has NO redundancy? Obviously, you can (hopefully) cite the use-cases for RAID0. Your bias just prevents you from seeing the usefulness of RAID5.
It does happen though. I've had to rebuild 2 servers alone this year because of multiple drive failures. One server had 3 drives fail. But that's because of neglect. Us engineers only have so much time. Especially with the introduction to lean manufacturing.
RAID 5 + Global spare though is usually pretty safe bet if it's a critical app server. Otherwise RAID 5 is perfectly fine.
In at least 2 of the links it is stated that when a RAID 5 rebuild is in process and a URE is encountered, all data is lost.
I'm not sure what crap hardware that author has been using. When a URE is encountered during a rebuild the rebuild halts and you're back where you started - with a degraded array.
Now I'm not saying "RAID 5 is the BEST!" but the "facts" presented are false.
FYI I've rebuilt several hundred RAID 5 arrays over the last 15 years and have experienced a URE during rebuild exactly 2 times. You can cut down on UREs by performing a scheduled "Patrol Read" or functional equivalent. There is no way to know if the data is readable unless you read it. You can have a (fictional) "SUPER DUPER RAID 3000" with a ridiculous amount of redundancy but it's still theoretically possible to lose your data due to URE unless it's read and verified.
The second link is quite wrong in the premises. It says that filesystems are really reliable today, well, they are not. There are lot of research showing how all filesystems are flawed today (except ZFS) with respect to data corruption protection. Only ZFS protects the data against data corruption. Read the research papers you will find here: http://en.wikipedia.org/wiki/ZFS#Data_integrity http://en.wikipedia.org/wiki/Hard_disk_error_rates...
2000+ MB/s ethernet interface (2x10Gb/s), 10 hard-drives able to to delivers at least 500MB/s EACH (grand total of 5000MB/s), Xeon quad-core CPU, and tested with ONE client, it delivers less than 120MB/s?!? That's what I expect from an USB 3 2.5" external hard-drive, not a SAN of this price, it's totally deceptive!
For the few who know, I am the Brent that brought up Raid 5 on the Mike Tech Show (saying how it is not the way to go in any case)
Raid 10 is the performance king, Raid 1 is great for cheap redundancy, and Raid 10, or OBR10, should be what everyone uses in big sets. If you need all the disk capacity, use Raid 6 instead of Raid 5 because if a drive fails during a rebuild, then you lose everything. Raid 6 is better because you can lose a drive. Rebuilding is a scary process with Raid 5, but Raid 1 or 10, it is literally copying data from 1 disk to another.
From the drives' perspective, rebuilding a RAID 5 array is exactly the same as rebuilding a RAID 1 or 10 array: Read the whole disk(s) (or to be more exact, sectors with data) once, and write the whole target disk once. It is only different for the controller. I fail to see why is one scarier than the other.
If your drive fails while rebuilding a RAID 1 array, you are exactly as screwed. The only thing why R5 is worse here is because you have n-1 disks unprotected while rebuilding, not just one, giving you approximately (=negligibly smaller than) n-1 times data loss chance.
Rebuilding a Raid 5 requires reading data from all of the other disks, whereas Raid 10 requires reading data from 1 other drive. Raid 1 rebuilds are not complex, nor Raid 10. Raid 5/6 rebuilding is complex, requires activity from other disks, and because of the complexity has a higher chance of failure.
I've got one of these with 3 extra shelves of disks and 1TB of SSD cache. There's a limit of 3 shelves in a single volume, but 120TB (3 shelves of 12 4Tb disks, raid5 on each shelf) with the SSD cache performs pretty well. For reference, NFS performance is substantially better than CIFS or iSCSI.
It copes fine with the 150 virtual machines that support a 20 person development team.
So much cheaper than a NetAPP or similar - but I haven't had a chance to test the multi-NAS failover - to see if you truly get enterprise quality resilience.
well at least half a dozen morons got schooled on the different types of RAID arrays. gg, always glad to see the experts put the "less informed" (okay i'm getting nicer) ppl in their place.
I'd be interested in knowing greater detail on the link aggregation setup. There's no mention of the load balancing configuration in particular. The reason I ask is because it's probably *not* a good idea to bond 1Gbps links with 10Gbps links in the same bundle unless you have access to more advanced algorithms (and even then I wouldn't recommend it). The likelihood of limiting a single stream to ~1Gbps is fairly good, and may limit overall throughput depending on the number of clients. It's even possible (though admittedly statistically unlikely) that you could limit the entirety of the system's network performance to saturating a single 1Gbe connection.
The 802.3ad testing in this article is fundamentally flawed. 802.3ad does NOT, repeat NOT, create a virtual link whose throughput is the sum of its components. What it does is provide a mechanism for automatically selecting which link in a set (bundle) to use for a particular packet based on its source and destination. The definition of "source and destination" depends on the particular hashing algorithm you choose, but the common algorithms will all hash a network file system client / server pair to the same link.
In a 4 1Gb/s + 2 10Gb/s 802.3ad ling aggregation group, you would expect that 2/3rd's of the clients would get hashed to the 1 Gb/s links and 1/3rd would get hashed to the 10Gb/s links. In a situation where all clients are running in lock-step (i.e. everyone must complete their tests before moving on to the next), you would expect the 10 Gb/s clients to be limited by the 1 Gb/s ones, thus providing a ~ 6 Gb/s line rate ~= 600 MB/s user data result.
Since 2 * 10 Gb/s > 6 * 1 Gb/s, I recommend retesting with only the two 10 Gb/s links in the 802.3ad aggregation group.
Indeed, that's what I was going to get at when I asked more about the particulars of the setup in question. Thanks for just laying it out, saved me some time. ;)
Thanks for the note. I indeed realized this issue after processing the data for the Synology unit. Our subsequent 10GbE reviews which are slated to go out over the next week or so (the QNAP TS-470 and the Netgear ReadyNAS RN-716) have been evaluated with only the 10GbE links in aggregated mode (and the 1 GbE links disconnected).
I will repeat the Synology multi-client benchmark with RAID-5 / 2 x 10Gb 802.3ad and update the article tomorrow.
I have updated the piece with the graphs obtained by just using the 2 x 10G links in 802.3ad dynamic link aggregation. I believe the numbers don't change too much compared to teaming all the 6 ports together.
Just for more information on our LACP setup:
We are using the GSM7352S's SFP+ ports teamed with link trap and STP mode enabled. Obviously, dynamic link aggregation mode. The Hash Mode is set to 'Src/Dest MAC, VLAN, EType,Incoming Port'.
I did face problems in evaluating other units where having the 1 Gb links active and connected to the same switch while the 10G ports were link-aggregated would bring down the benchmark numbers. I have since resolved that by completely disconnecting the 1G links in multi-client mode for the 10G-enabled NAS units.
Hi all, while I understand that RAID5 surely has its domains, RAID10 is generally a way better choice, both for redundancy and performance.
The RAID5 read-modify-write penalty present itself in a pretty heavy way when using anything doing many small writes, as databases ans virtual machines. So, then only case where I would create a RAID5 array is when it will be used as a storage archive (eg: fileserver).
On the other hand, many, many sysadmins create "by default" RAID5 arrays pretending to consolidate on it many virtual machines. Unless you have a very high-end RAID controller (w/512+ MB of NVCache), they will badly suffer from RAID5 and alignment issues, which are basically non-existent on RAID10.
One exception can be done for SSD arrays: in that case, a parity-based scheme (RAID5 or, better, RAID6) can do its work done very well, as SSD have no seek latency and tend to be of lower capacity than mechanical disks. However, alignment issues remain significant, and need to be taken into account when creating both the array and the virtual machines on top of it.
Since the introduction of 4k sector size disks things have changed a lot, at least in the ZFS world. Everyone who is thinking about building their storage system with ZFS and RaidZ should see this Video.
It depends on the OS implemented if this is a current problem. Many of the commercial ZFS vendors have had this fixed for awhile 18 to 24 months. FreeNAS in its latest release 9.2.0 have fixed this issue. ZFS has been a command-line heavy operation that you really understand drive setup and to tune it for the best speed.
I don't know much about FreeNAS but like FreeBSD they get their ZFS from Illumos. Illumos ZFS implementation has no fix. What is ZFS supposed to do if you write 8k to a RaidZ with 4 data disks if the sector size of a disk is 4k?
The video explain what happens on Illumos. You will end up with something like this
1st 4k data -> disk1 2nd 4k data -> disk2 1st 4k data -> disk3 2nd 4k data -> disk4 Parity -> disk5
So you have written the same data twice plus parity. Much like mirroring with the additional overhead of calculating and writing the parity. FreeNAS has changed the ZFS implementation in that regard?
Yes this is true, but for this very same reason many enterprise disks remain at 512 Byte per sector. Take the "enterprise drives" from WD: - the low cost WD SE are Advanced Format ones (4K sector size) - the somewhat pricey WD RE have classical 512B sector - the top-of-the line WD XE have 512B sector
So, the 4K formatted disks are proposed for storage archiving duties, while for VMs and DBs the 512B disks remain the norm.
Raid5 is fine so long as you keep patrol reads enabled. All those horror stories about failed rebuilds come from people that don't know what patrol reads are for, and turned them off for some stupid reason or another, or were too cheap to buy a real raid controller.
We’ve updated our terms. By continuing to use the site and/or by logging into your account, you agree to the Site’s updated Terms of Use and Privacy Policy.
Facebook
Twitter
RSS
51 Comments
Back to Article
Qiasfah - Thursday, December 26, 2013 - link
The text rendering with the tables and text is messed up in the mobile version :(Ryan Smith - Friday, December 27, 2013 - link
Qiasfah, thank you for letting us know. The article has been tweaked to keep that from happening.YoshoMasaki - Saturday, December 28, 2013 - link
The drop down box I'm seeing on WP8 goes off my screen, and changing the selection doesn't change the graph. The usual button type picker you use for SSD reviews and such works fine.ErrantOpinion - Monday, December 30, 2013 - link
The drop downs work in Internet Explorer, but not Chrome/Opera 15+ for me.Ryan Smith - Tuesday, December 31, 2013 - link
Fixed. Sorry about that. I hadn't tested that code on Chrome.P_Dub_S - Thursday, December 26, 2013 - link
Who uses RAID 5 now a days? All the research I have done points to OBR10. Can we see some OBR10 numbers?Here are some articles that explain why RAID 5 needs to die.
http://www.smbitjournal.com/2012/07/hot-spare-or-a...
http://www.smbitjournal.com/2012/11/one-big-raid-1...
http://www.smbitjournal.com/2012/05/when-no-redund...
hydromike - Thursday, December 26, 2013 - link
Tons of people still use RAID 5 in the enterprise. Further more lets call it by its real name RAID 10 instead of OBR10. You can get even further redundancy from RAID 50, RAID 60 and RAID 100.P_Dub_S - Thursday, December 26, 2013 - link
And when you go to rebuild that huge RAID 5 array and another disk fails your screwed.xxsk8er101xx - Friday, December 27, 2013 - link
Not if you setup a global spare.Gigaplex - Saturday, December 28, 2013 - link
It still needs to rebuild when it switches over to the spare.Gigaplex - Saturday, December 28, 2013 - link
No, you recover from backup. RAID is to increase availability in the enterprise, it is not a substitute for a backup.P_Dub_S - Thursday, December 26, 2013 - link
Please read that 3rd link and tell me if RAID 5 makes any sense with todays drive sizes and costs.Gunbuster - Thursday, December 26, 2013 - link
Re: that 3rd link. Who calls it resilvering? Sounds like what a crusty old unix sysadmin with no current hardware knowledge would call it.P_Dub_S - Thursday, December 26, 2013 - link
Whatever the name it doesn't really matter its the numbers that count and in TB drive sizes now a days RAID 5 makes zero sense.Kheb - Saturday, December 28, 2013 - link
No it doesnt. Not at all. First, you are taking into account only huge arrays used to store data and not to run applications (so basically only mechanical SATA, that is).Second, you are completeley ignoring costs (raid 5 or raid 6 vs raid 10). Third, you are assuming the raid 5 itself is not backed up or with some sort of software\hardware redundancy or tiering at lower levels (see SANs).So while I can agree that THEORETICALLY having raid 10 everywhere would indeed be safer, the costs (hdds + enclosures + controllers + backplanes) make this, and this time for real, have zero sense.
Ammaross - Thursday, December 26, 2013 - link
"Resilvering" is the ZFS term for rebuilding data on a volume. It's very much a current term still, but it does give us an insight into the current bias of the author, who apparently favors ZFS for his storage until something he proposes as better is golden.hydromike - Thursday, December 26, 2013 - link
How many times have you had to rebuild a RAID5 in your lifetime? I have over 100 times on over 10 major HARDWARE RAID vendors."And when you go to rebuild that huge RAID 5 array and another disk fails your screwed."
The other drive failing is a very small possibility in an enterprise environment that I was talking about, because of enterprise grade drives vs consumer. That is why most have either the raid taken offline for a much faster rebuild. Besides during that rebuild the RAID is still functional just degraded.
Also my point is lots of us still have hardware that is 2-5 years old that is still just working. The newest Arrays that I have setup as of late are 20 to 150 TB in size and we went with Freenas with ZFS which puts all other to shame. NetApp Storage appliances rebuild times are quite fast 6-12 hours for 40TB LUNS. It all depends upon the redundancy that you need. Saying that raid 5 needs to die is asinine. What if the data you are storing is all available in the public domain but have a local copy speeds up the data access rates. The rebuild is faster with a degraded LUN vs retrieving all of the data from the public domain again. There are many use cases for each RAID level just because one level does not fit YOUR uses it does not need to die!
P_Dub_S - Thursday, December 26, 2013 - link
So if you were to buy this NAS for a new implementation would you even consider throwing 10-12 disks in it and building a RAID 5 array? just asking. Even in your own post you state how you use Freenas with ZFS for your new arrays. RAID 5 is the dodo here let it go extinct.Ammaross - Thursday, December 26, 2013 - link
For all you know, he's running ZFS using raidz1 (RAID5 essentially). Also, saying RAID5 needs to die, one must then assume you also think RAID0 is beyond worthless, since it has NO redundancy? Obviously, you can (hopefully) cite the use-cases for RAID0. Your bias just prevents you from seeing the usefulness of RAID5.xxsk8er101xx - Friday, December 27, 2013 - link
It does happen though. I've had to rebuild 2 servers alone this year because of multiple drive failures. One server had 3 drives fail. But that's because of neglect. Us engineers only have so much time. Especially with the introduction to lean manufacturing.RAID 5 + Global spare though is usually pretty safe bet if it's a critical app server. Otherwise RAID 5 is perfectly fine.
JayJ - Friday, December 27, 2013 - link
In at least 2 of the links it is stated that when a RAID 5 rebuild is in process and a URE is encountered, all data is lost.I'm not sure what crap hardware that author has been using. When a URE is encountered during a rebuild the rebuild halts and you're back where you started - with a degraded array.
Now I'm not saying "RAID 5 is the BEST!" but the "facts" presented are false.
FYI I've rebuilt several hundred RAID 5 arrays over the last 15 years and have experienced a URE during rebuild exactly 2 times. You can cut down on UREs by performing a scheduled "Patrol Read" or functional equivalent. There is no way to know if the data is readable unless you read it. You can have a (fictional) "SUPER DUPER RAID 3000" with a ridiculous amount of redundancy but it's still theoretically possible to lose your data due to URE unless it's read and verified.
Computer Bottleneck - Saturday, December 28, 2013 - link
How do you feel about drives like the Western Digital Re which has a URE spec of 10^15 compared to other drives with a URE spec of 10^14 in RAID 5?802.11at - Friday, December 27, 2013 - link
Given the choice, I'll take RAID 10 all day in my enterprise environment.802.11at - Friday, December 27, 2013 - link
But FWIW, our HP LeftHand SAN is comprised of 6 nodes with 8 HDDs each in RAID 5 with the volumes actually running on a networked RAID 10.theangryintern - Monday, December 30, 2013 - link
Nice try, guy who writes for smbitjournalBrutalizer - Wednesday, January 1, 2014 - link
The second link is quite wrong in the premises. It says that filesystems are really reliable today, well, they are not. There are lot of research showing how all filesystems are flawed today (except ZFS) with respect to data corruption protection. Only ZFS protects the data against data corruption. Read the research papers you will find here:http://en.wikipedia.org/wiki/ZFS#Data_integrity
http://en.wikipedia.org/wiki/Hard_disk_error_rates...
tomdb - Thursday, December 26, 2013 - link
How much are consumer grade (if any exist) 10 GbE NAS's, switches and PCIe cards?bobbozzo - Thursday, December 26, 2013 - link
$3000 for a 6-bay Netgear w 10gigEhttp://www.anandtech.com/show/7523/netgear-launche...
bobbozzo - Thursday, December 26, 2013 - link
I don't think there are any 'consumer' 10gb switches yet.Master_shake_ - Saturday, December 28, 2013 - link
infiniband is getting cheaper in priceiAPX - Thursday, December 26, 2013 - link
2000+ MB/s ethernet interface (2x10Gb/s), 10 hard-drives able to to delivers at least 500MB/s EACH (grand total of 5000MB/s), Xeon quad-core CPU, and tested with ONE client, it delivers less than 120MB/s?!?That's what I expect from an USB 3 2.5" external hard-drive, not a SAN of this price, it's totally deceptive!
Ammaross - Thursday, December 26, 2013 - link
Actually, 120MB/s is remarkably exactly what I would expect from a fully-saturated 1Gbps link (120MB/s * 8 bits = 960Mbps). Odd how that works out.xxsk8er101xx - Friday, December 27, 2013 - link
That's because the PC only has a gigabit NIC. That's actually what you should expect.BrentfromZulu - Thursday, December 26, 2013 - link
For the few who know, I am the Brent that brought up Raid 5 on the Mike Tech Show (saying how it is not the way to go in any case)Raid 10 is the performance king, Raid 1 is great for cheap redundancy, and Raid 10, or OBR10, should be what everyone uses in big sets. If you need all the disk capacity, use Raid 6 instead of Raid 5 because if a drive fails during a rebuild, then you lose everything. Raid 6 is better because you can lose a drive. Rebuilding is a scary process with Raid 5, but Raid 1 or 10, it is literally copying data from 1 disk to another.
Raid 1 and Raid 10 FTW!
xdrol - Thursday, December 26, 2013 - link
From the drives' perspective, rebuilding a RAID 5 array is exactly the same as rebuilding a RAID 1 or 10 array: Read the whole disk(s) (or to be more exact, sectors with data) once, and write the whole target disk once. It is only different for the controller. I fail to see why is one scarier than the other.If your drive fails while rebuilding a RAID 1 array, you are exactly as screwed. The only thing why R5 is worse here is because you have n-1 disks unprotected while rebuilding, not just one, giving you approximately (=negligibly smaller than) n-1 times data loss chance.
BrentfromZulu - Friday, December 27, 2013 - link
Rebuilding a Raid 5 requires reading data from all of the other disks, whereas Raid 10 requires reading data from 1 other drive. Raid 1 rebuilds are not complex, nor Raid 10. Raid 5/6 rebuilding is complex, requires activity from other disks, and because of the complexity has a higher chance of failure.xxsk8er101xx - Friday, December 27, 2013 - link
You take a big hit on performance with RAID 6.Ajaxnz - Thursday, December 26, 2013 - link
I've got one of these with 3 extra shelves of disks and 1TB of SSD cache.There's a limit of 3 shelves in a single volume, but 120TB (3 shelves of 12 4Tb disks, raid5 on each shelf) with the SSD cache performs pretty well.
For reference, NFS performance is substantially better than CIFS or iSCSI.
It copes fine with the 150 virtual machines that support a 20 person development team.
So much cheaper than a NetAPP or similar - but I haven't had a chance to test the multi-NAS failover - to see if you truly get enterprise quality resilience.
jasonelmore - Friday, December 27, 2013 - link
well at least half a dozen morons got schooled on the different types of RAID arrays. gg, always glad to see the experts put the "less informed" (okay i'm getting nicer) ppl in their place.Marquis42 - Friday, December 27, 2013 - link
I'd be interested in knowing greater detail on the link aggregation setup. There's no mention of the load balancing configuration in particular. The reason I ask is because it's probably *not* a good idea to bond 1Gbps links with 10Gbps links in the same bundle unless you have access to more advanced algorithms (and even then I wouldn't recommend it). The likelihood of limiting a single stream to ~1Gbps is fairly good, and may limit overall throughput depending on the number of clients. It's even possible (though admittedly statistically unlikely) that you could limit the entirety of the system's network performance to saturating a single 1Gbe connection.mfenn - Friday, December 27, 2013 - link
The 802.3ad testing in this article is fundamentally flawed. 802.3ad does NOT, repeat NOT, create a virtual link whose throughput is the sum of its components. What it does is provide a mechanism for automatically selecting which link in a set (bundle) to use for a particular packet based on its source and destination. The definition of "source and destination" depends on the particular hashing algorithm you choose, but the common algorithms will all hash a network file system client / server pair to the same link.In a 4 1Gb/s + 2 10Gb/s 802.3ad ling aggregation group, you would expect that 2/3rd's of the clients would get hashed to the 1 Gb/s links and 1/3rd would get hashed to the 10Gb/s links. In a situation where all clients are running in lock-step (i.e. everyone must complete their tests before moving on to the next), you would expect the 10 Gb/s clients to be limited by the 1 Gb/s ones, thus providing a ~ 6 Gb/s line rate ~= 600 MB/s user data result.
Since 2 * 10 Gb/s > 6 * 1 Gb/s, I recommend retesting with only the two 10 Gb/s links in the 802.3ad aggregation group.
Marquis42 - Friday, December 27, 2013 - link
Indeed, that's what I was going to get at when I asked more about the particulars of the setup in question. Thanks for just laying it out, saved me some time. ;)ganeshts - Friday, December 27, 2013 - link
mfenn / Marquis42,Thanks for the note. I indeed realized this issue after processing the data for the Synology unit. Our subsequent 10GbE reviews which are slated to go out over the next week or so (the QNAP TS-470 and the Netgear ReadyNAS RN-716) have been evaluated with only the 10GbE links in aggregated mode (and the 1 GbE links disconnected).
I will repeat the Synology multi-client benchmark with RAID-5 / 2 x 10Gb 802.3ad and update the article tomorrow.
ganeshts - Saturday, December 28, 2013 - link
I have updated the piece with the graphs obtained by just using the 2 x 10G links in 802.3ad dynamic link aggregation. I believe the numbers don't change too much compared to teaming all the 6 ports together.Just for more information on our LACP setup:
We are using the GSM7352S's SFP+ ports teamed with link trap and STP mode enabled. Obviously, dynamic link aggregation mode. The Hash Mode is set to 'Src/Dest MAC, VLAN, EType,Incoming Port'.
I did face problems in evaluating other units where having the 1 Gb links active and connected to the same switch while the 10G ports were link-aggregated would bring down the benchmark numbers. I have since resolved that by completely disconnecting the 1G links in multi-client mode for the 10G-enabled NAS units.
shodanshok - Saturday, December 28, 2013 - link
Hi all,while I understand that RAID5 surely has its domains, RAID10 is generally a way better choice, both for redundancy and performance.
The RAID5 read-modify-write penalty present itself in a pretty heavy way when using anything doing many small writes, as databases ans virtual machines. So, then only case where I would create a RAID5 array is when it will be used as a storage archive (eg: fileserver).
On the other hand, many, many sysadmins create "by default" RAID5 arrays pretending to consolidate on it many virtual machines. Unless you have a very high-end RAID controller (w/512+ MB of NVCache), they will badly suffer from RAID5 and alignment issues, which are basically non-existent on RAID10.
One exception can be done for SSD arrays: in that case, a parity-based scheme (RAID5 or, better, RAID6) can do its work done very well, as SSD have no seek latency and tend to be of lower capacity than mechanical disks. However, alignment issues remain significant, and need to be taken into account when creating both the array and the virtual machines on top of it.
Regards.
sebsta - Saturday, December 28, 2013 - link
Since the introduction of 4k sector size disks things have changed a lot,at least in the ZFS world. Everyone who is thinking about building their
storage system with ZFS and RaidZ should see this Video.
http://zfsday.com/zfsday/y4k/
Starting at 17:00 comes the bad stuff for RaidZ users.
Here the one of the co creators of ZFS basically tells you.....
Stay away from RaidZ if you are using 4k sector disks.
hydromike - Sunday, December 29, 2013 - link
It depends on the OS implemented if this is a current problem. Many of the commercial ZFS vendors have had this fixed for awhile 18 to 24 months. FreeNAS in its latest release 9.2.0 have fixed this issue. ZFS has been a command-line heavy operation that you really understand drive setup and to tune it for the best speed.sebsta - Sunday, December 29, 2013 - link
I don't know much about FreeNAS but like FreeBSD they get their ZFS from Illumos.Illumos ZFS implementation has no fix. What is ZFS supposed to do if you write 8k to a RaidZ with 4 data disks if the sector size of a disk is 4k?
The video explain what happens on Illumos. You will end up with something like this
1st 4k data -> disk1
2nd 4k data -> disk2
1st 4k data -> disk3
2nd 4k data -> disk4
Parity -> disk5
So you have written the same data twice plus parity. Much like mirroring with the additional overhead of calculating and writing the parity. FreeNAS has changed the ZFS implementation in that regard?
sebsta - Sunday, December 29, 2013 - link
I did a quick search and at least in January this year FreeBSD had the same issues.See here https://forums.freebsd.org/viewtopic.php?&t=37...
shodanshok - Monday, December 30, 2013 - link
Yes this is true, but for this very same reason many enterprise disks remain at 512 Byte per sector.Take the "enterprise drives" from WD:
- the low cost WD SE are Advanced Format ones (4K sector size)
- the somewhat pricey WD RE have classical 512B sector
- the top-of-the line WD XE have 512B sector
So, the 4K formatted disks are proposed for storage archiving duties, while for VMs and DBs the 512B disks remain the norm.
Regards.
Methodical713 - Monday, December 30, 2013 - link
Raid5 is fine so long as you keep patrol reads enabled. All those horror stories about failed rebuilds come from people that don't know what patrol reads are for, and turned them off for some stupid reason or another, or were too cheap to buy a real raid controller.