Great article...look forward to more enterprise/IT professional based articles from Anandtech in the future. This is very timely for me as my company is just about to pull the trigger on a server upgrade. Interesting stuff.
Good article, although as an enterprise architect, I can tell you the one true benefit to LRDIMMS is in 2 and 4 socket vhost builds, because the double density RAM gives you the freedom to turn off NUMA spanning and still get near-ideal guest density.
Almost nobody runs caching servers that big, although at almost double performance over a 256GB build (the 100k + concurrent user norm) its kind of attractive to run 2 of these per DC instead of 6 smaller ones (which would actually be the real world comparison with those kind of deltas).
Real-world pricing, at least in the enterprise context, is quite a bit off from your numbers. In my employer's price bracket, we regularily buy similar servers as your 24*16GB config for about the same price (13k€) - but including a 3 year subscription VMWare Enterprise license, which is about 6 to 7 k€ on its own. No one pays list price on that kind of hardware.
Are you sure that there is not a big discount on the VMware license? And smaller enterprises will pay something close to the list price. I know that the typical discount is 10-20% for smaller quantities, not more.
Depends on the country Johan. The partner channel managers get to decide discounts on partner orders (which he is describing). Also, the bundling discount doesn't happen everywhere, but I could buy that server for like $15k CDN.
The VMware license cost seems out of this world to me too, because we license our hosts for anyone from 2500 to 5k CDN, depending on their agreement with VMware.
I don't really know where exactly the discount is applied, as the licenses are OEM and we don't get line-item pricing. In our market segment (large enterprise with Dell, medium-to-large with HP) we usually see at least 40% off on list prices, in some cases (networking equipment) up to 75%.
VMWare, on the other hand, is especially rigid with their pricing structure. Two years ago, when we negotiated for a 100 host branch office deployment, they referred to their list pricing. For them, we are not even big enough to speak directly to us.
seriously though .. DDR3 prices have been going up. as near as I can tell their approximately 2.3X the cost of what they once were. Memory makers are doing the semi-happy dance these days and likely looking forward to the 5x pricing schemes of yesteryear.
Interesting comment. I ran with gcc, Opencc with O2, O3 and Ofast. If the gcc binary is 100%, I get 110% with Opencc (-O2), 130% (-O3) and the same with Ofast.
i am guessing Open64 might be producing better code (atleast) when it comes to memory operations. i gave up on Open64 a while back and maybe i should try it out again.
The article is interesting, but alone it doesn't justify the expense for high-capacity LRDIMMs in a server. As server professionals, our goal is usually to maximise performance / cost for a specific role. In this example, I can't imagine that better performance (at a dramatically lower cost) would not be obtained by upgrading the storage pool instead. I'd love to see a comparison of increasing memory sizes vs adding more SSD caching, or combinations thereof.
Depends on the size of your data set as well, I'd guess, and whether or not you can fit the entire thing in memory.
If you can, and considering RAM is still orders of magnitude faster than SSDs I imagine memory still wins out in terms of overall performance. Too large to fit in a reasonable amount of RAM and yes, SSD caching would possibly be more cost effective.
First of all, if your workload is read intensive, more RAM will almost always be much faster than any flash cache. Secondly, it greatly depends on your storage vendor whether adding more flash can be done at "dramatically lower cost". The tier-one vendors still charge an arm and a leg for flash cache, while the server vendors are working at much more competitive prices. I would say that in general it is cheaper and more efficient to optimize RAM caching versus optimizing your storage (unless your are write limited).
Not only are you correct, but significantly so. Enterprise flash storage at decent densities is more costly PER GIG than DDR3. Not only that, but you need the 'cadillac' model SANs to support more than 2 SSDs. Not to mention fabric management is a lot more resource intensive and more prone to error.
Right now, the best bet (like always) to get performance is to stuff your servers with memory and distribute your workload. Because its poor network architecture that creates bottlenecks in any environment where you need to stuff more than 256GB of RAM into a single box.
Another thing about HPC is that, as long as a processor has: enough RAM to do its dataset on the CPU/GPU before it needs more data, the quantity of RAM is enough. Saving on RAM can let you buy more nodes, which gives you more performance capacity.
headline should have been: if you're serving static content, your main goal is to maximize ram per node. not exactly a shocker eh? in the real world, at least the HPC corner of it, 1G/core is pretty common, and 32G/core is absurd. hence, udimms are actually a good choice sometimes.
I would very much like to know what specific memory model (brand, model number) you are referring to regarding the 32GB LRDIMM—1866 option. I have searched at no avail. Johan? / Anyone? Thank you in advance! / Tomas
I don't know why people are still going after server hardware. I mean it's the 21st century. Now everything is on cloud. Where you have the ability to scale your server anytime you want to. I mean the hosting provider companies like: AWS, DigitalOcean, Vultr hosting https://www.cloudways.com/en/vultr-hosting.php, etc. has made it very easy to rent your server.
We’ve updated our terms. By continuing to use the site and/or by logging into your account, you agree to the Site’s updated Terms of Use and Privacy Policy.
Facebook
Twitter
RSS
27 Comments
Back to Article
subflava - Thursday, December 19, 2013 - link
Great article...look forward to more enterprise/IT professional based articles from Anandtech in the future. This is very timely for me as my company is just about to pull the trigger on a server upgrade. Interesting stuff.JohanAnandtech - Friday, December 20, 2013 - link
Thanks for sharing! :-)DERSS - Friday, December 27, 2013 - link
You guys are seriously super-cool; thanks.wsaenotsock - Thursday, December 19, 2013 - link
costed?blaktron - Thursday, December 19, 2013 - link
Good article, although as an enterprise architect, I can tell you the one true benefit to LRDIMMS is in 2 and 4 socket vhost builds, because the double density RAM gives you the freedom to turn off NUMA spanning and still get near-ideal guest density.Almost nobody runs caching servers that big, although at almost double performance over a 256GB build (the 100k + concurrent user norm) its kind of attractive to run 2 of these per DC instead of 6 smaller ones (which would actually be the real world comparison with those kind of deltas).
mexell - Thursday, December 19, 2013 - link
Real-world pricing, at least in the enterprise context, is quite a bit off from your numbers. In my employer's price bracket, we regularily buy similar servers as your 24*16GB config for about the same price (13k€) - but including a 3 year subscription VMWare Enterprise license, which is about 6 to 7 k€ on its own. No one pays list price on that kind of hardware.JohanAnandtech - Friday, December 20, 2013 - link
Are you sure that there is not a big discount on the VMware license? And smaller enterprises will pay something close to the list price. I know that the typical discount is 10-20% for smaller quantities, not more.blaktron - Friday, December 20, 2013 - link
Depends on the country Johan. The partner channel managers get to decide discounts on partner orders (which he is describing). Also, the bundling discount doesn't happen everywhere, but I could buy that server for like $15k CDN.The VMware license cost seems out of this world to me too, because we license our hosts for anyone from 2500 to 5k CDN, depending on their agreement with VMware.
mexell - Saturday, December 21, 2013 - link
I don't really know where exactly the discount is applied, as the licenses are OEM and we don't get line-item pricing. In our market segment (large enterprise with Dell, medium-to-large with HP) we usually see at least 40% off on list prices, in some cases (networking equipment) up to 75%.VMWare, on the other hand, is especially rigid with their pricing structure. Two years ago, when we negotiated for a 100 host branch office deployment, they referred to their list pricing. For them, we are not even big enough to speak directly to us.
dstarr3 - Thursday, December 19, 2013 - link
Wow. With 768GB of memory, I bet you could run Crysis.slideruler - Thursday, December 19, 2013 - link
Am I the only one who's concern with DDR4 in our future?Given that it's one-to-one we'll lose the ability to stuff our motherboards with cheap sticks to get to "reasonable" (>=128gig) amount of RAM... :(
just4U - Thursday, December 19, 2013 - link
You really shouldn't need more than 640kb.... :Djust4U - Thursday, December 19, 2013 - link
seriously though .. DDR3 prices have been going up. as near as I can tell their approximately 2.3X the cost of what they once were. Memory makers are doing the semi-happy dance these days and likely looking forward to the 5x pricing schemes of yesteryear.MrSpadge - Friday, December 20, 2013 - link
They have to come up with something better than "1 DIMM per channel using the same amount of memory controllers" for servers.theUsualBlah - Thursday, December 19, 2013 - link
the -Ofast flag for Open64 will relax ansi and ieee rules for calculations, whereas the GCC flags won't do that.maybe thats the reason Open64 is faster.
JohanAnandtech - Friday, December 20, 2013 - link
Interesting comment. I ran with gcc, Opencc with O2, O3 and Ofast. If the gcc binary is 100%, I get 110% with Opencc (-O2), 130% (-O3) and the same with Ofast.theUsualBlah - Friday, December 20, 2013 - link
hmm, thats very interesting.i am guessing Open64 might be producing better code (atleast) when it comes to memory operations. i gave up on Open64 a while back and maybe i should try it out again.
thanks!
GarethMojo - Friday, December 20, 2013 - link
The article is interesting, but alone it doesn't justify the expense for high-capacity LRDIMMs in a server. As server professionals, our goal is usually to maximise performance / cost for a specific role. In this example, I can't imagine that better performance (at a dramatically lower cost) would not be obtained by upgrading the storage pool instead. I'd love to see a comparison of increasing memory sizes vs adding more SSD caching, or combinations thereof.JlHADJOE - Friday, December 20, 2013 - link
Depends on the size of your data set as well, I'd guess, and whether or not you can fit the entire thing in memory.If you can, and considering RAM is still orders of magnitude faster than SSDs I imagine memory still wins out in terms of overall performance. Too large to fit in a reasonable amount of RAM and yes, SSD caching would possibly be more cost effective.
MrSpadge - Friday, December 20, 2013 - link
One could argue that the storage optimization would be done for both memory configurations.JohanAnandtech - Friday, December 20, 2013 - link
First of all, if your workload is read intensive, more RAM will almost always be much faster than any flash cache. Secondly, it greatly depends on your storage vendor whether adding more flash can be done at "dramatically lower cost". The tier-one vendors still charge an arm and a leg for flash cache, while the server vendors are working at much more competitive prices. I would say that in general it is cheaper and more efficient to optimize RAM caching versus optimizing your storage (unless your are write limited).blaktron - Friday, December 20, 2013 - link
Not only are you correct, but significantly so. Enterprise flash storage at decent densities is more costly PER GIG than DDR3. Not only that, but you need the 'cadillac' model SANs to support more than 2 SSDs. Not to mention fabric management is a lot more resource intensive and more prone to error.Right now, the best bet (like always) to get performance is to stuff your servers with memory and distribute your workload. Because its poor network architecture that creates bottlenecks in any environment where you need to stuff more than 256GB of RAM into a single box.
hoboville - Friday, December 20, 2013 - link
Another thing about HPC is that, as long as a processor has: enough RAM to do its dataset on the CPU/GPU before it needs more data, the quantity of RAM is enough. Saving on RAM can let you buy more nodes, which gives you more performance capacity.markhahn - Saturday, January 4, 2014 - link
headline should have been: if you're serving static content, your main goal is to maximize ram per node. not exactly a shocker eh? in the real world, at least the HPC corner of it, 1G/core is pretty common, and 32G/core is absurd. hence, udimms are actually a good choice sometimes.mr map - Monday, January 20, 2014 - link
Very interesting article, Johan!I would very much like to know what specific memory model (brand, model number) you are referring to regarding the 32GB LRDIMM—1866 option.
I have searched at no avail.
Johan? / Anyone?
Thank you in advance!
/ Tomas
Gasaraki88 - Thursday, January 30, 2014 - link
A great article as always.ShirleyBurnell - Tuesday, November 5, 2019 - link
I don't know why people are still going after server hardware. I mean it's the 21st century. Now everything is on cloud. Where you have the ability to scale your server anytime you want to. I mean the hosting provider companies like: AWS, DigitalOcean, Vultr hosting https://www.cloudways.com/en/vultr-hosting.php, etc. has made it very easy to rent your server.