You should also test ASRock Rack ROMED8-2T. It supports 7x PCIe Gen4 x16 expansion slots, 2TB of DDR4 3200 (with 256GB LRDIMMs), 2x10GB ethernet ports etc. And the memory slots don't block long PCIe cards. https://www.asrockrack.com/general/productdetail.a...
Now this is a board I want for my next 4xGPU deep learning workstation. Though EPYC cpus suck frequency wise (when compared to equivalent core count TRs). Basically it comes down to whether I want a good memory bandwidth (8 channels vs 4 channels), and whether I want the full 16 lanes per GPU for all four GPUs. Serious trade-off. Why can't they put a pcie switch into a TR board (I'm talking about ASRock TRX40 Creator)?
LOL that's cute you think there is so much traffic to those GPUs that you not only need PCIe4 - but 16 lanes per. Pretty sure your "4xGPU deep learning workstation" is a theoretical purchase, and will not materialize in any way shape or form. You could put those on 8 lane PCIe3 and get the same results. Moar Corez Moar Lanez No Cluez. Keep dreaming
This article doesn't sufficiently highlight the impact of the memory slots interfering with the PCIe slots. This severely limits the use cases for this motherboard. Someone mentioned using this board in a storage server, but I doubt such a system needs the bandwidth that Gen. 4 slots provide. I think Gigabyte would have been able to increase their market by going with a design similar to ASRock's, which has 1 DIMM slot per memory channel. This would have allowed the use of multiple accelerator boards, which can make use of the bandwidth the Gen. 4 slots provide. I get the impression that Gigabyte's designs are geared toward specific customers.
On the positive side, as far as I can tell, this board has been the only Gen. 4 motherboard on the market that supports EPYC CPUs, which to me is surprising given how long it's been since EPYC CPUs were announced.
My bad. I got confused with the MZ32-AR0, with which I have personal experience. It looks very similar, but it has PCIe Gen. 4 slots, all of which, except for one, are also obstructed by DIMM slots.
Omg, *almost* perfect motherboard for my upgrade. Is there anything similar, but with sTR3/threadripper? Seriously, with all that bandwith, noone has truly workstation board, without wifi, but with 1-2 10g ports integrated, and more than 3 usable slots... I need tr board, with 10g, and layout supporting 3slot gpu (7slots or lets say, in xlatx manner, last slot with 16lanes), two pcie ssd (intel Pseries, not m2), one slot with bifurbication for 2-4 m2 slots card, maybe sound card, and one free slot, or even two, for second (2slot) gpu, 25gb+ lan, or another monster ssd. I can get it for intel cpus easily, but i *really* want to leave that swamp for amd
Looks like the perfect board for my home Pfsense build. In all seriousness though looks like a pretty baller single socket Epyc board if only I had something lined up that could use it.
Does anyone know if it's possible to use Unbuffered ECC RAM with EPYC? Thinking of upgrading from a Threadripper to an EPYC for a Server, but I don't want to rebuy RAM as I already got enough DDR4 ECC UDIMMs.
Some Tyan boards seemingly support it from their spec sheets, but it's something that most reviews don't cover, unfortunately.
Patrick at Serve the Home has also mentioned support on some older boards, but unfortunately he didn't mention any specific models in the one comment I was able to dig up on their forums.
Also, curious if the board supports bifurcation - the lack of M.2 Slots would make it perfect for e.g., an ASUS HYPER M.2 X16 or Supermicro AOC-SLG3-2M2 card, but for that it needs to support bifurcation of an x16 into 4x4 or an x8 into 2x4 slots.
The Hyper M.2 is long enough that you wouldn't be able to insert the card into the first five slots, which is a bit of a bummer. I can't confirm bifurcation support but I'd be very surprised if it didn't have it.
We built our first iSCSI storage box around this motherboard. Have nothing but great experiences with it, and the IPMI implementation is light-years beyond what Supermicro supports (the GB IPMI is fully-web-enabled, using HTML5 KVM/console redirection, compared to the horrid Java implementation that Supermicro uses).
The only downside if the SlimSAS connectors. It's very easy to find SlimSAS-to-SATA connectors (even comes with some in the box). Pain in the ass to find SlimSAS-to-SFF8084 connectors to connect to multi-lane, direct-attach backplanes! They exist, but nobody actually carries them (at least nobody that we could find in North America). Had to get them from a cable maker in China, which took the very slow boat to arrive.
For our first build, I messed up the model number for the backplane, ending up with an SAS expander version, so we had to use an HBA anyway.
For our second iSCSI storage server, we switched to the MZ01-CE0 motherboard. This includes 10GBase-T ports instead of SFP+. Other than that, it's virtually the same board. Got the right model backplane, and the SlimSAS-to-SFF8084 cables, giving 16 SATA connections direct off the motherboard.
For our next ZFS storage server, we'll be using MZ01-CE0 in a 2U chassis, running the OS off the motherboard connectors, with LSI/Broadcom/whoever-they-are-now HBAs with external connectors going to multiple 45-bay storage chasssis. That setup works great with all the extra PCIe lanes. :)
"and the IPMI implementation is light-years beyond what Supermicro supports (the GB IPMI is fully-web-enabled, using HTML5 KVM/console redirection, compared to the horrid Java implementation that Supermicro uses)."
Errrgh, no. Supermicro has had HTML KVM since quite a while. I think all AST2400 BMCs offer it. I was quite surprised that I got it even for my old AST2400 BMC with an update. And it works as one would expect.
Why does nobody use the dual 10Gbe controller built into the EPYC chips? Each Zen die has 4 x 10 Gbe MAC's on board, yet the only motherboard I have ever seen to utilize them is AMD Wallaby platform development board here https://www.servethehome.com/amd-epyc-3251-benchma...
My only guess is that motherboard manufacturers are worried about compatibility issues. I sure would rather save the money, energy, and heat resources and use the embedded AMD 10Gbe than another manufacturer's chipset shoehorned into the motherboard for no good reason.
I keep running into workstation boards that are capped at 64gb of ram based on epyc or thread ripper when i'm search engine diving. Seems wrong to cap it at 64.
Forums: these are how the discussion board is categorised. For instance, there may be one forum for general discussion, and other separate forums to discuss each of the assignments and other aspects of the unit. Threads: these are individual subjects or questions within a forum. For instance, you may have a question about an upcoming assignment that you wish to discuss with your lecturer and the other students. You would create a thread to pose this question (assuming no one has previously asked the same question). Posts: these are individual replies within threads. If another student has an answer to the question you have posed, they would "post" inside your thread, to discuss your specific question. https://place-4-papers.com/write-my-discussion-boa...
We’ve updated our terms. By continuing to use the site and/or by logging into your account, you agree to the Site’s updated Terms of Use and Privacy Policy.
Facebook
Twitter
RSS
37 Comments
Back to Article
sonny73n - Wednesday, March 25, 2020 - link
Wow, two Gigabyte motherboards reviews in a week!?I have to find something else to read.
demu - Wednesday, March 25, 2020 - link
You should also test ASRock Rack ROMED8-2T.It supports 7x PCIe Gen4 x16 expansion slots, 2TB of DDR4 3200 (with 256GB LRDIMMs), 2x10GB ethernet ports etc.
And the memory slots don't block long PCIe cards.
https://www.asrockrack.com/general/productdetail.a...
p1esk - Wednesday, March 25, 2020 - link
Now this is a board I want for my next 4xGPU deep learning workstation. Though EPYC cpus suck frequency wise (when compared to equivalent core count TRs). Basically it comes down to whether I want a good memory bandwidth (8 channels vs 4 channels), and whether I want the full 16 lanes per GPU for all four GPUs. Serious trade-off. Why can't they put a pcie switch into a TR board (I'm talking about ASRock TRX40 Creator)?Deicidium369 - Sunday, April 12, 2020 - link
LOL that's cute you think there is so much traffic to those GPUs that you not only need PCIe4 - but 16 lanes per. Pretty sure your "4xGPU deep learning workstation" is a theoretical purchase, and will not materialize in any way shape or form. You could put those on 8 lane PCIe3 and get the same results. Moar Corez Moar Lanez No Cluez. Keep dreamingSlash3 - Wednesday, March 25, 2020 - link
It's not yet available at retail (soon™), but it should be an absolute beast of a board.Ktracho - Friday, March 27, 2020 - link
This article doesn't sufficiently highlight the impact of the memory slots interfering with the PCIe slots. This severely limits the use cases for this motherboard. Someone mentioned using this board in a storage server, but I doubt such a system needs the bandwidth that Gen. 4 slots provide. I think Gigabyte would have been able to increase their market by going with a design similar to ASRock's, which has 1 DIMM slot per memory channel. This would have allowed the use of multiple accelerator boards, which can make use of the bandwidth the Gen. 4 slots provide. I get the impression that Gigabyte's designs are geared toward specific customers.On the positive side, as far as I can tell, this board has been the only Gen. 4 motherboard on the market that supports EPYC CPUs, which to me is surprising given how long it's been since EPYC CPUs were announced.
phoenix_rizzen - Saturday, March 28, 2020 - link
There are no Gen 4 slots on this motherboard. It's all PCIe 3.Ktracho - Monday, March 30, 2020 - link
My bad. I got confused with the MZ32-AR0, with which I have personal experience. It looks very similar, but it has PCIe Gen. 4 slots, all of which, except for one, are also obstructed by DIMM slots.paulinus - Saturday, March 28, 2020 - link
Omg, *almost* perfect motherboard for my upgrade.Is there anything similar, but with sTR3/threadripper?
Seriously, with all that bandwith, noone has truly workstation board, without wifi, but with 1-2 10g ports integrated, and more than 3 usable slots...
I need tr board, with 10g, and layout supporting 3slot gpu (7slots or lets say, in xlatx manner, last slot with 16lanes), two pcie ssd (intel Pseries, not m2), one slot with bifurbication for 2-4 m2 slots card, maybe sound card, and one free slot, or even two, for second (2slot) gpu, 25gb+ lan, or another monster ssd.
I can get it for intel cpus easily, but i *really* want to leave that swamp for amd
ballsystemlord - Wednesday, March 25, 2020 - link
@Gavin will a threadripper CPU work in an EPYC motherboard?I'm just curious. I did search online. Thanks!
xyvyx2 - Wednesday, March 25, 2020 - link
physically the same, but not compatible :(https://en.wikipedia.org/wiki/Socket_sTRX4
Operandi - Wednesday, March 25, 2020 - link
Looks like the perfect board for my home Pfsense build. In all seriousness though looks like a pretty baller single socket Epyc board if only I had something lined up that could use it.Deicidium369 - Sunday, April 12, 2020 - link
a door stop would be an appropriate use for it - unlikely AMD would have the drivers ironed out.MenhirMike - Wednesday, March 25, 2020 - link
Does anyone know if it's possible to use Unbuffered ECC RAM with EPYC? Thinking of upgrading from a Threadripper to an EPYC for a Server, but I don't want to rebuy RAM as I already got enough DDR4 ECC UDIMMs.Slash3 - Wednesday, March 25, 2020 - link
Some Tyan boards seemingly support it from their spec sheets, but it's something that most reviews don't cover, unfortunately.Patrick at Serve the Home has also mentioned support on some older boards, but unfortunately he didn't mention any specific models in the one comment I was able to dig up on their forums.
MenhirMike - Wednesday, March 25, 2020 - link
Also, curious if the board supports bifurcation - the lack of M.2 Slots would make it perfect for e.g., an ASUS HYPER M.2 X16 or Supermicro AOC-SLG3-2M2 card, but for that it needs to support bifurcation of an x16 into 4x4 or an x8 into 2x4 slots.Slash3 - Wednesday, March 25, 2020 - link
The Hyper M.2 is long enough that you wouldn't be able to insert the card into the first five slots, which is a bit of a bummer. I can't confirm bifurcation support but I'd be very surprised if it didn't have it.phoenix_rizzen - Thursday, March 26, 2020 - link
We built our first iSCSI storage box around this motherboard. Have nothing but great experiences with it, and the IPMI implementation is light-years beyond what Supermicro supports (the GB IPMI is fully-web-enabled, using HTML5 KVM/console redirection, compared to the horrid Java implementation that Supermicro uses).The only downside if the SlimSAS connectors. It's very easy to find SlimSAS-to-SATA connectors (even comes with some in the box). Pain in the ass to find SlimSAS-to-SFF8084 connectors to connect to multi-lane, direct-attach backplanes! They exist, but nobody actually carries them (at least nobody that we could find in North America). Had to get them from a cable maker in China, which took the very slow boat to arrive.
For our first build, I messed up the model number for the backplane, ending up with an SAS expander version, so we had to use an HBA anyway.
For our second iSCSI storage server, we switched to the MZ01-CE0 motherboard. This includes 10GBase-T ports instead of SFP+. Other than that, it's virtually the same board. Got the right model backplane, and the SlimSAS-to-SFF8084 cables, giving 16 SATA connections direct off the motherboard.
For our next ZFS storage server, we'll be using MZ01-CE0 in a 2U chassis, running the OS off the motherboard connectors, with LSI/Broadcom/whoever-they-are-now HBAs with external connectors going to multiple 45-bay storage chasssis. That setup works great with all the extra PCIe lanes. :)
fazalmajid - Friday, March 27, 2020 - link
What OS do you use for your ZFS/iSCSI servers?phoenix_rizzen - Saturday, March 28, 2020 - link
FreeBSDphoenix_rizzen - Saturday, March 28, 2020 - link
Stupid lack of edit button.FreeBSD on the ZFS storage servers. Linux in the iSCSI servers.
kobblestown - Saturday, March 28, 2020 - link
"and the IPMI implementation is light-years beyond what Supermicro supports (the GB IPMI is fully-web-enabled, using HTML5 KVM/console redirection, compared to the horrid Java implementation that Supermicro uses)."Errrgh, no. Supermicro has had HTML KVM since quite a while. I think all AST2400 BMCs offer it. I was quite surprised that I got it even for my old AST2400 BMC with an update. And it works as one would expect.
Deicidium369 - Sunday, April 12, 2020 - link
Yeah the GB stuff is nice, but really just playing catchup to SM.oRAirwolf - Saturday, March 28, 2020 - link
Why does nobody use the dual 10Gbe controller built into the EPYC chips? Each Zen die has 4 x 10 Gbe MAC's on board, yet the only motherboard I have ever seen to utilize them is AMD Wallaby platform development board here https://www.servethehome.com/amd-epyc-3251-benchma...My only guess is that motherboard manufacturers are worried about compatibility issues. I sure would rather save the money, energy, and heat resources and use the embedded AMD 10Gbe than another manufacturer's chipset shoehorned into the motherboard for no good reason.
mode_13h - Monday, March 30, 2020 - link
...and VGA FTMFW!!mode_13h - Monday, March 30, 2020 - link
yes, I know, KVMs...I even have a VGA KVM, myself. Although, I mostly use it as just a keyboard+mouse switch.
It's just ironic to see a board with so much modern tech, and yet it's rare to even see a VGA port on new monitors.
Qasar - Tuesday, March 31, 2020 - link
its rare ??? you sure ?? seems most of the monitors that AT mentions, have vga ports on them.... even the new one from here : https://www.anandtech.com/show/15674/tuf-goes-free... has onetravelstar - Monday, March 30, 2020 - link
I'm curious as to why we have a review of an older motherboard restricted to PCIe Gen3 slots, when the newer Z323-AR0 comes with PCIe Gen4 slots.Pyxar - Wednesday, December 23, 2020 - link
I keep running into workstation boards that are capped at 64gb of ram based on epyc or thread ripper when i'm search engine diving. Seems wrong to cap it at 64.tomholms - Tuesday, September 7, 2021 - link
Forums: these are how the discussion board is categorised. For instance, there may be one forum for general discussion, and other separate forums to discuss each of the assignments and other aspects of the unit.Threads: these are individual subjects or questions within a forum. For instance, you may have a question about an upcoming assignment that you wish to discuss with your lecturer and the other students. You would create a thread to pose this question (assuming no one has previously asked the same question).
Posts: these are individual replies within threads. If another student has an answer to the question you have posed, they would "post" inside your thread, to discuss your specific question.
https://place-4-papers.com/write-my-discussion-boa...