Which mobo anyone recommend to make ~1000 core AMD EPIC-based supercomputer for floating point 3D simulations like particle-in-cell or molecular dynamics ?
I'd go with full servers instead at that point. Gigabyte's 10xGPU 4U server is the obvious choice for supercomputer-level performance unless you choose to go directly with NVidia's DGX A100 - but that's a 6U package instead of the traditional 4U server. The Gigabyte solution has more expansion, though. It supports 128 cores and 10 GPUs along with 8x U.2 NVMe and 4x SATA/SAS drives as well as an OCP 3.0 slot for up to 2x200Gb networking. That gives you 1024 cores, 80 A100 GPUs, a total of up to 32TB of RAM using 128GB DIMMs, 100+ TB of raw U.2 NVMe scratch capacity (at potentially up to 25+ GB/s, which is knocking on the door of DDR4), another 60+ GB of SAS capacity, 400Gb/s interconnect, and a 5 year warranty, all for the cool, cool price of $1.6M and a peak power consumption of over 30kW. 8 DGX A100s would cost about the same, but have less scratch space and take up more room, as well as consume up to 70% more power. But keep in mind that a DGX is a completely self-contained unit, whereas a normal server like the Gigabyte would require additional (and very power hungry) network switches and other management hardware and software. If you're dealing with a dataset that's being accessed remotely, and all you want is plug and play, then the DGX is the winner. But if for whatever reason you need to keep the data in the machine itself, or if the dataset is truly enormous and requires multiple DWPD, then you might be better off with a more classic server like the Gigabyte, which is also expandable and upgradeable over time as better technology and updates are released.
Was wondering if this mobo able to run on multi 3090's and any ram that uses 3200 DDR4? also what is the recommended PSU for this amusing the CPU are 280x2 and i believe only 3 (2sloth 3090;s can fit)
We’ve updated our terms. By continuing to use the site and/or by logging into your account, you agree to the Site’s updated Terms of Use and Privacy Policy.
Facebook
Twitter
RSS
12 Comments
Back to Article
Chaitanya - Friday, June 4, 2021 - link
What connectors are sitting between 1st 2 PCI-E slots? Very odd locations if the cables are bulky.MenhirMike - Friday, June 4, 2021 - link
Looks like OCuLink.MirrorMax - Saturday, June 5, 2021 - link
Slimline sas sff 8654 4iThere's a total of 5 of these connectors those between PCI ports are angled so they don't interfer with pcie cards
MirrorMax - Saturday, June 5, 2021 - link
Why wasn't Milan support just a bios update to the rev1 board like most other Rome motherboards, i don't see anything new otherwise.SanX - Saturday, June 5, 2021 - link
Which mobo anyone recommend to make ~1000 core AMD EPIC-based supercomputer for floating point 3D simulations like particle-in-cell or molecular dynamics ?Rudde - Saturday, June 5, 2021 - link
I'm curious, is there any reason why EPYC is better than professional GPUs? (Or AVX512)spdcrzy - Sunday, June 6, 2021 - link
I'd go with full servers instead at that point. Gigabyte's 10xGPU 4U server is the obvious choice for supercomputer-level performance unless you choose to go directly with NVidia's DGX A100 - but that's a 6U package instead of the traditional 4U server. The Gigabyte solution has more expansion, though. It supports 128 cores and 10 GPUs along with 8x U.2 NVMe and 4x SATA/SAS drives as well as an OCP 3.0 slot for up to 2x200Gb networking. That gives you 1024 cores, 80 A100 GPUs, a total of up to 32TB of RAM using 128GB DIMMs, 100+ TB of raw U.2 NVMe scratch capacity (at potentially up to 25+ GB/s, which is knocking on the door of DDR4), another 60+ GB of SAS capacity, 400Gb/s interconnect, and a 5 year warranty, all for the cool, cool price of $1.6M and a peak power consumption of over 30kW. 8 DGX A100s would cost about the same, but have less scratch space and take up more room, as well as consume up to 70% more power. But keep in mind that a DGX is a completely self-contained unit, whereas a normal server like the Gigabyte would require additional (and very power hungry) network switches and other management hardware and software. If you're dealing with a dataset that's being accessed remotely, and all you want is plug and play, then the DGX is the winner. But if for whatever reason you need to keep the data in the machine itself, or if the dataset is truly enormous and requires multiple DWPD, then you might be better off with a more classic server like the Gigabyte, which is also expandable and upgradeable over time as better technology and updates are released.SanX - Thursday, June 10, 2021 - link
GPUs do not suit our purposes. Tests show poor performanceSanX - Thursday, June 10, 2021 - link
Same poor performance is with IBM processorsSADRULESXD - Monday, July 5, 2021 - link
will this be able to use 3090s? and any ram that uses 3200?SanX - Thursday, June 10, 2021 - link
Nothing cool is also with price tag and power consumption for 1000-core equivalentSADRULESXD - Monday, July 5, 2021 - link
Was wondering if this mobo able to run on multi 3090's and any ram that uses 3200 DDR4? also what is the recommended PSU for this amusing the CPU are 280x2 and i believe only 3 (2sloth 3090;s can fit)