This really doesn't seem all that odd. I would imagine if you want a supercomputer with the most processing power you'd want it to have CPUs with the highest core count, and AMD wins that by a landslide. I don't really think the fact that AMD competes with them in a different department would make them pass on the best choice for what they're doing.
Well, if was purely core count, there are other CPUs. We have to include performance (on top of I/O). With 2 CPUs, Rome gives 128PCIe4.0 lanes (and 128 cores), though 128 lanes aren't enough for 8 GPUs + (several) NIC + NVMe, it is a lot better than anything short of 256 lanes PCIe3.0. Why is it a lot better? I'd hate to route 256 PCIe 3.0 lanes, nevermind the switch chips, and the accelerators only have 16 lanes max - PCIe 3.0 loses half of the bandwidth. This is especially important for the 200GBps+ NICs.
IBM POWER9 may have the I/O, but it doesn't have the cores nor much remaining acceptance in the community. IBM tried to bid for two of the exaflop projects, with and without Nvidia, and lost both (I don't know if Nvidia had any serious bids on their own). Nvidia has their own arm CPUs with PCIe4.0 and NVLink, but those are scaled for embedded systems, not HPC. Ampere and Cavium/Marvell, like Nvidia, didn't have the full arm infrastructure at the time of the contracts, anyways (they seem to have it now).
We also have Fujitsu with their A64FX arm-SVE CPU, but historically, Japanese CPUs have failed to gain traction outside of Japan. They seem to be making a serious push for exports this time around, however. But either way, it would have been too late for the exaflop contracts.
In the end, I agree AMD won this part of the supercomputer by a landslide, just for different reasons than core count alone. :D
In a dual socket system, Gen 2 Epyc can be configured to have up to 160 PCIe 4.0 lanes. This is done by reducing the IF lanes from 128 to 96 and adding the extra lanes to IO. Even in the 96 lane configuration, Gen 2 Epyc has more 50% more CPU-CPU bandwidth than it did in Gen 1.
NVSwitch connects the 8 to 16 GPUs together into a single GPU - the AMD system is just a traffic cop and IO. There are no direct CPU to individual GPU connection - the AMD connects to the 8/16GPU Cluster. The AMD system does not provide any compute performance to the mix - much as the previous DGX systems. So the cores in the A100 GPUs are the overwhelming vast majority of the compute power here.
These are Nvidia DGX-A100 systems - 95% of the compute power is from the 8 to 16 Ampere A100 GPUs connected via NVSwitch. The AMD system is a traffic cop and IO - they do not provide any compute power to the mix - exactly like the Xeons were just traffic cops and IO on the previous DGX systems.
IIRC, AMD are fervent about meeting 10% market share using what Lisa Su says is the sensible metric of achieving the volume to reach 10% of what the market was predicted to be some time in the past. So, since the current market size prediction is larger than the old one, they do not expect to reach 10% of the actual market. It only makes sense if AMD's competitors are somehow benefitting from an overall demand increase that AMD are not.
Makes sense to me. Counting your growth as a "failure" because you landed where you aimed to - but the target moved in ways you didn't expect at the time you took aim - isn't really a reflection of the effort put in.
It's not the sort of thing I'd expect them to do indefinitely, but at the moment they're approaching a heavily entrenched market from what is basically a standing start.
So this exact setup scaled up 40x should hit EFLOPS level with roughly 50 MWatts energy use. Seems like a non AI focused machine in 2023 hitting 2 EFLOPS with a 30MWatt power target should be a relatively easy target.
That's my favorite comment here so far, but then, I'm in a strange mood this evening... But, seriously, good to see that EPYC is making inroads here. Regarding older AMD systems, maybe I should take my old Athlon x64 (two cores!) out of storage and see what's what over a decade later.
If I had a spare ATX case right now, I'd boot my 1.3 Thunderbird up. But I don't, so I won't.
What I DO have is a 500 MHz K6-2 in an AT case. ... Yeah, that's not gonna be going very far in the modern world. I GUESS I could install Windows XP on it, but... it doesn't sound like a good time. Let's keep it with DOS and Win98.
... I mean, it doesn't seem wrong to me. A100s are significantly more expensive, have about 2.5x peak fp64 (from a quick google), and there are 4 per 7742.
"Just traffic cops" Not really. Not at all, in fact. That analogy breaks almost immediately because EPYC also forms the backbone for most of the traffic *infrastructure*.
Even if that weren't the case, not all traffic control systems are equivalent. If your city had to budget for 1000 extra traffic cops a year because they hadn't developed the tech to use traffic lights yet, that'd be a problem. In case it wasn't clear, this is an equally strained analogy for AMD having PCIe 4.0 and 64 cores per socket and Intel... not.
Nobody's mentioned where the name came from? Adam Selene. I doubt it's a coincidence. It's the last name of the created chairman in Robert Heinlein's The Moon is a Harsh Mistress. The colonists on the moon are trying to win independence from Earth, the computer develops autonomy, befriends his repairman, and they, along with a couple others, create a persona for the computer to project.
Selene means "(the) Moon" in ancient greek. Most often referred to as the Goddess of the Moon. The reference could imply any number of books, plays, songs, etc
I think the number of DGX nodes is actually 280, so there's twice as many A100s and EPYCs as listed in this article. Then the numbers will match: 280*2*64+280*8*108 = 277,760.
An update to your DoE chart ... "Aurora will use Cray’s next-generation supercomputer system, code-named “Shasta,” which will comprise more than 200 cabinets ..." https://newsroom.intel.com/news-releases/u-s-depar...
I think this article is a bit wrong, Selene uses 280 DGX A100 instead of 140. I don't know whether this is an extension to the previously announced SuperPOD or a completely new system but there's an article by NVIDIA themself stated that Selene uses 280 DGX A100 with 2240 GPUs in it instead of half as much as said by this article.
My old boss at Argonne National Labs said they are almost finished retrofitting the previous Xeon Phi-based CRAY/HPE cabinets the Aurora Exascale is replacing. The reason for the 2021 launch is they go through like 9 months of validation which seem ridiculous but these installations are damn near rocket science when it comes to coding the scheduler. I worked at Argonne 20 years ago when the "terrabyte storage facility" was considered hot with around 10TB of storage - critical for the advance photon source (APS) synchrotron beam data dumps.
We’ve updated our terms. By continuing to use the site and/or by logging into your account, you agree to the Site’s updated Terms of Use and Privacy Policy.
Facebook
Twitter
RSS
46 Comments
Back to Article
inighthawki - Monday, June 22, 2020 - link
This really doesn't seem all that odd. I would imagine if you want a supercomputer with the most processing power you'd want it to have CPUs with the highest core count, and AMD wins that by a landslide. I don't really think the fact that AMD competes with them in a different department would make them pass on the best choice for what they're doing.jeremyshaw - Monday, June 22, 2020 - link
Well, if was purely core count, there are other CPUs. We have to include performance (on top of I/O). With 2 CPUs, Rome gives 128PCIe4.0 lanes (and 128 cores), though 128 lanes aren't enough for 8 GPUs + (several) NIC + NVMe, it is a lot better than anything short of 256 lanes PCIe3.0. Why is it a lot better? I'd hate to route 256 PCIe 3.0 lanes, nevermind the switch chips, and the accelerators only have 16 lanes max - PCIe 3.0 loses half of the bandwidth. This is especially important for the 200GBps+ NICs.IBM POWER9 may have the I/O, but it doesn't have the cores nor much remaining acceptance in the community. IBM tried to bid for two of the exaflop projects, with and without Nvidia, and lost both (I don't know if Nvidia had any serious bids on their own). Nvidia has their own arm CPUs with PCIe4.0 and NVLink, but those are scaled for embedded systems, not HPC. Ampere and Cavium/Marvell, like Nvidia, didn't have the full arm infrastructure at the time of the contracts, anyways (they seem to have it now).
We also have Fujitsu with their A64FX arm-SVE CPU, but historically, Japanese CPUs have failed to gain traction outside of Japan. They seem to be making a serious push for exports this time around, however. But either way, it would have been too late for the exaflop contracts.
In the end, I agree AMD won this part of the supercomputer by a landslide, just for different reasons than core count alone. :D
inighthawki - Monday, June 22, 2020 - link
Good points!Wafflefries128 - Tuesday, June 23, 2020 - link
Two 7742 CPUs equate to 256 pcie lanes.With 8 gpus at x16 that's 128 pcie lanes for the GPUs, leaving an additional 128 lanes for storage, networking and other peripheral hardware.
Wafflefries128 - Tuesday, June 23, 2020 - link
Woops forgot about infinity fabric...nevermind you're right!schujj07 - Tuesday, June 23, 2020 - link
In a dual socket system, Gen 2 Epyc can be configured to have up to 160 PCIe 4.0 lanes. This is done by reducing the IF lanes from 128 to 96 and adding the extra lanes to IO. Even in the 96 lane configuration, Gen 2 Epyc has more 50% more CPU-CPU bandwidth than it did in Gen 1.Deicidium369 - Tuesday, June 23, 2020 - link
No. Each Epyc CPU has 128 Lanes. 64 of those lanes are used to connect to a 2nd CPU - leaving only 128 Lanes. Epyc tops out at 2 sockets.Intel has QPI/UPI to facilitate multi socket systems - and does not use PCIe Lanes
Deicidium369 - Tuesday, June 23, 2020 - link
Epyc has 128 PCIe4 lanes - 64 of which are used to connect to a 2nd CPU - there are no 4 or 8 socket EpycDeicidium369 - Friday, June 26, 2020 - link
NVSwitch connects the 8 to 16 GPUs together into a single GPU - the AMD system is just a traffic cop and IO. There are no direct CPU to individual GPU connection - the AMD connects to the 8/16GPU Cluster. The AMD system does not provide any compute performance to the mix - much as the previous DGX systems. So the cores in the A100 GPUs are the overwhelming vast majority of the compute power here.808Hilo - Sunday, July 12, 2020 - link
Japanese chips? I d hate to read their user manual.https://www.daytranslations.com/blog/wp-content/up...
Att. Ryan Smith
eat less. You look sick and obesity hurts the brain.
Deicidium369 - Friday, June 26, 2020 - link
These are Nvidia DGX-A100 systems - 95% of the compute power is from the 8 to 16 Ampere A100 GPUs connected via NVSwitch. The AMD system is a traffic cop and IO - they do not provide any compute power to the mix - exactly like the Xeons were just traffic cops and IO on the previous DGX systems.Yojimbo - Monday, June 22, 2020 - link
IIRC, AMD are fervent about meeting 10% market share using what Lisa Su says is the sensible metric of achieving the volume to reach 10% of what the market was predicted to be some time in the past. So, since the current market size prediction is larger than the old one, they do not expect to reach 10% of the actual market. It only makes sense if AMD's competitors are somehow benefitting from an overall demand increase that AMD are not.Spunjji - Friday, June 26, 2020 - link
Makes sense to me. Counting your growth as a "failure" because you landed where you aimed to - but the target moved in ways you didn't expect at the time you took aim - isn't really a reflection of the effort put in.It's not the sort of thing I'd expect them to do indefinitely, but at the moment they're approaching a heavily entrenched market from what is basically a standing start.
ewjop - Monday, June 22, 2020 - link
"Now, in June 2010, AMD scores a big win for its Zen 2 microarchitecture by getting to #7. But there's a twist in this tale."June 2010 with Zen 2 architecture? Typo(?)
ozzuneoj86 - Monday, June 22, 2020 - link
" Now, in June 2010, AMD scores a big win for its Zen 2 microarchitecture by getting to #7. But there’s a twist in this tale."Typo, June 2010.
boozed - Monday, June 22, 2020 - link
"The previous best AMD system, built on Opteron CPUs, was Titan, which held the #1 spot in 2012 but slowly dropped out of the top 10 by June 2019."It seems surprising that it would take seven years to drop out of the top 10!
yeeeeman - Tuesday, June 23, 2020 - link
Supercomputers are not built and exchanged like socks. They are usually investments for many years, 10 years at least.Deicidium369 - Tuesday, June 23, 2020 - link
US top 500BenSkywalker - Monday, June 22, 2020 - link
So this exact setup scaled up 40x should hit EFLOPS level with roughly 50 MWatts energy use. Seems like a non AI focused machine in 2023 hitting 2 EFLOPS with a 30MWatt power target should be a relatively easy target.nandnandnand - Monday, June 22, 2020 - link
2021: 1.5 EFLOPs, ~30 MW. Close enough.bananaforscale - Wednesday, June 24, 2020 - link
The unit is FLOPS, not FLOP. FLoating point Operations Per Second. Not FLOP.Someguyperson - Monday, June 22, 2020 - link
I'm sure Jensen got the "friends & family discount" from his first cousin once removed.PreacherEddie - Monday, June 22, 2020 - link
And my Phenom II X3 sits just outside the Top 500, although it is not new for this year.eastcoast_pete - Monday, June 22, 2020 - link
That's my favorite comment here so far, but then, I'm in a strange mood this evening...But, seriously, good to see that EPYC is making inroads here. Regarding older AMD systems, maybe I should take my old Athlon x64 (two cores!) out of storage and see what's what over a decade later.
Lord of the Bored - Tuesday, June 23, 2020 - link
If I had a spare ATX case right now, I'd boot my 1.3 Thunderbird up. But I don't, so I won't.What I DO have is a 500 MHz K6-2 in an AT case.
...
Yeah, that's not gonna be going very far in the modern world. I GUESS I could install Windows XP on it, but... it doesn't sound like a good time. Let's keep it with DOS and Win98.
Spunjji - Friday, June 26, 2020 - link
My old XP install on an Athlon 750 with 128MB RAM was a bad time, so I imagine it would be pretty heinous on a K6-2 D:schujj07 - Tuesday, June 23, 2020 - link
That CPU was faster than the fastest supercomputers in the world in the mid 90s.yeeeeman - Tuesday, June 23, 2020 - link
Well, this is kind a lacklustre win, since most of the compute power in this supercomputer is made by the nvidia agx100 cards, not by the epycs.Sharma_Ji - Tuesday, June 23, 2020 - link
Run troll, run.Unashamed_unoriginal_username_x86 - Tuesday, June 23, 2020 - link
... I mean, it doesn't seem wrong to me. A100s are significantly more expensive, have about 2.5x peak fp64 (from a quick google), and there are 4 per 7742.Deicidium369 - Tuesday, June 23, 2020 - link
He is correct - the CPU (whether it's Intel or AMD) are really just traffic cops - the main computational power if from the GPUsAnarchoPrimitiv - Friday, June 26, 2020 - link
What do you think is in charge of feeding enough information to those GPUs to ensure that they don't waste cycles being idle?Spunjji - Friday, June 26, 2020 - link
"Just traffic cops"Not really. Not at all, in fact. That analogy breaks almost immediately because EPYC also forms the backbone for most of the traffic *infrastructure*.
Even if that weren't the case, not all traffic control systems are equivalent. If your city had to budget for 1000 extra traffic cops a year because they hadn't developed the tech to use traffic lights yet, that'd be a problem. In case it wasn't clear, this is an equally strained analogy for AMD having PCIe 4.0 and 64 cores per socket and Intel... not.
msroadkill612 - Tuesday, June 23, 2020 - link
Size counts for nerds too.teldar - Tuesday, June 23, 2020 - link
Nobody's mentioned where the name came from?Adam Selene. I doubt it's a coincidence.
It's the last name of the created chairman in Robert Heinlein's The Moon is a Harsh Mistress.
The colonists on the moon are trying to win independence from Earth, the computer develops autonomy, befriends his repairman, and they, along with a couple others, create a persona for the computer to project.
SaturnusDK - Tuesday, June 23, 2020 - link
Selene means "(the) Moon" in ancient greek. Most often referred to as the Goddess of the Moon. The reference could imply any number of books, plays, songs, etceddman - Wednesday, June 24, 2020 - link
Uuu, what?!https://en.wikipedia.org/wiki/Selene
nft76 - Tuesday, June 23, 2020 - link
I think the number of DGX nodes is actually 280, so there's twice as many A100s and EPYCs as listed in this article. Then the numbers will match: 280*2*64+280*8*108 = 277,760.Deicidium369 - Tuesday, June 23, 2020 - link
Typical DGX install has 2 CPUs and upto 16 GPUs. NVSwitch tops out currently at 16 GPUsFyzhkar - Tuesday, June 23, 2020 - link
Yep, there's an article by NVIDIA themself that stated thisMooreInfo - Tuesday, June 23, 2020 - link
An update to your DoE chart ... "Aurora will use Cray’s next-generation supercomputer system, code-named “Shasta,” which will comprise more than 200 cabinets ..."https://newsroom.intel.com/news-releases/u-s-depar...
Fyzhkar - Tuesday, June 23, 2020 - link
I think this article is a bit wrong, Selene uses 280 DGX A100 instead of 140. I don't know whether this is an extension to the previously announced SuperPOD or a completely new system but there's an article by NVIDIA themself stated that Selene uses 280 DGX A100 with 2240 GPUs in it instead of half as much as said by this article.Deicidium369 - Tuesday, June 23, 2020 - link
Depends on whether the DGX A100 is 8 GPU or 16,nft76 - Tuesday, June 23, 2020 - link
I believe DGX A100s only have 8 GPUs. DGX-2 did come with 16 V100s though.mode_13h - Wednesday, June 24, 2020 - link
I think it's only 8, because they each have more links.Samus - Wednesday, June 24, 2020 - link
My old boss at Argonne National Labs said they are almost finished retrofitting the previous Xeon Phi-based CRAY/HPE cabinets the Aurora Exascale is replacing. The reason for the 2021 launch is they go through like 9 months of validation which seem ridiculous but these installations are damn near rocket science when it comes to coding the scheduler. I worked at Argonne 20 years ago when the "terrabyte storage facility" was considered hot with around 10TB of storage - critical for the advance photon source (APS) synchrotron beam data dumps.