Except AMD. They knew memory, and bandwidth, was critical to feed modern IPC and have been pushing HBM and large caches for some time. They have a chip in prototype stage with 1GB of on-package cache.
Uh I dunno about that. Sure some 128GB offerings are here/coming, but I think they would have given the MI300 a wide DDR bus if they had seen the LLM craze coming.
Wide DDR buses increases power consumption and PCB size. The RTX4090 has a 384bit bus with 21Gpbs GDDR6X for a total of 1TB/sec bandwidth. Going to a 512bit bus, which is really wide in GPUs, only brings that to 1.3TB/sec. The AMD MI250 has 3.2TB/sec bandwidth with HBM2e. Standard GDDR cannot get the bandwidth you get with HBM.
Samsung stopped HBM production due to losing its JDLA with Netlist. The production they had was infringing. They will have to pay NLST. SK Hynix is the only producer with a licensing agreement with Netlist
We’ve updated our terms. By continuing to use the site and/or by logging into your account, you agree to the Site’s updated Terms of Use and Privacy Policy.
Facebook
Twitter
RSS
9 Comments
Back to Article
brucethemoose - Tuesday, April 18, 2023 - link
A year or two ago, 80GB of RAM on a single accelerator seemed silly.Now... it feels cramped. Pretty much all the hardware makers underestimated how quickly model sizes would scale.
Samus - Tuesday, April 18, 2023 - link
Except AMD. They knew memory, and bandwidth, was critical to feed modern IPC and have been pushing HBM and large caches for some time. They have a chip in prototype stage with 1GB of on-package cache.brucethemoose - Tuesday, April 18, 2023 - link
Uh I dunno about that. Sure some 128GB offerings are here/coming, but I think they would have given the MI300 a wide DDR bus if they had seen the LLM craze coming.schujj07 - Tuesday, April 18, 2023 - link
Wide DDR buses increases power consumption and PCB size. The RTX4090 has a 384bit bus with 21Gpbs GDDR6X for a total of 1TB/sec bandwidth. Going to a 512bit bus, which is really wide in GPUs, only brings that to 1.3TB/sec. The AMD MI250 has 3.2TB/sec bandwidth with HBM2e. Standard GDDR cannot get the bandwidth you get with HBM.Threska - Wednesday, April 19, 2023 - link
Higher power consumption.https://www.quora.com/Is-AMD-Vega-ultimately-a-fai...
nandnandnand - Tuesday, April 18, 2023 - link
AMD has to fix their CUDA problem or die in the LLM haze.Threska - Wednesday, April 19, 2023 - link
Die stacking is the future.https://youtu.be/5uuRduTpl4Q
DanNeely - Wednesday, April 19, 2023 - link
When they only make 20% more HBM than Samsung claiming that SK Hynix is the only company mass producing HBM sounds weird.Mashered Trader - Sunday, May 28, 2023 - link
Samsung stopped HBM production due to losing its JDLA with Netlist. The production they had was infringing. They will have to pay NLST. SK Hynix is the only producer with a licensing agreement with Netlist