Original Link: https://www.anandtech.com/show/14771/hot-chips-31-live-blogs-genz-chipset-for-exascale-fabric
Hot Chips 31 Live Blogs: Gen-Z Chipset for Exascale Fabric
by Dr. Ian Cutress on August 20, 2019 5:40 PM EST- Posted in
- Hot Chips
- Interconnect
- Live Blog
- Gen-Z
- HPE
06:08PM EDT - That's a wrap, time for a break.
06:07PM EDT - Using 16FF TSMC CoWoS
06:04PM EDT - Cadet Cluster up to 13 DL-385
06:03PM EDT - Cadet is an early access platform
06:03PM EDT - Prototype builds coming soon
06:01PM EDT - This shows the three chip types
06:01PM EDT - labs
06:01PM EDT - Test chips already in the laps
06:00PM EDT - Next gen will use 400g signalling, PCIe G5
06:00PM EDT - 5x messaging vs the competition
05:59PM EDT - One mask/one reticle does one switch chip, one bridge chip, and three e20 modules
05:59PM EDT - All manufactured on one set of masks
05:58PM EDT - Uses bridge die and switch die
05:58PM EDT - Also have electrical to optical module
05:58PM EDT - 80ns idle switch latency
05:57PM EDT - 1D all to all mesh
05:57PM EDT - 25.6mm x 47.5mm with 5 chips
05:57PM EDT - 14B transistors
05:57PM EDT - 2.5D packaging
05:56PM EDT - 6 fibers per dicetion = 1.27 Tbps per optical module
05:56PM EDT - 212 Gbps per fiber with 4 fibers
05:56PM EDT - 2 bits per symbol with PAM4
05:55PM EDT - Integrated optics into the chipset avoids the need for excess SerDes activations
05:54PM EDT - Optical simplifies it, and saves power
05:53PM EDT - Can be cost competitive with copper in the rack
05:52PM EDT - Motivation for optical interconnect
05:52PM EDT - packet deadlines eliminate any chance of duplicate packets in the fabric
05:51PM EDT - Support explicit congestion notification to the requesters
05:51PM EDT - Chipset manages fabric congestion
05:50PM EDT - low latency 2ns FEC is available if needed
05:50PM EDT - 3 different levels of CRC, 2 levels of retry
05:49PM EDT - Five of these switches are interconnected to build the 60-port switch
05:48PM EDT - age-based arbitration
05:48PM EDT - 16 virtual channels
05:48PM EDT - Fully buffered crossbar organization
05:48PM EDT - Can do it to any fabric attached memory
05:47PM EDT - Support direct load-store access
05:46PM EDT - Provide 12 optical Gen-Z links
05:46PM EDT - Gen-Z bridge connected to four x16 PCIe 4.0 links
05:46PM EDT - Built with Rome as it has enough PCIe
05:45PM EDT - Bridges are required as no CPU is natively Gen-Z today
05:44PM EDT - 60-port switch module has integrated optical Gen-Z links
05:44PM EDT - 3D HyperX scalable to 61440 nodes with 4 swich crossings
05:44PM EDT - Develop a topology to create system fabric based on Gen-Z and HyperX topology
05:43PM EDT - Supports direct attach and switched fabric
05:43PM EDT - Supports cluster based interconnects
05:43PM EDT - Gen-Z is memory semantic
05:42PM EDT - Customers are doing these workloads, and requiring infrastructure to do them
05:42PM EDT - Dataset sizes are creating complexity and compute requirements
05:41PM EDT - Big data analytics and modelling are what customers are running
05:41PM EDT - Funded in part by DoE
05:40PM EDT - One of the key competing interconnects of the future is Gen-Z, and Hewlett Packard Enterprise have a Gen-Z chipset to show at Hot Chips today.