The Special Memory Powering the AI Revolution

Category: Memory & HBM · Duration: 13 min · ▶ Watch

Speakers: Asianometry

Switch language → 中文

Segments (8)

  • 00:00:00 · Introduction: AI and High Bandwidth Memory
    • The video introduces High Bandwidth Memory (HBM) as a critical component in modern AI accelerators like the Nvidia H100 and AMD MI300.
  • 00:01:07 · Beginnings: What is HBM?
    • HBM is a JEDEC standard for a 3D-stacked DRAM interface designed to provide very high data transfer rates through multiple independent memory channels.
  • 03:13:00 · Point to Point vs. Wide and Slow
    • The segment contrasts the ‘narrow and fast’ point-to-point architecture of GDDR with HBM’s ‘wide and slow’ approach, which offers better power efficiency and scalability for AI.
  • 06:52:00 · History
    • The HBM standard originated from AMD’s efforts in the late 2000s to overcome the power consumption and scaling limits of GDDR memory.
  • 07:45:00 · TSVs and Microbumps
    • The core technologies enabling HBM’s stacked architecture are Through-Silicon Vias (TSVs) and microbumps, which create vertical connections between dies.
  • 08:53:00 · Building an Ecosystem
    • The biggest challenge in commercializing HBM was not a single technical hurdle but coordinating a complex supply chain ecosystem involving multiple partners.
  • 10:18:00 · Market & Competition
    • The HBM market, once a small niche, is rapidly growing due to the AI boom, with SK Hynix and Samsung dominating the market share.
  • 12:27:00 · Conclusion
    • AI has transformed HBM from a niche, high-end memory into one of the hottest and most critical components in the semiconductor supply chain.

Memory Facts (12)

  • [00:10:00] Nvidia H100 has a die size of 814mm² and is packaged on CoWoS-S.
    • 814mm²
  • [00:11:00] Nvidia H100 is surrounded by 6 stacks of HBM memory.
    • 6
  • [00:12:00] The H100 SXM version uses HBM3, with each stack being 16GB for 80GB total memory.
    • 16GB, 80GB
  • [00:23:00] HBM stands for High Bandwidth Memory.
  • [00:28:00] The Nvidia H100 uses the HBM3 variant, the first commercial product to do so.
    • HBM3
  • [00:35:00] AMD’s MI300 has 8 stacks of HBM, forming 128GB or 192GB of unified memory.
    • 8, 128GB, 192GB
  • [00:39:00] AMD MI300 has a bandwidth of 5.6 TB/s.
    • 5.6 TB/s
  • [00:40:00] AMD MI300 has 72% greater bandwidth and 60% to 140% higher capacity than the Nvidia H100 SXM 80GB.
    • 72%, 60%, 140%
  • [01:30:00] HBM introduces the concept of stacking DRAM dies and running many independent memory channels through the stack.
  • [04:37:00] SK Hynix’s latest HBM3 chip has 12 layers and 24GB of memory.
    • 12, 24GB
  • [06:18:00] HBM features a wide memory bus of 1024 bits per stack.
    • 1024 bits
  • [06:22:00] GDDR5 has a 64-bit memory bus.
    • 64 bits

Bottleneck Claims (3)

  • [03:14:00] GDDR memory architecture is less suitable for heavy AI processing.
    • Evidence: It uses a ‘point-to-point’ connection where each memory channel connects to just one module, making it harder to scale total memory capacity. It also relies on higher clock speeds (‘narrow and fast’), which consumes more power and generates more heat.
  • [05:43:00] Higher clock speeds in memory are an engineering challenge and consume more power.
    • Evidence: This is due to the need to distribute clock signals and the fact that transistors must switch more frequently, leading to higher power consumption and heat, bumping against system power budgets.
  • [07:25:00] GDDR was reaching its limits, especially concerning power consumption.
    • Evidence: This limitation was the primary motivation for AMD to start developing the architecture that would become HBM.

Predictions (1)

  • [10:33:00, 2023-2028] The HBM chip market is forecasted to grow from $2 billion in 2023 to $6.3 billion by 2028.

Key Technologies (6)

  • HBM (High Bandwidth Memory): A JEDEC standard for a 3D-stacked DRAM interface that provides high bandwidth and power efficiency through a wide, parallel memory bus.
  • Silicon Interposer: A layer of silicon used to connect multiple HBM die stacks to a GPU or CPU, acting like a mini-PCB.
  • DDR (Double Data Rate): A standard for general-purpose memory modules used in PCs.
  • GDDR (Graphics Double Data Rate): The traditional memory interface standard for graphics cards, characterized by a ‘narrow and fast’ architecture.
  • TSV (Through-Silicon Via): Vertical electrical connections (small holes) that pass through a silicon wafer or die, enabling 3D-stacked chips like HBM.
  • Microbumps: Tiny solder joints that connect the dies in a stacked configuration, used in conjunction with TSVs.

Companies Mentioned (9)

Nvidia · SK Hynix · AMD · JEDEC · UMC · ASE Group · Samsung · Micron · TSMC

Notable Quotes (2)

An explanation from a 2015 interview where he, as a senior fellow at AMD, explained that his team started architecting HBM after seeing the limits of GDDR, especially regarding power consumption. — Bryan Black (AMD) @ 07:14:00

He related that the first HBM product didn’t encounter any single overwhelming technical challenge, but rather the biggest problem was working through a long list of all the ‘newness’. — Bryan Black (AMD) @ 09:08:00

Key Topics

High Bandwidth Memory (HBM) · AI Accelerators (Nvidia H100, AMD MI300) · DRAM Technology · GDDR vs. HBM · Semiconductor Packaging (CoWoS, Interposers) · 3D Stacking · Through-Silicon Vias (TSVs) · JEDEC Standards · Memory Bandwidth · Power Consumption in Memory · Semiconductor Supply Chain

Takeaways

  • High Bandwidth Memory (HBM) is a critical enabling technology for the current AI boom, used in top accelerators from Nvidia and AMD.
  • HBM achieves superior bandwidth and power efficiency over traditional GDDR by using a ‘wide and slow’ architecture: a very wide memory bus (1024-bit) with a lower clock speed.
  • The technology relies on advanced 3D packaging, stacking multiple DRAM dies vertically and connecting them with Through-Silicon Vias (TSVs) and microbumps.
  • The HBM standard was pioneered by AMD and SK Hynix to overcome the power and scaling limitations of GDDR memory for high-performance computing.
  • Building the HBM ecosystem was a massive challenge, requiring coordination between chip designers (AMD), memory makers (SK Hynix), foundries (UMC), and packaging firms (ASE).
  • The HBM market is currently dominated by South Korean companies SK Hynix and Samsung, who are fiercely competing and investing heavily to expand capacity to meet surging AI-driven demand.