NVIDIA's H100 has: 80GB HBM3, 3.35TB/s bandwidth, 990 TFLOPS FP16

NVIDIA's A100 has: 80GB HBM2e, 2TB/s bandwidth, 312 TFLOPS FP16

NVIDIA's A100 features: 80GB HBM2e, 2TB/s bandwidth, 312 TFLOPS FP16

NVLink provides: high-bandwidth GPU-to-GPU interconnect (900 GB/s on H100)

NVLink provides: high-bandwidth GPU-to-GPU interconnect (900 GB/s on H100)

HBM (High Bandwidth Memory) provides: stacked DRAM with much higher bandwidth than DDR

High Bandwidth Memory (HBM) provides stacked DRAM with much higher bandwidth than DDR

PCIe bandwidth limits: ~64 GB/s for PCIe 5.0 x16, bottleneck for CPU-GPU transfer

PCIe 5.0 x16 bandwidth limit ~64 GB/s, bottleneck for CPU-GPU transfer

2024–present global memory supply shortage

Global DRAM shortage began in 2024

CPU cache

L1/L2 cache hierarchy reduces global memory latency