Micron Presentation Highlights Widening AI Memory Wall Bottleneck
Data from Micron presented at Hot Chips reveals that while AI compute performance scales at roughly 3x every two years, HBM bandwidth grows at less than 2x. This widening gap is driving the industry toward near-memory and processing-in-memory (PIM) solutions, such as Samsung's PIM-enabled LPDDR5X which achieved a 3.01x token generation speedup on Llama 3.1 8B. Because Large Language Model (LLM) inference is severely limited by memory bandwidth rather than raw compute, traditional GPU/TPU scaling alone can no longer provide optimal performance gains. Integrating compute capabilities directly within memory architectures is fast becoming a necessary paradigm shift for future AI hardware efficiency. The presented chart compares compute scaling from Google TPU v3 to Nvidia R200 against memory bandwidth scaling from HBM2e to HBM4 on a logarithmic scale, illustrating an accelerating divergence. Proposed remedies range from placing memory physically closer to the processor via shorter interconnects to embedding multiply-accumulate units directly inside the memory dies.
## BACKGROUND
The 'memory wall' is the growing performance imbalance between fast compute processors and relatively slower memory data transfer bandwidth. In AI workloads like LLM inference, processors spend considerable time idling while waiting for massive model parameters to move from off-chip DRAM or HBM into compute registers. Processing-in-Memory (PIM) addresses this bottleneck by adding small execution units directly inside memory chips to compute data locally without shuttling it over external memory buses.