Micron Researches "Near-GPU NAND" Flash to Expand AI Memory Capacity
Micron is exploring high-durability "near-GPU NAND" flash modules positioned directly inside GPU packaging or on the PCB. This architecture provides GPUs with direct access to hundreds of gigabytes of high-speed NAND storage for non-latency-critical AI workloads. This concept helps overcome strict VRAM capacity limits during LLM inference, enabling massive AI models to run on fewer physical GPUs. It establishes a new intermediate tier in the memory hierarchy between expensive ultra-fast VRAM (like HBM) and slower traditional storage pools. Unlike standard TLC or QLC NAND, these near-GPU chips offer lower storage density to prioritize higher I/O bandwidth, speed, and endurance. Operating similarly to HBM or GDDR7 integration, the module acts as a fast caching layer directly accessible by the GPU without traversing multi-protocol storage buses.
## BACKGROUND
Running large language models (LLMs) requires holding billions of parameters and context data in memory during inference. High-Bandwidth Memory (HBM) offers the necessary speed but is limited in capacity and very costly, while standard NVMe SSDs have high latency due to PCIe connections. Placing modified NAND flash directly alongside the GPU bridges the memory bottleneck by expanding usable capacity at lower costs.