Moore Threads Releases Prefill-as-a-Service Whitepaper for MTT S5000 GPUs
Moore Threads has released a technical whitepaper introducing a "Prefill-as-a-Service" architecture on its MTT S5000 GPUs. This approach decouples the prefill and decode stages of LLM inference to address cost and latency bottlenecks in long-context scenarios. As LLM context windows expand to support up to 1 million tokens, traditional inference methods cause significant resource waste due to mismatched hardware requirements between prefill and decode phases. Implementing disaggregated inference on domestic Chinese GPU hardware like the MTT S5000 is a key step for optimizing local AI infrastructure and reducing total cost of ownership. The architecture separates inference into a compute-bound Prefill pool to optimize Time-to-First-Token (TTFT) and a memory-bound Decode pool to optimize Inter-Token Latency (ITL). This hardware-level separation prevents compute units from idling during decode and avoids memory bandwidth bottlenecks during prefill.
## BACKGROUND
Large Language Model (LLM) inference consists of two main phases: prefill, which processes the input prompt and generates the Key-Value (KV) cache, and decode, which generates subsequent tokens one by one. Prefill is a compute-intensive task limited by floating-point performance, whereas decode is a memory-intensive task limited by memory bandwidth. Running both on the same GPU often leads to inefficient resource utilization because the hardware cannot be optimized for both tasks simultaneously.