Why LLM Inference Speed Matters More Than Marginal Intelligence for AI Agents
A developer discussion highlights the trade-off between LLM capability and inference speed, arguing that once a model meets a baseline for agentic tasks, speed becomes the primary bottleneck for iteration. The author specifies a target performance of approximately 500 tokens per second (tps) for prefill and 25 tps for decode. In agentic workflows where models must execute multiple sequential steps and tool calls, slow inference speeds can make development and debugging impractical. Prioritizing speed over marginal intelligence gains allows developers to iterate faster and build more responsive autonomous systems. The author defines their hardware sweet spot as ~500 tps prefill and ~25 tps decode, choosing to run faster, slightly less capable models if a smarter model cannot meet this threshold. This highlights how hardware constraints directly dictate the choice of LLMs in practical agentic applications.
## BACKGROUND
LLM inference consists of two main phases: the prefill phase, which processes the input prompt in parallel to build the KV cache, and the decode phase, which sequentially generates output tokens. Agentic workflows involve LLMs acting as autonomous agents that plan, use tools, and execute multi-step tasks, requiring frequent and rapid model calls.