Fast Local LLM Decode Speeds Shift Performance Bottleneck to CPU Tool Calls
A developer optimizing a custom local LLM inference engine on an Nvidia RTX 5090 achieved decode speeds around 700 tokens per second across 12 server slots, only to discover that CPU-bound tool call execution became the main performance bottleneck. Running 25 concurrent AI agents completely maxed out an AMD Ryzen 7 9800X3D CPU across all threads while the GPU remained underutilized. As next-generation GPUs drastically accelerate LLM prefill and decode speeds, system bottlenecks are shifting from VRAM bandwidth and GPU compute to classical CPU and I/O tasks. This highlights that scaling complex multi-agent coding workflows requires balanced hardware setups rather than just raw GPU power. By reusing prefills for jobs sharing initial context and refining prompt engineering to submit fewer, more compact tasks (reducing prompts from 20,000 to 7,000), total execution time dropped from 43 to 18 minutes. The developer also found that keeping more active agents than available server slots helped absorb inevitable delays during CPU-bound tool execution and testing.
## BACKGROUND
LLM inference consists of a prefill phase, where the input prompt is processed in parallel to build the key-value (KV) cache, and a decode phase, where tokens are generated sequentially. In agentic AI workflows, models rely on tool calling (or function calling) to execute external code, run tests, or query systems. When GPU decode speeds are fast enough, executing these CPU-bound external tools becomes the rate-limiting step in multi-agent environments.