Running 8 Concurrent 250K Context Agents on Dual RTX 3090s
A developer successfully ran 8 concurrent coding agents with 250K contexts each on dual RTX 3090 GPUs by offloading the full KV cache to system RAM and using a rank-64 key sketch for token retrieval. This breakthrough demonstrates a novel local implementation of sparse attention and KV cache offloading, enabling high throughput for long-context LLM execution on consumer hardware. The system utilizes custom Triton kernels to build a rank-64 key sketch and fetch only the top ~1% relevant tokens per decode step across PCIe, achieving roughly 600+ tokens per second total across 8 agents.
## BACKGROUND
KV cache offloading stores the key-value states of transformer models in host memory or secondary storage rather than scarce GPU VRAM, significantly reducing memory consumption for long-context tasks. Sparse attention mechanisms reduce computational overhead by focusing only on a subset of tokens rather than computing full attention across the entire context window.