Developer Pushes 27B LLM Inference to 2,000 Prefill Tokens/Sec on RTX 3090
Developer /u/iamMess released custom CUDA optimizations for Qwen 27B inference on a single consumer RTX 3090 GPU, achieving nearly 2,000 prefill tokens per second and 132 decode tokens per second. The performance boost relies on a custom INT8 kernel that significantly accelerates prompt prefilling while preserving output quality. Achieving enterprise-grade throughput on a 27B parameter model using a single consumer GPU makes local LLM hosting dramatically faster and more practical for individual developers. It demonstrates how low-level quantization and speculative decoding techniques can stretch the limits of consumer AI hardware. The developer's custom INT8 kernel achieves a 0.99997 similarity score compared to FP32 floating-point calculations, rendering quality loss virtually imperceptible. Code and benchmarks for testing these optimizations have been made publicly available on GitHub.
## BACKGROUND
LLM inference consists of two phases: prefill, where the entire input prompt is processed in parallel, and decode, where new tokens are generated sequentially one by one. Speculative decoding speeds up the decode phase by using a smaller draft model to propose tokens that the larger target model verifies in parallel.