Basalt 0.4.0 Released with Qwen 27B Achieving Up to 770 Tok/s on RTX 5090
Basalt version 0.4.0 has been released, introducing multi-model support, video processing, and its first dense model execution for the Qwen 27B model. This setup achieves up to 770 tokens per second for structured output on an RTX 5090 graphics card. This release demonstrates massive performance milestones for running large language models locally on consumer hardware through optimized custom kernels and quantization. It highlights how specialized inference engines can dramatically reduce latency and increase throughput for everyday users and developers. The Qwen 3.8 27B dense model maintains 92.8% top-1 accuracy compared to the full BF16 model while delivering up to 776 tok/s on structured outputs, 262 tok/s in conversation, and a prefill rate of 12,000 tok/s at a 4k context window.
## BACKGROUND
Quantization reduces the precision of model weights from high-precision formats like 32-bit floats to lower bits to shrink model size and speed up inference. Dense models activate all parameters for every input during inference, offering simpler training but traditionally lower throughput compared to sparse architectures.