llama.cpp b10094 Automates Speculative Decoding Type Inference
The llama.cpp release b10094 introduces automatic inference of speculative decoding types from draft repository sidecars. This allows users to run speculative decoding without manually specifying the --spec-type flag when using draft repositories containing sidecars like mtp, dflash, or eagle3. This update simplifies the command-line interface and improves user experience by reducing configuration friction for speculative decoding, which is a key technique for accelerating LLM inference. The inference mechanism prioritizes sidecars in the order of mtp > dflash > eagle3. Providing an explicit --spec-type flag will disable this automatic inference, and draft repositories without sidecars will continue to resolve to full models.
## BACKGROUND
Speculative decoding is an optimization technique that speeds up LLM inference by using a smaller, faster "draft" model to generate candidate tokens, which are then verified in parallel by a larger "target" model. Projects like llama.cpp support various speculative decoding implementations, such as EAGLE, to achieve significant speedups without losing output quality.