DLite: History-Free Speculative Decoding for Agentic AI
Researchers have introduced DLite, a history-free speculative decoding framework designed for agentic AI that reuses target model hidden states to perform local speculative predictions without maintaining historical KV cache. Experiments show that DLite achieves an acceptance length of 2 to 6 and outperforms state-of-the-art methods like DSpark by up to 70%. Standard draft models suffer up to an 80% degradation in acceptance length during agentic workloads due to long, continuously evolving contexts and high KV cache overhead. DLite transforms this complex long-context modeling problem into a local simple prediction problem, significantly reducing system overhead and accelerating inference for agentic AI applications. DLite utilizes a lightweight sequential head to restore intra-block causal dependencies with low overhead and is trained using a tailored two-stage training strategy. Code and checkpoints are made available via GitHub and HuggingFace.
## BACKGROUND
Speculative decoding is an inference-time optimization for large language models that uses a smaller draft model to propose multiple candidate tokens, which are then verified simultaneously by a larger target model. The KV cache (Key-Value cache) stores intermediate key and value vectors from previous input tokens during autoregressive inference, but grows linearly with sequence length and creates heavy resource bottlenecks.