SpecPilot: Autonomous Algorithm Discovery and Systems Optimization for Speculative Decoding
SpecPilot is a new autonomous agent-based framework designed to discover and optimize algorithms and system configurations specifically for speculative decoding in large language models. Without requiring training or causing any quality loss, it significantly increases inference speedups compared to stock systems. This framework addresses the limitation that standard speculative decoding configurations do not fit every deployment, drafter, target model, or GPU. By automating algorithm discovery, it pushes performance much closer to theoretical limits across various serving environments. SpecPilot utilizes a cost model, parallel agents, token forcing to prevent reward hacking, and structured memory to find mechanisms like a 12-bit weight format and new batching strategies. Across evaluations, it raised single-request speedups over autoregressive decoding from 4.46x to 10.87x and batching speedups from 1.94x to 3.93x.
## BACKGROUND
Speculative decoding is an inference-time optimization technique for autoregressive large language models that generates multiple candidate tokens per decoding step using a smaller draft model, which are then verified simultaneously by a larger target model. Autoregressive decoding is a process where a model generates a sequence of outputs recursively by predicting the next token based on previous outputs, which is often memory-bound and slow.