Cactus Hybrid: Training Gemma 4 to Predict Its Own Correctness for Smart Routing
Cactus Compute has introduced Cactus Hybrid, a method that post-trains Gemma 4 E2B with a lightweight 68k-parameter probe layer to output confidence scores based on its hidden states. This allows developers to dynamically route low-confidence queries from local devices to larger cloud models like Gemini 3.1 Flash-Lite. This hybrid routing approach significantly reduces API costs and latency by keeping 65-85% of queries on-device while maintaining the performance level of frontier cloud models. It solves the unreliability of traditional routing methods like token entropy or self-evaluation prompts. The probe layer reads an intermediate layer during decoding to predict the probability of being wrong, achieving an average of 0.814 AUROC compared to 0.549 for token entropy. However, it currently only supports single-sequence decoding up to 1024 tokens and must be custom-built for each model.
## BACKGROUND
In hybrid AI architectures, small models run locally on-device for speed and privacy, while larger models run in the cloud for complex tasks. LLM routing is the process of deciding which model should handle a given query, but traditional heuristics like token entropy (measuring the uncertainty of the next predicted word) often fail to accurately reflect whether the model's final answer is correct.