DynamicTune Distills 0.8B LLM to Outperform 2B Model Without Backpropagation

Researchers introduced DynamicTune, a technique using zero-backprop closed-form weight surgery to transfer reasoning dynamics from a larger teacher model to Qwen3.5-0.8B in approximately 12 minutes on consumer hardware. The modified 0.8B student achieved a 42.15% normalized accuracy on the ARC-Challenge benchmark, outperforming both traditional supervised fine-tuning and the uncompressed Qwen3.5-2B baseline model (41.10%). This method challenges the traditional belief that model parameter scale and gradient-based backpropagation are strictly required to enhance reasoning capabilities in smaller language models. By enabling rapid knowledge injection on a single consumer GPU without training tokens or gradient descent, it significantly lowers the hardware barrier for LLM optimization. To prevent representation collapse caused by high spectral entropy in intermediate layers, DynamicTune restricts weight surgery to just four sparse anchor layers (0, 7, 15, and 23) using damped Levenberg-Marquardt Tikhonov pseudoinverse and adaptive spectral rank truncation. The benchmarks were independently verified by TPN Bench on datacenter NVIDIA L4 GPUs using the standard lm_eval framework, showing a statistically significant 3.23-sigma gain.