LittleBit: Ultra-Low Bit Quantization via Latent Factorization for LLMs
Researchers introduced LittleBit, a novel quantization-aware training method that compresses large language models down to 0.1 bits per weight. The technique uses SVD-inspired latent matrix factorization to binarize factor matrices, reducing Llama2-13B to under 0.9 GB, representing an approximate 31× memory reduction. Ultra-low-bit quantization makes running 13B-parameter language models feasible on consumer devices with extremely limited RAM and VRAM. This significantly lowers the barrier for deploying efficient, private local AI models on edge hardware. To mitigate severe information loss from binarizing low-rank factors, LittleBit incorporates a multi-scale compensation mechanism across row, column, and latent dimensions to learn per-rank importance. This combined approach allows the model to maintain functional accuracy even under extreme bitrates.
## BACKGROUND
Model quantization reduces memory usage by storing floating-point neural network weights using lower-precision formats. Quantization-Aware Training (QAT) simulates these precision losses during the training phase, allowing the model's weights to adjust dynamically and preserve performance far better than post-training quantization.