Tencent Open-Sources Compressed 770B Hy4 MoE Model, Shrinking Footprint to 214GB
Tencent's Hunyuan team has open-sourced a compressed, lightweight release of its 770B parameter Hy4 preview Mixture-of-Experts model, shrinking its weight size from nearly 1.5TB to approximately 214GB. The model utilizes the Sherry sparse ternary quantization algorithm alongside a MIX-STQ1_0 mixed-precision strategy to drastically cut memory requirements while preserving benchmark capabilities. This release makes serving massive 700B+ parameter MoE models feasible on accessible consumer or mid-tier hardware setups rather than expensive enterprise GPU clusters. Additionally, it demonstrates practical multi-device heterogeneous inference, allowing laptops and secondary desktop GPUs to split low-bit MoE workloads over standard local networks. The compression leverages Sherry's 1.25-bit sparse ternary quantization on expert layers, dynamic bit-width selection between 1.31-bit (STQ1_0) and 2.06-bit (IQ2_XXS), and maintains long-context reading capabilities near the original BF16 baseline. Benchmark scores showed minimal drops, with MCP Atlas slipping slightly from 83.7 to 83.2 and SWE-Bench multi from 82.9 to 81.3, achieving 1.02 tokens/sec when offloaded across a laptop and an A4000 server via prima.cpp.
## BACKGROUND
Quantization reduces the memory footprint of large language models by converting high-precision floating-point weights (such as 16-bit BF16) into lower-bit formats like 1-bit or 2-bit representations. Mixture-of-Experts (MoE) models like Hy4 contain vast total parameter counts (770B) but only activate a fraction of parameters (49B) per token, making them prime candidates for low-bit quantization algorithms.