Developer Successfully Converts Qwen3-0.6B Model Into Mixture of Experts Architecture

A developer successfully converted the Qwen3-0.6B chat model into a Mixture of Experts (MoE) architecture while maintaining GGUF compatibility. To overcome the lack of a true shared expert in the Qwen3 architecture, the developer implemented a pseudo-shared expert to help recover lost quality. This project showcases creative, grassroots experimentation in modifying LLM architectures at a very small scale. It highlights the technical hurdles of converting dense models into MoE configurations while keeping them compatible with popular local inference formats like GGUF. Benchmarks show that at a small 0.6B scale, the MoE conversion yields no speed improvement over the dense version and performs worse across standard quality metrics. The developer notes that the converted model gets specific facts wrong and advises using non-thinking mode since thinking capabilities were never retrained.