~/LARGE LANGUA/clover-2-a-27-8b-model-using-dynamic-lora-expert-routing

Clover 2: A 27.8B Model Using Dynamic LoRA Expert Routing

Clover 2 is a new 27.8B open-source model that uses a trained router to dynamically select between seven specialized LoRA adapters on a per-token basis during inference. Instead of statically blending weights, the model actively decides which expert to use at every single token. This approach offers a practical, open-source alternative to traditional static model merging by bringing true per-token Mixture-of-Experts routing to PEFT adapters. It achieves measurable benchmark improvements in math and long-context tasks while reducing generic AI-writing habits. The release includes the base model, all seven PEFT-format adapters, the trained router, and a vLLM plugin to execute the per-token routing behavior. Currently, self-hosting requires vLLM because llama.cpp support for the hybrid attention architecture is still bleeding-edge and GGUF files are not yet available.

## BACKGROUND

Mixture of Experts (MoE) is a machine learning technique that uses multiple specialized sub-models to divide a problem space, enabling efficient scaling. LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method that adapts pre-trained models by modifying a small subset of parameters instead of full retraining.

## REFERENCES

## KEYWORDS

#Large Language Models#Mixture of Experts#LoRA#Open Source AI#Inference Optimization

$ subscribe --daily

Clover 2: A 27.8B Model Using Dynamic LoRA Expert Routing | Daily News