~/LARGE LANGUA/ovh-announces-olala-mini-7a1b-open-source-moe-model

OVH Announces Olala mini 7A1B Open-Source MoE Model

OVH has announced the open-source 'Olala mini 7A1B' language model, featuring a 7-billion-parameter Mixture-of-Experts architecture with 1 billion active parameters per layer. It combines hybrid attention like Differential-TPA and Mamba-3 and will be released under the Apache 2.0 license on Hugging Face. This model makes efficient local AI execution practical by achieving high speeds on modest consumer GPUs and standard CPUs without requiring high-end data center hardware. By lowering hardware barriers, it broadens accessibility for developers running large language models locally. Olala mini operates in BF16 format, requires at least 8 GB of VRAM, and supports a 32,000-token context window while hitting 160 tokens per second on an RTX A4500 and 50 tokens per second on a CPU.

## BACKGROUND

A Mixture of Experts (MoE) model divides an AI network into multiple expert sub-networks, routing each input to only a fraction of the parameters to improve processing speed and efficiency. Mamba is a sequence modeling architecture based on state space models designed to overcome traditional Transformer limitations like high memory usage during long sequences.

## REFERENCES

## KEYWORDS

#Large Language Models#Mixture of Experts#Open Source AI#Model Optimization

$ subscribe --daily