~/AI INFERENCE/tech-enthusiast-runs-2-8t-parameter-kimi-k3-ai-model-using-intel

Tech Enthusiast Runs 2.8T Parameter Kimi K3 AI Model Using Intel Optane

A tech enthusiast successfully ran a massive 2.8-trillion parameter Kimi K3 AI model on a CPU-only setup using discontinued Intel Optane persistent memory and llama.cpp. The experiment achieved an inference speed ranging from 0.75 to 1.5 tokens per second depending on the configuration and quantization level. This creative hardware hack demonstrates that massive large language models can be executed locally without expensive enterprise GPUs by utilizing high-capacity, low-cost discontinued server memory. It showcases practical methods for repurposing legacy hardware to handle modern artificial intelligence workloads in home laboratory settings. The setup utilized a 40-core 3rd Gen Intel Xeon Scalable processor alongside Intel Optane persistent memory 200 series operating in App Direct mode. Disabling direct access (DAX) to leverage DRAM caching improved the initial token generation speed from 0.85 to 1.50 tokens per second using a 1-bit quantization.

## BACKGROUND

Intel Optane persistent memory is a non-volatile memory technology based on 3D Xpoint that slots into standard server DIMM slots to bridge the gap between traditional volatile DRAM and high-performance SSDs. Meanwhile, llama.cpp is a popular open-source software library designed to perform efficient large language model inference on standard CPU hardware.

## REFERENCES

## KEYWORDS

#AI Inference#Hardware#Intel Optane#Large Language Models#llama.cpp

$ subscribe --daily