Running a Massive 566GB AI Model Locally on a Jetson AGX Thor

A developer successfully ran a massive 566GB FP8 AI model locally on a 128GB Jetson AGX Thor development kit. This extreme hardware constraint resulted in an excruciatingly slow processing speed of 0.646 tokens per second. This post highlights the extreme lengths local LLM enthusiasts go to push hardware boundaries by running frontier-scale models on edge devices. It serves as a humorous anecdote illustrating the massive gap between enterprise-grade model sizes and edge hardware memory limits. The MiMo-2.6-Pro model was read from disk mid-inference and painstakingly shunted through the 128GB memory of the Jetson AGX Thor. Every decoder layer and routed expert was traced step-by-step despite the bottlenecked token generation rate.