Local 52-Page Document Extraction Demo on iPhone 16 Using Bonsai 8B
A demonstration showcases the KernelAI mobile app performing local document extraction on a 52-page document using the Snowflake Arctic Embed model and PrismML's Bonsai 8B LLM on an iPhone 16. The update, KernelAI v2, also introduces features like web search and custom model imports. This demonstrates the viability of running complex Retrieval-Augmented Generation (RAG) pipelines locally on mobile hardware, enhancing user privacy and offline capabilities. Using highly optimized models like Bonsai 8B allows resource-constrained devices to run large language models efficiently. Bonsai 8B is a natively trained 1-bit LLM that fits into just 1.15 GB of storage, making it highly suitable for mobile deployment. The KernelAI app is free, ad-free, and also supports integration with Apple's built-in foundation models.
## BACKGROUND
Retrieval-Augmented Generation (RAG) is a technique that improves LLM responses by querying external documents for relevant information before generating text. Traditionally, RAG and LLM inference require powerful cloud servers due to high memory and compute demands. However, recent advances in model quantization, such as 1-bit architectures like PrismML's Bonsai, drastically reduce model sizes to fit on consumer mobile devices.