Running Local LLMs and NLP Tools on 8GB VRAM Hardware
A community inquiry on r/LocalLLaMA sought recommendations for running small language models, text embeddings, and rerankers on hardware constrained to 8GB VRAM, such as an RTX 2050. The user requested lightweight, tool-capable, and multilingual models suitable for everyday office automation. While frontier LLMs continue to expand in parameter count, maintaining efficient small-scale models (1B to 8B parameters) is crucial for accessible local deployment on consumer hardware. It highlights the ongoing demand for privacy-conscious, localized AI workflows like RAG and function calling on budget devices. Executing tasks sequentially rather than loading models concurrently allows an 8GB VRAM system to handle quantized 3B to 7B LLMs alongside lightweight embedding and reranking models. Applying 4-bit or 8-bit quantization reduces memory consumption significantly, making tool-use capabilities feasible even on entry-level GPUs.
## BACKGROUND
Running AI models locally requires GPU Video RAM (VRAM) to hold model weights and process contexts during inference. Quantization is a compression method that converts high-precision model weights (like FP16) to lower precision (like INT4 or INT8) to drastically reduce memory usage. In Retrieval-Augmented Generation (RAG) pipelines, embedding models convert text into vector spaces while rerankers re-evaluate and refine search results to provide higher accuracy before context is passed to the LLM.