Local RAG on 8GB RAM: The Complete Guide

The 8GB Memory Budget

That leaves ~4GB for the operating system and applications — tight but workable on a modern 8GB machine. The two levers if you need more room: drop context to 2K tokens, or use a 1–2B model. The math follows the rule from Q4_K_M vs Q8_0: a 4B model at Q4 ≈ 0.56 bytes/param ≈ 2.5GB, plus context and overhead.

The 8GB Stack

• Chat model: phi4-mini — 3.8B, MIT license, 128K context, 74.4% HumanEval. The best quality-per-gigabyte in its class (see the sub-12B ranking).
• Embedding model: nomic-embed-text — 137M params, 274MB, 8192-token context. Small enough to live comfortably beside the LLM.
• Vector store: ChromaDB — runs embedded in your Python process, so there's no separate server eating RAM. (Or FAISS — see the comparison.)
• Runtime: Ollama serves both models via one local API — the same stack as the 30-minute RAG tutorial.

The Working Pipeline

Pull the models, then run the pipeline. This is the same code as the 30-minute tutorial, tuned for low memory:

# Install once
pip install chromadb ollama

# Pull the two models
ollama pull phi4-mini
ollama pull nomic-embed-text

import chromadb
import ollama

client = chromadb.PersistentClient(path="./rag_db")
collection = client.get_or_create_collection("docs")

def index_documents(docs):
    for i, doc in enumerate(docs):
        embedding = ollama.embed(model="nomic-embed-text", input=doc)["embeddings"][0]
        collection.add(ids=[f"doc-{i}"], embeddings=[embedding], documents=[doc])

def ask(question):
    q_emb = ollama.embed(model="nomic-embed-text", input=question)["embeddings"][0]
    results = collection.query(query_embeddings=[q_emb], n_results=3)
    context = "\n\n".join(results["documents"][0])
    response = ollama.chat(model="phi4-mini", messages=[
        {"role": "system", "content": "Answer using only the context."},
        {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}
    ])
    return response["message"]["content"]

For keyword-heavy queries, add hybrid search; for precision, add a reranker — but on 8GB, start simple. The architecture here is exactly what Lawyer Assistant uses at full scale: local models, local storage, zero cloud.

Memory-Saving Tips

• Chunk at 300–500 chars — smaller chunks mean smaller context, which directly cuts KV-cache RAM.
• Keep context to 2–4K tokens — the KV cache is the hidden memory cost at long context.
• Close the browser — Chrome alone can eat 2–3GB. Run the pipeline in a focused terminal session.
• Use Q4_K_M, not Q8 — half the weight memory for ~1–2% quality loss (the numbers are in our quant guide).
• Batch embeddings — embed documents in one call (input=list) instead of one at a time.
• Persist ChromaDB to disk — PersistentClient keeps RAM free between runs.

When 8GB Isn't Enough

If you're processing very long documents, running many concurrent queries, or need frontier-level reasoning, 8GB will feel cramped. The honest upgrade paths, in order: 16GB RAM (unlocks 8B models and long context — the single best upgrade), a used GPU with 12–16GB VRAM (24GB cards like the RTX 3090 are the community favorite), or Apple Silicon with unified memory, where the whole RAM pool is available to the model. The full hardware tier map is in the local-model ranking.

Frequently Asked Questions (FAQ)

Can I run RAG on 8GB of RAM?

Yes. An 8GB machine can run a complete local RAG pipeline if you budget carefully: a small embedding model (~300–500MB), a 3–4B chat model at Q4 (~2–3GB), and ChromaDB in-process. The 30-minute RAG tutorial on this blog works on 8GB with phi4-mini or qwen3:4b.

What LLM should I use for RAG on 8GB RAM?

Phi-4-mini (3.8B, MIT) is the best balance — 74.4% HumanEval, 128K context, ~2.5GB at Q4. Qwen3-4B and Gemma 3 4B are solid alternatives. Avoid 8B+ models, which eat 5GB+ and leave nothing for context and the OS.

Which embedding model fits 8GB RAM?

nomic-embed-text (137M, 274MB, 8192-token context) is the ideal fit, with mxbai-embed-large (335M, ~650MB) as a higher-quality option. Both run via Ollama and are CPU-friendly.

How much RAM does the context window use?

Context memory grows with the KV cache — roughly 1–2GB extra for a 3–4B model at long context. Keep context to 2–4K tokens for a typical Q&A on 8GB, or use a smaller chunk size to stay comfortable.

Will RAG on 8GB be fast?

CPU-only generation is a few tokens/sec on a 3–4B model — usable for Q&A, not for real-time chat. Embedding and retrieval are fast (ms). The bottleneck is generation, so batch questions and accept a 10–30 second answer time per query.

What can I do if I run out of memory?

Drop to Q4_K_M (or Q4_0) instead of Q8, switch to nomic-embed-text, reduce context to 2048, close browsers, or use a 1–2B model for generation. See our Q4_K_M vs Q8_0 guide for the size math.

Sources

• ChromaDB — official docs
• Phi-4-mini — Ollama library
• nomic-embed-text — Ollama library
• Local AI Zone — GGUF model directory
• GGUF Loader — zero-CLI local inference