How to Build a RAG System in 30 Minutes (Local, Free)

Q: What is RAG and why should I build one?

RAG (Retrieval-Augmented Generation) grounds an LLM's answer in your own documents by retrieving the most relevant passages first and feeding them to the model as context. It fixes the two big problems with raw LLMs: outdated knowledge and hallucination. The model can only answer from what you retrieved, so answers are verifiable.

Q: Do I need a GPU to build a local RAG system?

No. A small 7B chat model (Q4) and an embedding model like nomic-embed-text run on CPU with 8-16GB of RAM. Everything in this tutorial is free and runs locally — no GPU, no cloud, no API keys.

Q: Which embedding model should I use with Ollama?

mxbai-embed-large is the best general-purpose default (1024-dim, top MTEB scores). nomic-embed-text is smaller (274MB) and handles long documents up to 8192 tokens. Qwen3 Embedding is the choice for Chinese-heavy content.

Q: How big should my chunks be?

500-1000 characters per chunk with a small overlap (100 characters) is the empirical sweet spot. Smaller chunks lose semantic completeness; larger chunks mix topics and make retrieval noisy. Test with your own documents.

Q: Can I use RAG with my own PDFs and documents?

Yes. Loaders like PyPDFLoader (for PDFs) and other LangChain document loaders turn files into text, which you chunk, embed, and store exactly like the example in this tutorial.

Q: Is ChromaDB good enough for production RAG?

For personal projects and up to ~100K documents, yes. For high-performance single-machine search, consider FAISS. For million-scale production deployments with high availability, use Milvus or a managed vector database.

What RAG Actually Does (in 30 Seconds)

A raw LLM answers from whatever it learned during training — so it goes stale, and it hallucinates when it doesn't know. RAG fixes both: before the model writes anything, you retrieve the most relevant passages from your own documents, and the model answers using only that context. If the information isn't in your documents, it should say so instead of inventing an answer.

This is the same architecture behind Lawyer Assistant, a fully local legal research app that grounds every answer in your documents and shows citations. What we build here is the same pipeline in miniature — and everything stays on your machine.

📄 Chunk

Split documents into overlapping pieces that each contain one complete idea.

🧮 Embed

Turn every chunk into a vector — a list of numbers capturing its meaning.

🔎 Retrieve

At query time, find the chunks whose meaning is closest to your question.

💬 Generate

Feed the retrieved chunks to a local LLM and get a grounded, verifiable answer.

What You Need

Python 3.8+ and pip
Ollama installed and running (how Ollama fits in the local-AI stack)
8–16GB of RAM — CPU is fine, no GPU required
A document to index (PDF, text, markdown — whatever you want to ask questions about)

Step 1: Install and Pull Models

Install the Python packages and download the two models (one for embeddings, one for answers):

pip install ollama chromadb langchain langchain-community pypdf

ollama pull mxbai-embed-large
ollama pull qwen2.5:7b

Why these models: mxbai-embed-large is the best general-purpose embedding model on Ollama — 1024-dimension vectors, top MTEB benchmark scores, and it runs on a laptop CPU. qwen2.5:7b is a strong, small chat model. Want a lighter setup? nomic-embed-text (274MB, 8192-token context) works for embeddings, and any chat model you already have works for answers.

Step 2: Load and Chunk Your Document

Load the file, then split it into chunks. Chunking is the step people skip — and it's the one that determines retrieval quality:

from langchain_community.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter

loader = PyPDFLoader("./your_document.pdf")
docs = loader.load()

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,   # characters per chunk
    chunk_overlap=100 # overlap keeps context across chunk boundaries
)
chunks = splitter.split_documents(docs)
print(f"Split into {len(chunks)} chunks")

💡 The sweet spot: 500–1000 characters per chunk. Too small → chunks get fragmented and can't form complete answers. Too large → one chunk mixes several topics and retrieval gets noisy. The 100-character overlap preserves context across boundaries.

Step 3: Embed and Store in ChromaDB

Turn every chunk into a vector and store it in ChromaDB, the zero-config vector database:

import chromadb
import ollama

client = chromadb.PersistentClient(path="./chroma_store")
collection = client.get_or_create_collection("my_docs")

for i, chunk in enumerate(chunks):
    response = ollama.embed(
        model="mxbai-embed-large",
        input=chunk.page_content,
    )
    embedding = response["embeddings"][0]

    collection.add(
        ids=[str(i)],
        embeddings=[embedding],
        documents=[chunk.page_content],
        metadatas=[{"source": chunk.metadata.get("source", "unknown")}],
    )

print(f"Indexed {len(chunks)} chunks")

PersistentClient saves the index to disk, so you only embed once — restart the script later and it's still there.

Step 4: Retrieve the Relevant Chunks

When a question arrives, embed it with the same model and ask ChromaDB for the nearest chunks:

query = "What are the requirements for X?"

query_embedding = ollama.embed(
    model="mxbai-embed-large",
    input=query,
)["embeddings"][0]

results = collection.query(
    query_embeddings=[query_embedding],
    n_results=3,  # top 3 most relevant chunks
)
print(results["documents"][0])

That's semantic search: it matches by meaning, not keywords. Ask "how do I cancel my subscription" and it finds the chunk about "ending your plan" even if the words never match. (For the full picture of how this works and how hybrid search improves on it, see the Lawyer Assistant RAG deep dive.)

Step 5: Generate a Grounded Answer

Feed the retrieved chunks to the chat model as context — and instruct it to answer only from that context:

context = "\n\n".join(results["documents"][0])

response = ollama.chat(
    model="qwen2.5:7b",
    messages=[
        {
            "role": "system",
            "content": (
                "Answer based only on the provided document content. "
                "If the documents don't contain the answer, say so honestly."
            ),
        },
        {
            "role": "user",
            "content": f"Document content:\n{context}\n\nQuestion: {query}",
        },
    ],
)

print(response["message"]["content"])

That's it — you now have a working, fully local RAG system. The honesty instruction in the system prompt is what separates RAG answers from hallucinated ones: the model is told to admit when your documents don't cover the question.

Making It Better (10-Minute Upgrades)

Batch embeddings for speed. Calling the API one chunk at a time is slow on big documents. Pass a list: ollama.embed(model="mxbai-embed-large", input=batch_texts)["embeddings"].
Filter by distance. ChromaDB returns the top-N chunks regardless of relevance. Keep only results with distance below your threshold (e.g. < 0.3) so unrelated questions don't get answered from noise.
More chunking strategies. Paragraph-based splitting works well for structured docs; smaller chunks (500 chars) suit fragmented content like chat logs. Test on your own documents.
Add reranking. Retrieve 10 chunks, then rerank them before taking the top 3 — this measurably improves answer quality and is a core part of the production Lawyer Assistant pipeline.
Scale the vector store. ChromaDB is perfect up to ~100K chunks. For high-performance single-machine search use FAISS; for million-scale production, use Milvus or a managed vector database.

Frequently Asked Questions (FAQ)

What is RAG and why should I build one?