How to Build RAG with LangChain + Ollama Locally (2026 Tutorial)

Setup

# Ollama: pull the models once
ollama pull qwen3:8b          # generation
ollama pull bge-m3            # embeddings

# Python
pip install langchain langchain-community \
            langchain-chroma langchain-ollama \
            chromadb pypdf

Step 1: Load and Split

from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter

loader = PyPDFLoader("policy-handbook.pdf")
docs = loader.load()

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
    separators=["\n\n", "\n", ".", " ", ""],
)
chunks = splitter.split_documents(docs)
print(f"{len(chunks)} chunks")

Splitter choice matters enormously — the chunking guide covers when to upgrade to document-aware or contextual splitting.

Step 2: Embed and Store

from langchain_ollama import OllamaEmbeddings
from langchain_chroma import Chroma

embeddings = OllamaEmbeddings(model="bge-m3")

vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db",   # saved to disk
)
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

ChromaDB persists to a folder — re-run this and you load instead of re-embed. The ChromaDB vs FAISS guide explains the store choice.

Step 3: Retrieve and Answer

from langchain_ollama import ChatOllama
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough

llm = ChatOllama(model="qwen3:8b", temperature=0)

prompt = ChatPromptTemplate.from_template("""
Answer using ONLY the context below. If the context
doesn't contain the answer, say so. Cite the source
section when possible.

Context:
{context}

Question: {question}""")

def format_docs(docs):
    return "\n\n".join(f"[{d.metadata.get('source','?')}] {d.page_content}"
                       for d in docs)

rag_chain = (
    {"context": retriever | format_docs,
     "question": RunnablePassthrough()}
    | prompt
    | llm
)

answer = rag_chain.invoke(
    "What is the leave policy for part-time staff?")
print(answer.content)

That's the pipeline. Load → split → embed → retrieve → ground → answer — every stage local, every stage replaceable. The 30-minute plain-Python version shows the same logic without the framework, if you prefer minimal dependencies.

Leveling Up the Same Pipeline

Upgrade	How	Guide
Better chunking	Document-aware or contextual splitting	Chunking strategies
Hybrid search	Add BM25 alongside ChromaDB, merge results	Hybrid search
Reranking	Cross-encoder over the top-k candidates	Reranking in RAG
Evaluation	Recall@k + faithfulness on an eval set	RAG evaluation
Agents	Make retrieval a tool the model decides to call	Agentic RAG

Frequently Asked Questions (FAQ)

What is LangChain and why use it for RAG?

LangChain is a framework with ready-made components for every RAG stage — document loaders, text splitters, embedding wrappers, vector-store integrations, and chat models. You write the glue instead of the plumbing. For local RAG it works with Ollama on both the embedding and generation sides.

Can LangChain use Ollama for both embeddings and generation?

Yes — ChatOllama handles chat/generation and OllamaEmbeddings handles embeddings (e.g. BGE-M3 or nomic-embed-text). Both point at your local Ollama server, so the entire RAG pipeline runs offline with no API keys.

Which vector store should I use with LangChain?

ChromaDB is the default choice for local work (langchain-chroma, persistent to disk, zero server). FAISS is faster for pure in-memory search. Both have one-line LangChain integrations — the ChromaDB vs FAISS guide in this blog compares them.

Is LangChain still the right choice in 2026?

For beginners and teams wanting batteries included, yes. LangChain has real complexity overhead and its API shifts between versions — pin your version. If you want minimal dependencies, the 30-minute RAG guide in this blog shows the same pipeline in ~40 lines of plain Python.

What is the best embedding model for LangChain local RAG?

BGE-M3 (dense + sparse + multi-vector, strong multilingual) and nomic-embed-text are the common local picks via Ollama. See the embedding-models ranking in this blog for the full table with context lengths and licenses.

How do I improve a LangChain RAG pipeline?

In order of impact: better chunking (document-aware, contextual retrieval), hybrid search (BM25 + embeddings), reranking with a cross-encoder, and evaluating with a proper eval set. Each is covered by its own guide in this blog.