Agentic RAG: Combining Agents with Retrieval

Classic RAG answers a question in one pass: embed the query, search, stuff the top chunks into the prompt, generate. It is fast, simple, and it solves the majority of "ask my documents" problems. But it breaks down the moment a question is multi-hop — "what did the Q3 report say about churn, and how does that compare to the marketing deck?" — because no single retrieval answers that.

Agentic RAG hands the pipeline to an agent. The agent decides what to look for, calls retrieval as a tool, reads what comes back, decides it needs more, searches again, and only then writes the final answer. This guide explains how the loop works, the patterns that actually matter in 2026, and how to build it with local models.

What Agentic RAG Actually Is

Think of classic RAG as a vending machine: you press a button, it drops a result. Agentic RAG is a researcher: it takes notes, follows threads, asks follow-up questions of the data, and comes back when it is confident.

Concretely, the agent loop wraps the same retrieval building blocks you already know — embeddings, vector search, BM25, rerankers — behind tool calls. The LLM decides which tool to call (search, lookup, web, calculator, code), with what arguments, and inspects the results before deciding the next step. The loop terminates when the model emits a final answer.

The Four Patterns That Matter

The patterns compose. A strong agentic RAG setup uses query rewriting and multi-hop and a reflection step — which is why it costs 3–10× more tokens than a single-shot pipeline.

Agentic RAG vs Classic RAG

A Local Agentic RAG Loop (LangGraph + Ollama)

Here is the minimal loop — agent → retrieval tool → reflection — running entirely on a local stack. It assumes Ollama is serving a tool-calling model like qwen3:8b and ChromaDB holds your chunks (the same setup as our local RAG on 8GB guide).

from langgraph.graph import StateGraph, END
from langchain_ollama import ChatOllama
from langchain_core.tools import tool
from langchain_community.vectorstores import Chroma

llm = ChatOllama(model="qwen3:8b", temperature=0)
store = Chroma(persist_directory="./chroma_db")

@tool
def search_docs(query: str) -> str:
    """Search the document store and return the top 4 chunks."""
    results = store.similarity_search(query, k=4)
    return "\n\n".join(f"[{i+1}] {r.page_content}" for i, r in enumerate(results))

tools = [search_docs]
model = llm.bind_tools(tools)

def agent(state):
    response = model.invoke(state["messages"])
    return {"messages": [response]}

def should_continue(state):
    last = state["messages"][-1]
    return "tool_calls" if last.tool_calls else END

def tool_node(state):
    out = []
    for call in state["messages"][-1].tool_calls:
        if call["name"] == "search_docs":
            out.append(search_docs.invoke(call["args"]))
    return {"messages": out}

graph = StateGraph(dict)
graph.add_node("agent", agent)
graph.add_node("tools", tool_node)
graph.set_entry_point("agent")
graph.add_conditional_edges("agent", should_continue, {"tool_calls": "tools", END: END})
graph.add_edge("tools", "agent")

app = graph.compile()
result = app.invoke({"messages": [("user", "How does Q3 churn compare to the marketing claims?")]})
print(result["messages"][-1].content)

The loop is deliberately short: the agent searches, sees the chunks, and either searches again or answers. Add a reflection node — "check that every claim in your draft is supported by a retrieved chunk, re-search if not" — and you have the full pattern. LangGraph's state graph makes the control flow explicit, which is why it beats ad-hoc while loops for anything beyond a demo.

When to Add Each Piece (Decision Guide)

• Query rewriting — add first. Users type badly; one rewritten search usually fixes 80% of retrieval misses. Cheapest win in the whole list.
• Multi-hop — add when your questions span documents ("compare", "across", "from X to Y").
• Reflection — add when wrong answers are expensive (legal, medical, contracts). It roughly doubles latency; budget for it.
• Extra tools — add only when questions mix docs with computation (aggregate a table, run a formula). Every extra tool is a new failure mode.
• Step limits — always. A 4–6 step cap turns a runaway loop into a graceful "I could not find enough evidence" answer.

Does It Need a Big Model?

No — and this is where local AI shines. Tool calling and multi-step reasoning are now solid on 7–14B models; Qwen3.5-9B and Gemma 4 handle agent loops reliably at Q4 on a 16GB machine. Because there is no per-token API bill, the 3–10× token overhead of an agent loop is just a few extra seconds of compute — which makes agentic RAG far more attractive locally than on metered APIs.

How to Evaluate It

Classic RAG metrics (recall@k, NDCG, faithfulness) still apply, but agentic RAG adds loop-level metrics you should track:

• Convergence rate — % of questions answered before the step limit (target: >95%)
• Tool-call success — % of search calls that returned useful (non-empty, on-topic) results
• Steps per question — if this creeps up, retrieval quality is slipping and the agent compensates by searching more
• End-to-end answer accuracy — the only number that ultimately matters, on a labeled test set (build one from 50–100 real questions)

Start small: measure classic RAG on your test set first, then add pieces one at a time and re-measure. If query rewriting alone closes the gap, stop there — every added pattern is added complexity.

Where It Fits in the Local AI Stack

Agentic RAG is the next rung on the ladder this blog has been building: embeddings → vector search → hybrid search → reranking → agents → agentic RAG. The retrieval fundamentals from the embedding models and vector databases posts are exactly what the agent calls as tools — nothing changes at the storage layer, only the control flow above it.

For a privacy-first use case, agentic RAG is the reason Lawyer Assistant can answer "does this clause contradict the other agreement?" — a genuinely multi-hop question that a single-pass pipeline would miss, all without a single document leaving the machine.

Frequently Asked Questions

What is agentic RAG?

Agentic RAG gives the retrieval pipeline to an AI agent: instead of one embed-and-search step, the agent decides what to search, calls retrieval as a tool, reads results, searches again, and iterates until it can answer.

How is agentic RAG different from classic RAG?

Classic RAG is a fixed single-pass pipeline. Agentic RAG is a loop with query rewriting, multiple retrievals, tool calls, and self-correction before the final answer.

Does agentic RAG need a big model?

No — 7–14B local models like Qwen3 and Gemma 4 handle tool calling and agent loops well, and local inference makes the extra tokens essentially free.

Can I run agentic RAG locally?

Yes. LangGraph, LlamaIndex, and Haystack all run against Ollama or llama.cpp with ChromaDB or FAISS. A full stack fits on 16GB of RAM with a Q4 7–14B model.

When does agentic RAG beat classic RAG?

For multi-hop, ambiguous, or cross-document questions. For simple fact lookups, classic RAG is faster, cheaper, and more predictable.

How do you evaluate agentic RAG?

Track convergence rate, tool-call success, steps per question, and end-to-end answer accuracy on a labeled test set — in addition to classic retrieval and faithfulness metrics.