How to Build an AI Agent with LangGraph + Ollama (2026 Tutorial)

Setup: The Local Stack

# 1. Start Ollama and pull a tool-calling model
ollama pull qwen3:8b        # strong function calling
ollama serve

# 2. Install LangGraph
pip install langgraph langchain-ollama langchain

Model choice matters for tool calling — the sub-12B ranking and coding ranking both note function-calling strength. Qwen3-8B is the reliable default; Qwen3.5-4B for small hardware.

Step 1: Define the Tools

Tools are ordinary Python functions with type hints — LangGraph (via LangChain) turns them into the JSON schema the model calls.

from langchain_core.tools import tool

@tool
def search_docs(query: str) -> str:
    """Search the local knowledge base for documents about the query."""
    # In production: hit ChromaDB — see the 30-minute RAG pipeline
    return f"Top result for '{query}': the 2026 policy handbook, section 4."

@tool
def calculate(expression: str) -> str:
    """Evaluate a math expression safely."""
    # Whitelist only — never eval() raw user input
    import ast, operator
    ops = {ast.Add: operator.add, ast.Sub: operator.sub,
           ast.Mult: operator.mul, ast.Div: operator.truediv}
    node = ast.parse(expression, mode="eval").body
    def walk(n):
        if isinstance(n, ast.Constant): return n.value
        return ops[type(n.op)](walk(n.left), walk(n.right))
    return str(walk(node))

tools = [search_docs, calculate]

The docstrings are the model's instructions — write them as usage guides, not descriptions. See Tool Calling with Local LLMs for the full pattern guide.

Step 2: Build the Graph

from langgraph.graph import StateGraph, START, END
from langgraph.prebuilt import ToolNode, tools_condition
from typing import Annotated, TypedDict
from langgraph.graph.message import add_messages
from langchain_ollama import ChatOllama

class AgentState(TypedDict):
    messages: Annotated[list, add_messages]

llm = ChatOllama(model="qwen3:8b", temperature=0)
llm_with_tools = llm.bind_tools(tools)

def agent(state: AgentState):
    return {"messages": [llm_with_tools.invoke(state["messages"])]}

graph = StateGraph(AgentState)
graph.add_node("agent", agent)
graph.add_node("tools", ToolNode(tools))
graph.add_edge(START, "agent")
graph.add_conditional_edges("agent", tools_condition)
graph.add_edge("tools", "agent")          # loop back
graph.add_edge("agent", END)              # only reached when no tool call
app = graph.compile()

The shape is simple: agent calls the model; if the model requested tools, tools_condition routes to the tools node, which runs them and loops back to agent. The loop ends when the model answers without a tool call. That's the whole agent.

Step 3: Run It

from langchain_core.messages import HumanMessage

result = app.invoke({"messages": [
    HumanMessage("What does the handbook say about leave policy? "
                 "Also compute 17*43 for me.")
]})

for m in result["messages"]:
    print(f"{m.type}: {m.content}")

Watch the message list: you'll see the model call a tool, the tool result come back, and the model produce the final grounded answer. That's the decide–act–observe loop in action.

Step 4: Production Guards

Max iterations: app.invoke(input, config={"recursion_limit": 10}) — a confused agent must not loop forever.
Human-in-the-loop: LangGraph's interrupt_before pauses before risky nodes — mandatory for anything that sends emails or edits files.
Persistence: checkpointer=MemorySaver() gives the graph memory across turns and enables resuming after interrupts.
Streaming: app.stream(...) yields token-by-token updates for the UI.

from langgraph.checkpoint.memory import MemorySaver
from langgraph.types import interrupt

app = graph.compile(
    checkpointer=MemorySaver(),
    interrupt_before=["tools"],   # human approves tool calls
)
# First run with a thread_id; LangGraph pauses before tools
config = {"configurable": {"thread_id": "session-1"}}
app.invoke({"messages": [HumanMessage("Email the draft to the client")]}, config)

Leveling Up: From One Agent to a System

Give the agent retrieval. The search_docs tool above can call a ChromaDB query — that's agentic RAG, the pattern that lets agents decide when to search.
Expose it over MCP. Wrap your tools as Model Context Protocol servers so any MCP client (Claude Desktop, Cursor, or your own UI) can use them.
Split into specialist agents. When one agent juggles too many tools, it degrades — see Multi-Agent Systems for the coordinator pattern.
Watch the memory. Long agent runs grow the context; KV cache management and speculative decoding keep local agents fast.

🚀 Why local agents matter

An agent that calls tools on your machine never sends a prompt anywhere. For a privacy-first product — like Lawyer Assistant — that's the difference between an agent and a liability. The loop pattern in this tutorial is exactly the engine behind such assistants: local model + local tools + explicit graph control.

Frequently Asked Questions (FAQ)

What is LangGraph?

LangGraph is a Python (and JS) framework for building agent workflows as stateful graphs — nodes are steps, edges are transitions, and a shared state carries data between them. It adds cycles, persistence, and human-in-the-loop control that plain LangChain chains lack.

Can I build agents with Ollama and local models?

Yes — LangGraph's ChatOllama integration speaks to Ollama's OpenAI-compatible API, and tool calling works with models that support it (Qwen3, Qwen2.5, Llama 3.1+, Mistral). The whole agent runs locally with no cloud dependency.

What is a good local model for agent tool calling?

In 2026 the reliable picks are Qwen3-8B or Qwen3.5-9B (excellent function calling), Gemma 4 12B (multimodal agent work), and GLM-4-9B. For small hardware, Qwen3.5-4B still handles simple tool loops. Models without tool-calling support can use a text-based protocol, but it's less robust.

How is an agent different from a chatbot?

A chatbot responds. An agent acts: it can call tools, inspect results, decide next steps, and loop until the task is done. The loop — decide, act, observe, repeat — is the defining structure, and LangGraph makes that loop explicit and controllable.

Do agents need the cloud?

No — agent infrastructure (model, tools, graph runtime) all run locally. What agents often want is access to external services (APIs, databases, browsers), and that's a design choice, not a requirement. A fully private agent is very achievable in 2026.

What are the limits of local-model agents?

Long multi-step plans, complex reasoning chains, and large contexts are where small local models stumble compared to frontier models. Mitigations: keep steps small, add a reranker or memory, and set a max-iteration guard so a confused agent doesn't loop forever.