Tool Calling with Local LLMs: A Practical Guide (2026)

How Tool Calling Works

1. You declare tools. Each tool is a name, a description, and a JSON Schema for its arguments.
2. The model decides. When a tool would help, the model returns a tool_calls message instead of a plain answer — e.g. search_kb(query="leave policy", limit=3).
3. You execute. Your code dispatches to the real function, with validation.
4. You feed back the result. The tool output is appended as a tool message, and the model answers (or calls again).

The key mental model: the model proposes; your code disposes. Everything that actually happens — file writes, API calls, DB queries — runs through code you control. That's the safety property that makes local tool calling safe.

Which Local Models Support It

Full rankings with downloads: sub-12B by power and sub-12B for coding.

The Minimal Ollama Tool Loop

import json, requests

# 1. Declare tools (JSON Schema)
tools = [{
    "type": "function",
    "function": {
        "name": "search_kb",
        "description": "Search the internal knowledge base for documents about a topic.",
        "parameters": {
            "type": "object",
            "properties": {
                "query": {"type": "string", "description": "Search phrase"},
                "limit": {"type": "integer", "minimum": 1, "maximum": 10}
            },
            "required": ["query"]
        }
    }
}]

# 2. The real function behind the tool
def search_kb(query: str, limit: int = 3) -> str:
    # In production: query ChromaDB (see the 30-min RAG guide)
    return f"[KB] Top {limit} results for '{query}'"

messages = [{"role": "user", "content": "What's our leave policy?"}]

for _ in range(5):  # iteration guard
    r = requests.post("http://localhost:11434/api/chat", json={
        "model": "qwen3:8b",
        "messages": messages,
        "tools": tools,
    })
    msg = r.json()["message"]
    messages.append({"role": "assistant", "content": msg.get("content") or ""})

    if msg.get("tool_calls"):
        for call in msg["tool_calls"]:
            fn = call["function"]
            if fn["name"] == "search_kb":
                result = search_kb(**fn["arguments"])
                messages.append({"role": "tool",
                                 "name": fn["name"],
                                 "content": result})
        continue

    print(msg["content"])   # final grounded answer
    break

That's the entire loop: declare tools → send → execute → feed back → repeat. requests and the Ollama API — no framework required. For graph control (loops, interrupts, persistence), layer LangGraph on top — see Build an AI Agent with LangGraph + Ollama.

Failure Modes and Fixes

The Safety Boundary

• Validate arguments against the schema again in code — the model's JSON is a suggestion, not a contract.
• Whitelist operations — a "run_shell" tool should reject anything not on the allowlist.
• Require confirmation for destructive or expensive tools (email, writes, payments).
• Log every call — tool logs are your audit trail (GDPR-relevant and AI-Act-relevant).
• Local means contained — with a local model and local tools, no tool call ever touches the network. See the offline workspace guide.

Where Tool Calling Takes You

• Agents: the loop above, with a stop condition — LangGraph + Ollama agent tutorial.
• MCP: expose your tools as MCP servers and every client can use them.
• RAG: make retrieval a tool so the model decides when to search — agentic RAG.
• Multi-agent: route between specialist agents — when one agent isn't enough.

Frequently Asked Questions (FAQ)

What is tool calling (function calling)?

Tool calling is when an LLM, instead of answering directly, emits a structured request to call a function you defined — with arguments in JSON. Your code runs the function and returns the result, and the model continues from there. It's the mechanism behind agents, RAG lookups, and API integrations.

Which local models support tool calling well?

The reliable 2026 picks: Qwen3 and Qwen3.5 (excellent), GLM-4-9B, Gemma 4, and Llama 3.1+ derivatives. Check each model card for tool-calling support — some small models only handle single-call, not multi-call parallel invocations.

How do I implement tool calling with Ollama?

Ollama's API accepts a tools array of JSON schemas alongside the model name. The response contains a message with tool_calls, which your code executes and appends as tool messages before calling again. The example in this post is a complete minimal loop.

Why does my local model call the wrong tool or wrong arguments?

Usually one of: tool descriptions too vague (the model doesn't know when to use them), too many tools in one call (models degrade past ~10), arguments colliding with context, or a model without proper function-calling fine-tuning. Fix the schema and descriptions before blaming the model.

Do I need a special model for tool calling?

Native tool calling needs a model trained for it — a JSON output in a specific format. Models without it can be coerced with prompt-based protocols (asking for JSON output), which is more fragile. Prefer natively supported models; they're common in 2026.

Is tool calling safe with local models?

The model is a request generator — your code decides what actually runs. That's the safety boundary: validate arguments, whitelist operations, require confirmation for destructive tools, and log every call. Local execution also means no data leaves the machine for any tool call.

Sources & Further Reading

• How to Build an AI Agent with LangGraph + Ollama
• MCP Explained: Model Context Protocol for Local AI
• Agentic RAG: Combining Agents with Retrieval
• How to Build a RAG System in 30 Minutes (Local, Free)
• Top 10 AI Models Under 12B Parameters
• Lawyer Assistant: privacy-first legal RAG
• Ollama OpenAI-compatible API docs (tools)