Local agents and tool use

A chat model that only talks is a smart parrot. Wire the same model up to call tools, so it can read a file, run a query, hit an API, and now it's an agent. Open models can pull this off in 2026. They just come with caveats the cloud-frontier crowd never has to think about.

This is post 11 of 13 in the Local LLMs series. By the end you'll know which open models to trust with tool use, why the loops break when they break, and how to ship a working local agent regardless.

Tool use in one paragraph

Here's the mechanism. Instead of just generating text, the model can emit a structured "function call": a name plus a JSON object of arguments. Your code parses that, runs the named function, grabs the result, and feeds it back to the model as another message. The model keeps going with the new information. You loop until it decides it's done.

It's a five-step dance:

1. User asks the model something that needs a tool.
2. Model emits a function call: {"name": "read_file", "arguments": {"path": "README.md"}}.
3. Your code runs the function, gets the file contents.
4. You feed the contents back to the model.
5. Model generates the actual answer using the file contents.

Cloud frontier models (Claude, GPT-5) do this nearly flawlessly. Open models do it too. They just have more ways to trip.

Which open models actually do tool use

The 2026 leaderboard for function calling on open weights:

• Qwen 2.5 Instruct (7B+). The most reliable open option. Trained heavily on tool use. JSON output is generally well-formed.
• Llama 3.1 / 3.2 Instruct (8B+). Tool use is supported, but flakier than Qwen. The 70B is solid. The 8B sometimes invents tool names.
• Hermes 3 (Llama-based fine-tunes). Tuned specifically for tool calling. Often beats vanilla Llama Instruct.
• Mistral Small / Medium 3. Solid function calling, especially for European languages.
• DeepSeek V4. Tool use is supported but less battle-tested in production agent loops.

What doesn't work well: 3B-class models for multi-step agentic loops. Guide them carefully and they can manage a single-shot tool call. They cannot reliably plan and execute a sequence of them. For agents, 7B is the practical floor. 14B and up is where it starts feeling reliable.

A minimal local agent

Ollama exposes function-calling through the OpenAI-compatible API. You define tools as JSON schemas and the model emits structured calls. Here's a working agent that can read files and run shell commands:

# local-agent.py

from openai import OpenAI
import subprocess
import json

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

tools = [
    {
        "type": "function",
        "function": {
            "name": "read_file",
            "description": "Read a file from disk",
            "parameters": {
                "type": "object",
                "properties": {"path": {"type": "string"}},
                "required": ["path"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "run_command",
            "description": "Run a shell command and return stdout",
            "parameters": {
                "type": "object",
                "properties": {"cmd": {"type": "string"}},
                "required": ["cmd"],
            },
        },
    },
]

def execute(name, args):
    if name == "read_file":
        return open(args["path"]).read()
    if name == "run_command":
        return subprocess.check_output(args["cmd"], shell=True, text=True)

def agent(user_prompt, max_steps=5):
    messages = [{"role": "user", "content": user_prompt}]
    for _ in range(max_steps):
        resp = client.chat.completions.create(
            model="qwen2.5:14b", messages=messages, tools=tools
        )
        msg = resp.choices[0].message
        messages.append(msg)
        if not msg.tool_calls:
            return msg.content
        for call in msg.tool_calls:
            args = json.loads(call.function.arguments)
            result = execute(call.function.name, args)
            messages.append({
                "role": "tool",
                "tool_call_id": call.id,
                "content": str(result),
            })
    return "Hit step limit."

print(agent("How many Python files are in the current directory?"))

Run it. With Qwen 2.5 14B, it should:

1. Decide it needs to run a shell command.
2. Call run_command with find . -name "*.py" | wc -l.
3. Receive the count.
4. Reply with the answer.

Three messages, one tool call, one short answer. That's the basic loop, start to finish.

Where local agents fail

An honest catalog of what goes wrong, roughly in order of how often you'll see it:

• Wrong tool name. The model invents a function that doesn't exist ("name": "list_files"). Common on smaller models. Defense: validate names against your tool list before executing, and return an error message that lists the actual tool names.
• Malformed JSON arguments. Trailing commas, unquoted strings, missing required fields. Common across all sizes. Defense: try to parse, and on failure feed the error back to the model with "your JSON was invalid, please retry."
• Loops without progress. The model calls the same tool with the same args twice. Or worse, three times. Defense: detect the repetition, then inject a message saying "you already tried this; pick a different approach."
• Premature stopping. The model gives up after one tool call when the task needs three. Defense: nothing automatic. Sometimes you just have to add explicit step planning to the prompt.
• Hallucinated results. The model claims a tool returned data when no tool ran at all. Defense: never trust the model's narration of tool output. Always show the user what tools actually ran.

None of this disqualifies local agents. It just means they need more defensive scaffolding than Claude or GPT-5 do.

Frameworks that make this easier

You don't have to write agent loops from scratch. The 2026 frameworks that play well with local models:

• smolagents (HuggingFace). Lightweight, built for any OpenAI-compatible backend. The closest thing to "just import and use."
• LangGraph. Heavier, with state machines and explicit graphs. Good for production agents where you need control-flow guarantees.
• CrewAI. Multi-agent framework. Each agent gets a role, tools, and goals. Works with Ollama.
• LlamaIndex. Started as a RAG framework, now has agents too. Pairs naturally with the embedding stack from the previous post.

All four can point at Ollama or LM Studio. Pick smolagents for first projects. Graduate to LangGraph when you need real graphs.

Tool design for local models

Cloud frontier models are forgiving about tool design. Local models are pickier. Rules that help:

• Few tools per agent. Three to six is the sweet spot for 14B-class models. Push past ten and the model starts mixing them up.
• Clear, narrow descriptions. "Read a file from the local filesystem given a path" beats "Get content." Smaller models lean hard on the description.
• Required fields, no optional ones. Optional fields confuse smaller models. Make everything required, with sensible defaults documented in the description.
• Idempotent where possible. The model will retry. When it does, the second attempt shouldn't break things.
• One job per tool. Don't build a manage_database tool. Build read_row, write_row, list_tables. Smaller models compose better than they branch.

These are the same principles that make tools easy for human users. Local models are basically junior developers, and clear interfaces help.

A realistic local agent stack

What I actually run locally:

• Model: Qwen 2.5 14B Instruct on a 24GB+ machine.
• Framework: smolagents for prototypes, hand-rolled loops for anything I ship.
• Tools: 4 to 6 max, each one-purpose, all required-args, all returning strings or numbers.
• Defensive scaffolding: tool-name validation, JSON parse retries, repetition detection.
• Fallback: if local fails the task three times, escalate to a cloud frontier model.

That escalation pattern is the honest part. Local agents handle 70% of agent tasks beautifully. The other 30% (deep planning, ambiguous instructions, novel tool combinations) still wants a frontier model. Don't fight that. Design for it.

What's next

Chat, RAG, agents, all running on your own hardware. The last big lever is changing the model itself: fine-tuning. Next post covers LoRA, QLoRA, when fine-tuning beats prompting and RAG (rarely), and how to do it on consumer hardware.