Agentic / native-tool-use models

Why this matters

In 2024-2026, frontier labs stopped treating "tool use" and "agent loops" as application-layer scaffolding. Instead, they trained models natively for those skills. Anthropic released Computer Use Claude. OpenAI shipped GPT-4o function calling and o1 with tools. Anthropic and others released agentic benchmarks (TAU-bench, SWE-bench, WebArena). Open models (Llama 3.x function-calling fine-tunes, Qwen2.5-Coder-Agentic, Hermes-3, NousResearch's Hermes-Tools) followed.

This lesson is the bridge between what a base LLM can do and what an agent system needs — explained from the model's perspective. It complements the agentic course, which is about building with these capabilities.

Learning objectives

1. Understand what "function calling" trained into a base model means.
2. Describe the Computer Use paradigm.
3. Recognise SWE-bench-style coding agents.
4. Explain how reasoning RL (Phase 4.5) feeds agentic capability.
5. Pick the right agent-capable model for a given task.

1. Native function calling

A base LLM can be prompted to emit structured JSON for tool calls. A natively trained model is fine-tuned with millions of (task, tool-call, result) examples so that it:

• Learns the exact tool-call format used by an API.
• Learns when to call vs. answer directly.
• Learns to handle errors, retries, and multi-step plans.
• Tightens the parameter generation (no hallucinated tool names, no malformed JSON).

The training data is mostly synthetic: a stronger model generates trajectories, a verifier filters successful ones, and the target model trains on those.

Models you should know:

For a local agent, Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct give you a strong tool-using base.

2. Tool-use SFT data shape

A modern training example for tool use looks like:

{
  "messages": [
    {"role": "system", "content": "You are an assistant with tools..."},
    {"role": "user", "content": "What's the weather in Bengaluru?"},
    {"role": "assistant",
     "content": null,
     "tool_calls": [{"id": "1", "function": {
        "name": "get_weather",
        "arguments": "{\"city\": \"Bengaluru\"}"}}]},
    {"role": "tool", "tool_call_id": "1",
     "content": "{\"temp_c\": 26, \"condition\": \"clear\"}"},
    {"role": "assistant", "content": "It's 26 °C and clear in Bengaluru."}
  ],
  "tools": [{"type":"function", "function":{
      "name":"get_weather", "parameters":{...}}}]
}

The model learns the chat-template special tokens that demarcate tool calls vs assistant text vs tool responses. This is why tool reliability differs sharply between a base model and a tool-tuned model on identical prompts.

Public datasets: Salesforce/xlam-function-calling-60k, glaiveai/glaive-function-calling-v2, NousResearch/hermes-function-calling-v1.

You can SFT a base model (Lesson 4.1) on these and get a respectable tool-using model in a few GPU-hours.

3. Computer Use (Anthropic, 2024-2025)

Computer Use trains the model to:

• View screenshots (multimodal input).
• Output actions: click(x, y), type("text"), scroll(direction), wait(), etc.
• Handle dynamic UIs (waiting for elements, error popups).

Effectively, the model is an agent that can drive a virtual computer: opening apps, filling forms, navigating browsers. Claude 3.5 Sonnet → Claude 4 added increasingly reliable Computer Use.

OpenAI's "Operator" (early 2025) is OpenAI's analogous product on top of GPT-4o + a custom browser stack.

Open-source attempts:

• xLAM (Salesforce) — open tool-calling models.
• Browser Use library — pairs an LLM with a browser environment; works with any tool-using LLM.
• OpenInterpreter — local computer-use agent.

The mechanism is plain function-calling at heart, with the tools being screenshot, click, type, etc., and the model trained on long trajectories of GUI interaction.

4. SWE-bench coding agents

SWE-bench (Jimenez et al., 2024) gives the model a real GitHub issue and asks it to produce a patch that passes the project's tests. Solving it requires:

• Reading thousands of files.
• Running tests in a sandbox.
• Iterating on errors.

Frontier scores in 2024 were ~20%; by late 2025, top systems exceed 65% on SWE-bench Verified. Best systems combine:

• A strong coding-trained base (GPT-4o, Claude 4, DeepSeek-Coder-V3, Qwen2.5-Coder-32B).
• An agent scaffolding (Aider, OpenHands, SWE-agent, OpenDevin, Devin, Cursor's own).
• Tools: read_file, edit_file, run_tests, grep, git.
• Often a verifier + best-of-N.

For learning: study Aider (open) and OpenHands (open). Use them as references for production agent loops.

5. Reasoning + tools — the strongest pattern

In 2025+, the leaders in agentic benchmarks are reasoning models with tool access (Claude with extended thinking + tools, o3 + tools, DeepSeek-R1 + tools).

Why?

• Reasoning models naturally plan (long CoT before acting).
• They self-correct when a tool fails.
• They balance "think more" vs "call a tool" decisions.

Practical implication: when latency permits, use a reasoning model behind your agent. The cost is higher, but the success rate on multi-step tasks can double.

6. Agentic benchmarks

Beyond SWE-bench:

• TAU-bench — customer-service style multi-turn tool use.
• WebArena / WebShop — realistic web tasks.
• MetaGPT / GAIA — multi-skill agent tasks.
• OS-World / SWE-Lancer — computer-use tasks.
• AgentBench — broad agent evaluation suite.

When fine-tuning your own agent model, build a small "in-domain" benchmark. Public benchmarks generalise weakly to specific products.

7. The minimal architecture for an agent

loop until done or max_steps:
    1. Build a prompt: system + tools manifest + history.
    2. Call the LLM.
    3. If response is a tool call: execute the tool, append result.
    4. Else: return the final answer.

The model does the heavy lifting; the scaffold is small. We covered scaffolding implementations in course/04_multi_agent/01-06_*. This lesson is the LLM-internals view.

8. When to fine-tune for your own agent

Fine-tune (typically LoRA SFT + a touch of DPO) when:

• Tool surface is specific (e.g., your internal APIs).
• Open tool-calling models keep producing malformed args.
• You want a smaller / cheaper model than GPT-4o.
• You need on-prem deployment.

Skip fine-tuning when:

• Out-of-the-box function calling already works.
• Use case changes weekly (the fine-tune becomes stale).

The 2026 sweet spot: open ~7-30B model, LoRA fine-tuned on a few thousand domain-specific tool trajectories, served via vLLM.

Hands-on lab (4 hours)

agent_model_lab.ipynb:

1. Use Qwen2.5-7B-Instruct with transformers's tools API. Define a get_weather tool. Have it call the tool and respond.
2. Compare with Llama-3.1-8B-Instruct on the same prompt. Note format differences.
3. Run a small SFT on glaiveai/glaive-function-calling-v2 (5k examples) to teach a smaller Llama-3.2-1B proper tool calling. Measure improvement.
4. Build a small "SWE-bench-lite": create 5 toy GitHub-style issues against a tiny Python repo, ask the model to produce patches, run pytest. Track success rate.
5. Try Anthropic's Computer Use API on a simple browsing task (e.g., "search Wikipedia for X and return the first paragraph").
6. Bonus: instrument a simple ReAct agent with langgraph (from course/03_single_agent/); use your fine-tuned model for tool calls.

Common pitfalls

1. Calling a tool-tuned model with the wrong template → it still answers, but worse. Always use the model's apply_chat_template with tools=.
2. Defining 50 tools → models degrade with too many tools; aim for ≤10 per task; use sub-agents.
3. Skipping recovery — when a tool errors, the model often loops; build "you must rethink" prompts.
4. Long conversation context blowing tool quality — every model degrades after ~30+ turns of tools. Summarise periodically.
5. Trusting fine-tune trajectories without rule-based filtering — synthetic tool data is noisy.

Self-check

1. What does it mean for a model to be "natively trained for function calling"?
2. Why do reasoning models pair well with tools?
3. What does Computer Use add over plain tool calling?
4. When should you fine-tune your own tool-using model?
5. What benchmark measures real-world software engineering ability?

References

• Anthropic (2024-2025), "Computer Use" / Claude tool use docs.
• Jimenez et al. (2024), "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?"
• Yang et al. (2024), "SWE-Agent."
• Salesforce (2024), "xLAM: Large Action Models" + xlam datasets.
• NousResearch (2024), "Hermes 3 Technical Report."
• Yao et al. (2022), "ReAct: Synergizing Reasoning and Acting in Language Models."
• Patil et al. (2023), "Gorilla: Large Language Model Connected with Massive APIs."
• TAU-bench, WebArena, GAIA papers.