AskAiml
Sign in Get started

Reliability, tools, and observability for agents

Retries, timeouts, tool error handling, LangSmith tracing, and the runbook every production agent needs.

🤖 Module 3 7 min read Not started

Why this matters

Demo agents work 70% of the time. Production agents need to work 99%+ of the time, fail safely the other 1%, and let you debug what happened. The patterns in this lesson — circuit breakers, idempotency, structured logs, tracing — are the boring expensive skills that make you unfireable.

Learning objectives

  1. Design tools that are safe to retry.
  2. Add retries, timeouts, circuit breakers, rate limits.
  3. Detect and break infinite agent loops.
  4. Trace every run with LangSmith (or LangFuse).
  5. Emit OpenTelemetry spans for vendor-neutral observability.

1. Tool design rules (the senior version)

The contract you give an LLM must be enforceable. Bake these into every tool you write:

python
from pydantic import BaseModel, Field
from langchain_core.tools import tool
from typing import Literal

class CreateRefundArgs(BaseModel):
    invoice_id: str = Field(..., pattern=r"^INV-[0-9]{4}-[0-9]{4,6}$")
    amount_inr: float = Field(..., gt=0, le=100_000)
    reason: Literal["duplicate", "fraud", "service-failure", "goodwill"]
    confirm: bool = Field(..., description="MUST be true to actually run")

@tool
def create_refund(args: CreateRefundArgs) -> dict:
    """Refund a customer. Requires confirm=True. Use only after explicit user approval."""
    if not args.confirm:
        return {"status": "preview", "message": "Set confirm=True to actually create."}
    # actual call to billing API with idempotency key
    return billing.refund(invoice_id=args.invoice_id, amount=args.amount_inr,
                          reason=args.reason,
                          idempotency_key=f"agent-{args.invoice_id}")

What this gives you:

  • Strict types and regex prevent garbage IDs from the LLM.
  • confirm flag is a free "are you sure?" gate.
  • Idempotency key makes retries safe.
  • Bounded amount caps blast radius if the model goes wild.

A model that calls create_refund(invoice_id="INV-2026-7842", amount_inr=999_999, confirm=True, reason="fraud") is rejected at the schema level before reaching billing.

2. Retries, timeouts, and circuit breakers

python
# uv add tenacity httpx
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
import httpx

@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential(min=1, max=8),
    retry=retry_if_exception_type((httpx.TimeoutException, httpx.HTTPStatusError)),
    reraise=True,
)
def call_billing(payload: dict) -> dict:
    with httpx.Client(timeout=httpx.Timeout(10.0, connect=3.0)) as c:
        r = c.post("https://billing/api/refund", json=payload)
        r.raise_for_status()
        return r.json()

Timeouts: every external call. Always. Especially LLM calls and tool HTTP requests.

Circuit breaker: if a downstream API fails repeatedly, stop calling it for a window.

python
# uv add purgatory  # or pybreaker
from purgatory import SyncCircuitBreakerFactory

cb = SyncCircuitBreakerFactory(
    default_threshold=5,            # failures to open
    default_ttl=30,                 # cool-down seconds
)

with cb.get_breaker("billing"):
    return call_billing(payload)

Circuit-breaker states:

  • Closed — calls go through.
  • Open — calls fail fast for ttl seconds.
  • Half-open — one trial call to see if it works again.

This is what keeps a flaky downstream from poisoning your whole agent latency.

3. Loop detection and step caps

LLMs sometimes get stuck calling the same tool with the same args. Detect and break.

python
seen = set()

def tool_with_loop_guard(name, args, run):
    sig = (name, json.dumps(args, sort_keys=True))
    if sig in seen:
        return {"error": "loop detected: same tool+args called twice"}
    seen.add(sig)
    return run()

Plus a hard step cap — max_steps=8 for complex agents, max_steps=5 for simple ones. Always.

4. Rate limiting (do not annoy your provider)

python
# uv add aiolimiter
from aiolimiter import AsyncLimiter
limiter = AsyncLimiter(max_rate=10, time_period=1)  # 10 calls / sec

async def call_llm(prompt):
    async with limiter:
        return await client.responses.create(model="gpt-4.1-mini", input=prompt)

For multi-tenant apps, key-scope the limiter (one per user_id) so a noisy customer cannot hog quota.

5. Logging for agents (structured, never print)

python
# uv add structlog
import structlog
log = structlog.get_logger()

log.info("tool.call",
         tool="create_refund",
         args={"invoice_id": "INV-...","amount": 340},
         user_id=user_id, request_id=req_id, duration_ms=42)

Hard rules:

  • Always include user_id, request_id, agent_name, node, model.
  • Never log raw prompts/answers in production unless redacted (PII risk).
  • JSON output so structured log shippers (Loki, Datadog) can index.

Modern stacks ship logs to Loki + Grafana or Datadog. Your job is to emit them in a clean shape.

6. LangSmith tracing (the easy 80%)

LangSmith automatically traces LangChain and LangGraph runs when you set two env vars:

LANGSMITH_TRACING=true
LANGSMITH_API_KEY=ls__...
LANGSMITH_PROJECT=my-agent

For non-LangChain code, use the @traceable decorator and the wrap_openai helper.

python
# uv add langsmith
from langsmith import traceable
from langsmith.wrappers import wrap_openai
from openai import OpenAI

oa = wrap_openai(OpenAI())  # wraps so every LLM call is auto-traced

@traceable(run_type="tool")
def get_context(question: str) -> str:
    # imagine a vector retrieval
    return "Some context"

@traceable
def assistant(question: str) -> str:
    ctx = get_context(question)
    r = oa.chat.completions.create(
        model="gpt-4.1-mini",
        messages=[
            {"role": "system", "content": f"Answer using context.\n\n{ctx}"},
            {"role": "user",   "content": question},
        ],
    )
    return r.choices[0].message.content

print(assistant("What is RAG?"))

You will see:

  • A trace tree (assistant > get_context, assistant > openai).
  • Latency per span, token cost, full inputs/outputs.
  • Errors highlighted.

In LangGraph, the entire graph plus every node is traced automatically when env vars are set. No code changes needed. That alone is a magnificent quality-of-life upgrade.

7. LangFuse and others (open-source / self-hostable)

If you cannot send data to a managed service:

  • LangFuse — open source, self-hostable, similar UX to LangSmith.
  • Helicone — proxy-style.
  • Phoenix (Arize) — best for embedding drift and RAG monitoring.
  • PromptLayer — simple.
python
# uv add langfuse
from langfuse.openai import openai  # drop-in
r = openai.chat.completions.create(model="gpt-4.1-mini", messages=[...])

8. OpenTelemetry — vendor-neutral and the way enterprise wants it

For enterprises that already have a tracing platform (Tempo / Jaeger / Datadog APM), instrument your agent with OpenTelemetry. LangChain has an OTel integration.

python
# uv add openinference-instrumentation-openai openinference-instrumentation-langchain opentelemetry-sdk opentelemetry-exporter-otlp
from openinference.instrumentation.openai import OpenAIInstrumentor
from openinference.instrumentation.langchain import LangChainInstrumentor

OpenAIInstrumentor().instrument()
LangChainInstrumentor().instrument()

Now every OpenAI/LangChain call emits OTel spans. Set OTEL_EXPORTER_OTLP_ENDPOINT=https://your-collector and you are wired into the rest of the company stack.

OpenInference is the open semantic conventions for LLM tracing — backed by Arize, Microsoft, OpenAI, and the LF AI & Data Foundation.

9. Cost and latency dashboards (one Grafana panel = senior signal)

Track these metrics per agent:

  • Requests / min.
  • p50, p95, p99 latency.
  • Tokens in / out per request.
  • USD cost per request.
  • Error rate by node / tool.
  • Cache hit rate.
  • Avg steps per agent run.

Send via Prometheus client (prometheus_client library) or a managed APM. Building one Grafana dashboard for your project — and screenshotting it in your README — is one of the strongest portfolio moves there is.

Hands-on lab (5 hours)

Take your LangGraph agent from Lesson 3.3.

  1. Add tenacity retry + purgatory circuit breaker around every external HTTP call.
  2. Add a loop guard inside the tool dispatcher.
  3. Add structured logging (structlog) with user_id, request_id, node.
  4. Turn on LangSmith tracing (env vars only).
  5. Add OpenInference + OTel instrumentation, exporting to a local Jaeger (docker run jaegertracing/all-in-one).
  6. Add Prometheus metrics: counters for agent_steps_total, gauges for agent_in_flight, histograms for agent_latency_seconds. Expose /metrics on FastAPI.
  7. Build a Grafana dashboard panel reading those metrics. Screenshot.

Acceptance:

  • A simulated downstream outage opens the breaker; agent fails fast within 30s.
  • LangSmith trace is shareable via URL.
  • Jaeger shows the OTel waterfall.
  • README has Grafana screenshot.

Common pitfalls

  1. No timeouts. A hanging tool freezes your event loop.
  2. Retrying non-idempotent operations. Add idempotency keys.
  3. Catching Exception. Be specific — let bugs surface.
  4. print() in production code. Use structlog/JSON.
  5. Tracing PII. Add a redactor before sending to managed observability tools.

Self-check

  1. What does a circuit breaker's "half-open" state do?
  2. Why does idempotency matter for refund tools?
  3. What is the simplest way to enable LangSmith tracing for a LangGraph app?
  4. Why would you choose OpenTelemetry over LangSmith in some companies?
  5. What three metrics would you alarm on for an agent service?

References

Sign in to save your progress and earn badges.