Production observability and cost engineering

Why this matters

The most common reason agent products fail in production is opacity and runaway cost, not bad models. The skill of "I can debug, attribute, and shrink the bill" makes you the senior engineer the team relies on.

This lesson is the production-focused expansion of Lesson 3.5 — same tools, deeper patterns.

Learning objectives

1. Stand up a LangSmith + Langfuse + OpenTelemetry/Phoenix stack and pick the right one.
2. Tag traces with tenant_id, user_id, feature, version for surgical debugging.
3. Build a cost dashboard segmented by feature.
4. Apply the cost optimisation pyramid end-to-end.
5. Set SLOs and alarms.

1. Observability stack picks for 2026

In 2026 most stacks are: LangSmith for dev/experiments + OTel+Phoenix or Langfuse for prod.

2. The 6 fields you must tag on every trace

Senior agents always tag:

• tenant_id (or org_id) — the customer.
• user_id — the human.
• request_id — UUID for cross-system correlation.
• feature — chat, summarise, code_review, etc.
• version — your app/service version.
• model and provider.

Use LangSmith metadata or LangChain RunnableConfig:

config = {"metadata": {"tenant_id": tid, "user_id": uid, "feature":"chat",
                        "version": APP_VERSION},
          "tags": [f"v{APP_VERSION}", f"tenant:{tid}", "feature:chat"]}
chain.invoke(input, config=config)

In OTel:

from opentelemetry import trace
trace.get_current_span().set_attributes({
    "tenant.id": tid, "user.id": uid, "feature":"chat",
    "service.version": APP_VERSION,
})

Now you can slice metrics by any tag in dashboards.

3. Three dashboards every agent needs

Dashboard A — health

• Requests / minute (per feature).
• Error rate (per node, per tool).
• p50/p95/p99 latency (per feature).
• Token throughput.
• "I do not know" rate.

Dashboard B — quality

• Online task-success rate (judged on sampled traffic).
• RAG faithfulness (sampled).
• Refusal rate.
• Adversarial bucket hit count.

Dashboard C — cost

• USD / 1k requests by feature.
• Token mix (input vs output) by model.
• Cache hit rate.
• Cost share between models.
• Forecast vs budget.

A single Grafana repo with these three dashboards in JSON is interview gold.

4. Cost optimisation pyramid (production version)

From Lesson 1.5, expanded for prod:

1. Right-size the model. Run a real eval; pick smallest that meets quality SLO.
2. Prompt caching. Anthropic 90% off; OpenAI auto cache when shared prefix > 1024 tokens.
3. Semantic cache for FAQ-style queries.
4. Output budgets. Strict max_tokens. Stop sequences for templated outputs.
5. Context compression. LLMLingua-2 for large doc QA; chunk + retrieve instead.
6. Cascade. Try cheap → escalate to expensive only on low confidence.
7. Batch APIs (50% off) for nightly evals, embeddings, summarisations.
8. Self-host with vLLM when daily volume justifies (Lesson 5.5).
9. Move retrieval/rerank off the LLM when classical algorithms suffice (BM25, cosine).

Document each one's win in your README. "Prompt caching cut input cost 73% on the help-bot feature" is exactly the line a hiring manager wants to see.

5. SLOs and alarms

Define service-level objectives explicitly:

• 99.5% requests succeed (no 5xx).
• p95 latency < 4s.
• "I do not know" rate < 8%.
• Cost per request < $0.02 (chat); < $0.20 (deep-agent).
• Faithfulness ≥ 0.85 weekly average.

Alert on trend, not just single thresholds:

• 7-day rolling task-success drops > 5% → page on-call.
• Cost / 1k rises > 25% week-over-week → investigate.
• New 4xx error pattern (unknown_tool) > 1/hour → investigate.

Use Grafana / Datadog Monitors / Prometheus Alertmanager.

6. Trace-driven debugging workflow

Real production debugging:

1. User reports a bad answer. They send request_id.
2. Open LangSmith / Langfuse → search by request_id.
3. View the full trace tree: which retriever returned what, which chunks survived rerank, which tools were called with what args, which LLM call produced the bad reply.
4. Replay the run with one variable changed (different model, different chunk size, different system prompt). Compare results side-by-side.
5. Add a test case to your golden set so the bug never returns.

This loop — observe → replay → fix → add test — is the senior workflow. Practice it on your own projects.

7. Privacy in observability (the rule that gets violated)

You must redact before the trace leaves your perimeter:

• PII (Presidio / regex).
• Secrets (regex; deny-list known prefixes).
• Customer business secrets.

Most platforms support a redaction callback; otherwise wrap your client to scrub inputs/outputs before the trace upload.

A common 2026 pattern: store full traces internally (Phoenix or Langfuse self-hosted) and a redacted summary externally.

Hands-on lab (5 hours)

Add to your agent stack:

1. LangSmith metadata on every run with the 6 standard tags.
2. Phoenix OSS instance via Docker, instrumented via OpenInference.
3. Prometheus + Grafana with the 3 dashboards above.
4. Cost meter middleware that emits agent_cost_usd_total{feature=...}.
5. Alertmanager rules: alert if rate(agent_errors_total[5m]) > 5, alert if avg(agent_cost_usd_total[1h]) > 0.05.
6. PII redactor in the trace upload path.

Acceptance:

• Trigger a load test; dashboards populate live.
• Trip an alarm intentionally; receive the notification.
• README screenshots all three dashboards.

Common pitfalls

1. No tagging. You cannot slice by feature when it matters most.
2. Storing raw PII. Redact at the source, not "later."
3. Single threshold alerts. They fire forever or never. Use rate-of-change alarms.
4. Tracing only LLM calls. Trace the whole request including retrieval and tools.
5. Tracking cost in the API console. Always have a per-feature meter in your code.

Self-check

1. Why is tenant_id non-negotiable for B2B agents?
2. How does prompt caching show up in usage fields?
3. Two ways to detect retrieval drift in production.
4. Why do trend-based alerts beat single-threshold alerts?
5. How would you reduce a feature's cost by 50% if budgets force it?

References

• LangSmith metadata + tags
• Langfuse self-host
• Arize Phoenix
• OpenInference semantic conventions
• Datadog LLM Observability
• Anthropic prompt caching guide