AskAiml
Sign in Get started

Interview prep — system design templates

Reusable diagrams and talk tracks for RAG, multi-agent, evals, and cost — the systems interviewers actually ask about.

🎯 Module 6 6 min read Not started

These are the 6 system-design prompts that come up most often in agentic AI interviews. For each, you get the canonical answer skeleton (architecture + decisions + trade-offs). Practise drawing each one on a whiteboard in under 30 minutes.

How to attack any "design an X agent" question

A reliable framework:

  1. Clarify (2 min): users, scale, latency SLO, budget, regulatory constraints, channels, tool surfaces, failure modes you must handle.
  2. Sketch (5 min): a mermaid-style diagram with: client, gateway, agent runtime, models, retrieval store, tools, guardrails, observability, persistence, batch jobs.
  3. Pick technologies (5 min): name the library/service per box and why in 1 line each.
  4. Walk through one request (5 min): from "user types" to "answer rendered" with token + retrieval flow.
  5. Discuss failure modes & mitigations (5 min): rate limits, hallucinations, prompt injection, downstream outage, runaway cost.
  6. Ops & rollout (5 min): canary, online evals, alarms, rollback.
  7. What I would do next (3 min): two future improvements.

That structure works for every prompt below.

Template 1 — "Design an enterprise customer support agent"

Clients (web/Slack/Zendesk webhook)
  ↓
API gateway (auth, rate limit, redaction)
  ↓
LangGraph supervisor agent (Postgres checkpointer)
  ├── Triage classifier (Pydantic intent)
  ├── KB retriever (pgvector + BM25 + Cohere rerank-3)
  ├── Tools: refund_create, ticket_create, get_invoice, search_kb (via MCP)
  ├── Guardrails: NeMo input/output, Presidio PII, Llama Guard 3
  └── HIL via interrupt() for refunds > threshold
  ↓
Models: gpt-4.1-mini (router), claude-sonnet-4-6 (responder), claude-haiku-4-5 (judge)
  ↓
Mem0 user memory + Postgres conversation history
  ↓
Observability: LangSmith + Phoenix + Grafana cost dashboards
  ↓
CI: golden 200-case eval (DeepEval), faithfulness + tool-correctness gates

Top trade-offs to mention:

  • pgvector vs Qdrant: chose pgvector to keep one DB and support row-level security per tenant.
  • Cascade vs single-model: cascade saves 50% cost; we keep the same model only for tickets routed as "high-stakes" (refunds, escalations).
  • HIL bar: $50 in INR; we instrument what % of refunds are auto vs human-approved.

Failure modes:

  • Retrieval drift on policy updates → re-index pipeline triggered by Confluence webhook.
  • Prompt-injected ticket bodies → strip & quarantine retrieved content with extra "user_data" tags + Lakera Guard.
  • Provider outage → LiteLLM proxy auto-fallback Claude Haiku ↔ GPT-4.1-mini.

Template 2 — "Design a personal AI assistant (multi-tool)"

Voice / Web client → FastAPI gateway → LangGraph agent (SqliteSaver per device)
  → Tools: calendar, email-draft, notes RAG, web search, code sandbox
  → Mem0 long-term user memory + Anthropic prompt caching for stable system prompt
  → Guardrails: Presidio + dangerous-tool gating (interrupt() before send)
  → Models: claude-haiku-4-5 default; claude-opus-4-7 on user "deep think" toggle
  → LangSmith tracing with metadata {user_id, feature}
  → Cost dashboard with daily budget per user

Trade-offs:

  • Claude Haiku default for cost; Opus toggle for hard tasks.
  • Sqlite checkpointer per device (offline-friendly) vs cloud Postgres for sync.
  • HIL on email send: always require confirmation.

Template 3 — "Design a multi-document RAG over 1M PDFs"

Ingest pipeline (Airflow / Prefect):
  source → Unstructured parse → contextual chunk + parent-child → embed (voyage-3-large)
  → write to pgvector + Elasticsearch (BM25)

Online query path:
  user → multi-query expand (3 reformulations) → BM25 + dense → RRF top 50
  → Cohere rerank-3 → top 5 → LLM synthesis (gpt-4.1-mini)
  → if confidence low → CRAG fallback to Tavily web

Eval: RAGAS suite + golden 500-case → CI gate.
Deploy: Kubernetes; vLLM cluster for embeddings; LangSmith.

Trade-offs:

  • pgvector vs Qdrant: pgvector for ACID + joins, Qdrant for >10M vectors with low latency. At 1M, pgvector + HNSW is fine.
  • Voyage-3 vs OpenAI: voyage benchmarks higher on retrieval; choose based on eval lift.
  • LlamaParse for tables, Unstructured for general — pick by source type.

Template 4 — "Design a coding agent that opens PRs"

Trigger: GitHub issue or webhook
  ↓
LangGraph agent
  ├── Plan node (decompose issue into steps)
  ├── Read repo via tree-sitter + ripgrep MCP server
  ├── Edit files via filesystem MCP
  ├── Run tests in E2B sandbox
  ├── Reflect & re-plan up to 3 times
  └── Open PR via GitHub MCP server
  ↓
Models: claude-sonnet-4-6 (planner+coder), claude-haiku-4-5 (linter/triage)
  ↓
Observability: LangSmith trace; AgentEval trajectory metrics
  ↓
HIL: PR description requires human review before merge (default Git workflow)

Trade-offs:

  • Sandbox: E2B for ephemeral, Modal for higher concurrency, Daytona for full IDE.
  • Tree-sitter vs LSP: tree-sitter cheaper for navigation; LSP needed for refactors.
  • Cost ceiling per task: hard cap, e.g., $1 per PR.

Template 5 — "Design a high-throughput LLM gateway"

Clients → nginx (TLS, rate-limit) → LiteLLM Proxy
  → Model pools:
      • OpenAI/Anthropic API keys (rotated weekly)
      • Self-hosted vLLM cluster (autoscaled on queue depth)
  → Postgres for usage/billing
  → Redis for prompt + semantic cache
  → Prometheus + Grafana
  → Audit log via Kafka → S3

Concerns to raise:

  • Per-tenant rate limits in LiteLLM.
  • Bring-your-own-key vs shared keys.
  • Streaming SSE through nginx (correct buffering).
  • Failover: route 25% to alt provider on >2% error rate.

Template 6 — "Design a deep research agent"

A2A endpoint exposes the agent as a service.
LangGraph deep agent runtime:
  • planner writes plan.md
  • for each step → spawn sub-agent (research/analyst/writer)
  • virtual filesystem out/{run_id}/
  • reflect & retry on failed step
  • cost & step ceilings
Tools surface via MCP server: web_search, fetch_url, save_file, summarise
Observability: LangSmith tracing each sub-agent
HIL: human approval before publishing

Trade-offs:

  • Pure LangGraph vs deepagents library: deepagents faster to ship, LangGraph custom for complex flows.
  • Where to run: a single VM is enough for hour-long jobs; for parallel users use queue + workers.
  • Cost: typically $0.10-$0.30 per polished report at small volume.

Drawing tips

  • Always show data flow direction (arrows).
  • Always label boxes with the technology choice (not just "DB").
  • Always note the model (gpt-4.1-mini) per LLM call.
  • Always mention observability and evals. Most candidates forget.
  • Always close with "what I'd do next."

Practise the canonical 6 above. Vary slightly by industry. You will not be surprised in interviews.

Sign in to save your progress and earn badges.