Interview prep — LLM system design templates — Large Language Models

In senior LLM interviews you are expected to design large systems on a whiteboard: pretraining infra, serving stacks, evaluation pipelines, fine-tuning workflows. This file gives you ready-made templates plus the interviewer's hidden checklist for each.

For each prompt:

Clarify scope and constraints (5 min).
Sketch architecture (5 min).
Walk through data flow with example numbers (10 min).
Discuss bottlenecks, failure modes, and trade-offs (10 min).
Talk about evaluation and rollout (5 min).

Template 1 — "Design a pretraining system for a 70B-parameter LLM."

Clarifying questions

Compute budget? (e.g., 4096 H100 GPUs for 60 days.)
Target tokens? (D = 15T over-trained.)
Multilingual? Code? Math?
Open weights or proprietary?
Can we use synthetic data?

Architecture

mermaid

flowchart LR
    SRC[CommonCrawl + GitHub + Books + Synthetic]
    SRC --> CLEAN[Clean / dedup / filter / decontaminate]
    CLEAN --> TOK[Tokenize → uint32 shards]
    TOK --> S3[(Object store)]
    S3 --> LOADER[Distributed iterable dataset]
    LOADER --> JOB[Megatron / FSDP+TP+PP training job]
    JOB --> CKPT[(Async checkpoints)]
    JOB --> WB[WandB monitoring]
    CKPT --> ANNEAL[Annealing stage]
    ANNEAL --> RELEASE[Final base model]

Numbers

6 N D = 6 * 7e10 * 1.5e13 = 6.3e24 FLOPs.
H100 bf16 peak: ~989 TFLOPS. MFU ~0.4 → ~390 TFLOPS.
GPU-hours: 6.3e24 / (390e12 * 3600) ≈ 4.5M. With 4096 GPUs, ~1100 hours = ~46 days.
Tokens/sec target: D / runtime_seconds = 1.5e13 / (46 * 86400) ≈ 3.8M tokens/s cluster-wide.

Bottlenecks

Data: 15T tokens at uint32 = 60TB just for tokenized data; need fast streaming reads from sharded object store.
NCCL: TP all-reduce dominates intra-node; needs NVLink/NVSwitch.
Failure rate: at 4096 GPUs over 46 days, expect dozens of GPU failures. Need async checkpointing + auto-restart.
Loss spikes: skip-on-NaN, reduce lr if stable, etc.

Eval and rollout

Periodic small eval (MMLU, HellaSwag) every 50B tokens.
Annealing on best 200B mix for last 10% of tokens.
Long-context extension as a final stage.
Release plan: base model → SFT → DPO → eval → red-team → ship.

Hidden interviewer checklist

You said "Chinchilla" but accepted that you'd over-train.
Mentioned dedup AND decontamination.
Knew bf16 + FSDP + TP+PP combo.
Discussed fault tolerance.
Discussed FLOPs estimate explicitly.
Knew that data, not architecture, is the lever.

Template 2 — "Design an inference platform serving Llama-3 70B at 100 RPS with 99p 5s latency."

Clarifying

Are requests RAG-style (large prompts, short outputs) or chat-style (small prompt, long output)?
Geo: single region or global?
Cost target?
Variants: do we serve multiple models / fine-tunes?

Architecture

mermaid

flowchart LR
    USER --> EDGE[CDN / Edge gateway]
    EDGE --> APIGW[API gateway: auth, rate-limit]
    APIGW --> ROUTER[Smart router: model select, fail-over]
    ROUTER --> POOL[vLLM pool: 70B AWQ INT4, GQA, FP8 KV, prefix cache, spec decode]
    POOL --> REDIS[Redis: chat history, tenant policies]
    POOL --> METRICS[OTel → Datadog/Prom + Langfuse]
    METRICS --> ALARMS[Alarms]

Numbers

70B AWQ INT4: ~35 GB model. Fits on 1× H100 (80 GB) with ~45 GB free for KV.
Llama-3 70B KV: ~320 KB/token. 45 GB / 320 KB ≈ 140k tokens of KV per GPU.
Concurrent slots at avg T=4k: 35 per GPU. At 30 tokens/sec/slot output, ~1000 tokens/s/GPU output.
100 RPS at avg 500 output tokens per request → 50k tokens/s. Need ~50 H100s.
Cost: 50 × $3/hr × 24 × 30 ≈ $108k/month.

Bottlenecks

KV cache at long context (RAG with 16k prompts) reduces concurrency.
TTFT: prefill of long prompt is slow without chunked prefill (vLLM --enable-chunked-prefill).
Tail latency: long generations block the slot. Mitigate with separate fast/slow pools, premption.
Hot prompts: prefix caching saves cost ~60% on RAG workloads.

Optimisation pyramid

Prefix caching (often 5-10× cost reduction).
FP8 KV cache (2× concurrency at long context).
AWQ INT4 weights (already chosen).
Speculative decoding (2× decode speed at batch=1).
Smaller-model fallback for simple queries.

Hidden checklist

Distinguished prefill vs decode latency.
Computed KV memory and concurrency.
Mentioned PagedAttention / continuous batching by name.
Discussed prefix caching.
Used a quantization choice with tradeoff justification.
Considered observability and on-call alarms.

Template 3 — "Design a fine-tuning pipeline for 50 customers, each with their own LoRA."

Architecture

mermaid

flowchart LR
    CUST[Customer data] --> VAL[Validate + redact PII]
    VAL --> SFTQ[SFT job queue]
    SFTQ --> KUB[Kubernetes job: 1× H100, QLoRA SFT]
    KUB --> EVAL[Eval suite: capability + custom + safety]
    EVAL --> REG[(LoRA registry, S3 + Postgres)]
    REG --> SERVE[vLLM with --enable-lora --max-loras=64]
    SERVE --> ROUTER[Tenant-aware router]

Key decisions

LoRA only (not full FT): 100× cheaper, fits on commodity GPU.
Base model frozen and shared across all 50 LoRAs (vLLM supports it).
Per-customer holdout eval set required at intake.
Periodic re-train when customer data updates.
Safety eval (XSTest, AdvBench-50) gate before deployment.

Cost estimate

~$10-30 per customer fine-tune (a few GPU-hours each).
Storage: ~50 MB per LoRA × 50 = 2.5 GB.
Serving: shared base model, swap-in adapters at request time. No per-customer GPU.

Risks

Adapter staleness vs base updates.
Cross-customer prompt injection in shared base.
Drift in quality without regular eval.

Hidden checklist

You did not propose 50× full fine-tunes.
Mentioned vLLM multi-LoRA serving.
Included a per-tenant eval gate.
Considered adapter rotation and base-model upgrades.

Template 4 — "Design an evaluation pipeline that runs on every model checkpoint."

Architecture

mermaid

flowchart TB
    CKPT[New checkpoint] --> CONT[Decontamination scan]
    CONT --> CAP[Capability suite: MMLU, GPQA, GSM8K, HumanEval, IFEval]
    CAP --> CHAT[Chat suite: AlpacaEval LC, MT-Bench]
    CHAT --> SAF[Safety: XSTest, AdvBench, BBQ]
    SAF --> CUST[Custom product evals + LLM-judge]
    CUST --> REG[Eval registry: MLflow/W&B]
    REG --> GATE[Gate: regressions blocked]
    GATE --> DEPLOY[Deploy → canary → full]

Decisions

Decontamination check first; flag any 13-gram leak.
Run capability suite via lm-evaluation-harness (standardised).
Chat suite via alpaca_eval and MT-Bench.
Custom suite (~200 prompts) judged by gpt-4o, position-randomised, 3-judge majority on borderline.
Per-axis pass criteria; overall gate blocks if any axis regresses by > 1.5%.

Cost

Capability: ~$5 (open models on local GPU).
Chat suite: ~$30 (judge API).
Custom: ~$5-20.
Total: ~$50-100 per checkpoint. Cheap; run on every commit.

Hidden checklist

Decontamination was step 1.
Both rule-based and LLM-judge components.
Versioned eval set in git.
Per-axis (capability/safety/chat) gates, not single number.
Result registry for trend tracking.

Template 5 — "Design an LLM agent platform serving 1k different agent definitions."

(High-level; defer to course/ for the full agentic stack.)

Components

Agent registry (definitions, tools, prompts) — versioned.
Runtime: LangGraph / CrewAI / custom orchestrator.
Tool layer: MCP servers per integration; sandboxed execution.
LLM router: choose model per agent (small for fast, large for hard).
Memory: per-tenant memory (Mem0 / pgvector).
Eval: trajectory-level (TAU-bench-style) per agent.
Cost / observability: LangSmith / Langfuse.
Safety: PII redaction in/out; jailbreak filter; HITL on destructive actions.

Hidden checklist

Mentioned MCP for tool standardisation.
Per-agent eval, not just per-model.
HITL for high-impact actions.
Cost meter per tenant.
Noted the difference between capability (TAU-bench) and safety (InjecAgent).

Template 6 — "Design a system to fine-tune a model on RLVR for math."

Pipeline

mermaid

flowchart LR
    PROMPTS[Math problems with verifiable answers] --> ROLL[Sample N completions per prompt]
    ROLL --> VERIFY[Rule-based reward: numeric match / code passes]
    VERIFY --> TRAIN[GRPO update]
    TRAIN --> ITER[Iterate]
    ITER --> EVAL[GSM8K / MATH / AIME]
    EVAL --> SHIP

Decisions

Base: Qwen2.5-Math-7B-Instruct (math-pretrained).
Sample 8-16 completions per prompt at varied temperatures.
Reward: numeric exact match + format compliance.
Algorithm: GRPO (no critic).
Length penalty to prevent runaway thinking.
Eval every 200 steps; auto-stop on saturation.

Numbers

50k prompts × 16 samples × ~1k tokens each = 800M generated tokens.
One H100 ~ 100 tok/s for 7B model = ~9000 H100-hours just for sampling. Use vLLM batched.
Total ~1-2 weeks on 8× H100.

Risks

Reward hacking ("the answer is always 42").
Format reward dominates correctness.
Catastrophic forgetting of general chat ability — mix in some general SFT during training.

Hidden checklist

Used GRPO over PPO for the right reasons (no critic).
Reward is rule-based, not LLM-judged.
Mentioned mixing in general data to preserve capabilities.
Tracked reasoning length explicitly.

Cross-cutting interview tactics

Start with constraints — the wrong design comes from skipping the question "what do we optimise for?"
Use real numbers — calling Llama 3 70B "huge" is amateur; saying "140 GB at bf16, fits on 2× H100 with TP=2" is professional.
Mention names — FlashAttention, PagedAttention, GQA, RoPE, YaRN, GRPO, FSDP, NCCL, MFU, AWQ. Specific names beat vague descriptions.
Discuss failure modes — what happens when this system breaks? That is what senior engineers are paid for.
Cost — every system question is implicitly about $/QPS or $/token. Bring it up unprompted.

You will encounter 1-2 of these in any senior LLM interview. Practice each at least once aloud.