Fine-tuned open model served via vLLM — "BYO LLM"

After Phase 5. This project proves you can move from API-only to a full self-hosted model stack — a skill that immediately bumps you into senior infrastructure roles.

What you ship

A fine-tuned Llama 3.1 8B (or Qwen 2.5 7B / Mistral) specialised for a narrow domain task (SQL, support replies, JSON extraction, ...) served over vLLM with an OpenAI-compatible API, gated by LiteLLM Proxy, monitored via Prometheus + Grafana, and benchmarked against the original API baseline (Anthropic / OpenAI) on cost and quality.

Stack

• Dataset: 2k-5k high-quality examples (synthetic + human-reviewed).
• Trainer: Unsloth QLoRA → optional DPO.
• Model: Llama 3.1 8B / Qwen 2.5 7B (any open-weight chat model).
• Serving: vLLM Docker, AWQ-INT4 or FP8.
• Gateway: LiteLLM Proxy.
• Observability: Prometheus + Grafana + LangSmith.
• Comparison: an A/B harness against gpt-4.1-mini and claude-haiku-4-5.
• Deployment: Docker Compose (or a single GPU on Vast.ai / RunPod).

Step-by-step

Day 1 — Choose task + build dataset

• Pick a narrow task: e.g., "Convert natural-language questions into SQL for our schema" or "Reply to customer-support emails in our brand voice."
• Generate 2-5k examples: synthesise with GPT-4.1, then hand-review or rule-validate.
• Train/eval split 90/10. No leakage. Document data card.

Day 2-3 — Train

• Unsloth Colab or local 4090.
• QLoRA, r=16, lr=2e-4, 2 epochs.
• Save LoRA, merged safetensors, and GGUF artefacts.
• Run eval before and after — confirm domain accuracy lift, no regression on a general bench.

Day 4 — DPO (optional but impressive)

• Generate 500 preference pairs (chosen vs rejected) using the SFT model + GPT-4.1 judge.
• Run DPOTrainer for 1 epoch.
• Re-evaluate.

Day 5 — Serve with vLLM

• Build a Dockerfile (vllm/vllm-openai:latest base + your merged model).
• Add --quantization awq and --enable-prefix-caching.
• Confirm OpenAI-compatible API works.
• Add nginx with API-key auth and rate limit.

Day 6 — Gateway and benchmark

• Front with LiteLLM Proxy that aliases your model + GPT-4.1-mini + Claude Haiku.
• Build a benchmark.py that sends 200 production-shaped requests to all three and records: success, latency, cost.
• Render a markdown table.

Day 7 — Polish

• Write the cost story:
  • "Replacing GPT-4.1-mini with our fine-tuned 8B on 100k requests/day saved $X / month at the same quality."
• Publish the README with mermaid, training curves, eval matrix, latency p50/p95, and a final cost analysis spreadsheet.
• Optional: push merged model to HuggingFace + Modelfile to Ollama.

Architecture (mermaid)

flowchart LR
  CLIENT[Agent / App] --> NGX[nginx + TLS]
  NGX --> LL[LiteLLM Proxy]
  LL -->|alias: my-model| VLLM[vLLM container]
  LL -->|alias: claude-haiku| ANTH[Anthropic API]
  LL -->|alias: gpt-4.1-mini| OPENAI[OpenAI API]
  VLLM -->|/metrics| PROM[Prometheus]
  PROM --> GRAF[Grafana]

Acceptance checklist

• Domain accuracy ≥ 90% of the GPT-4.1-mini baseline (or better).
• Latency p95 < 2s on a single L40S / 4090.
• Demonstrated 50-90% cost cut at the test volume.
• LiteLLM dashboard with per-key usage.
• Grafana dashboard with queue depth, KV-cache, p95 latency.
• Reproducible: make benchmark runs the comparison.
• HuggingFace model card published.

What hiring managers love

• A clear break-even analysis (when does self-host beat API?).
• Cost saved expressed in dollars, not just percent.
• An honest "where the API still wins" section (e.g., long-context reasoning).
• A model card that demonstrates safety thought (refusal behaviour, license).

This project converts a "promising junior" CV into a "this person can run an LLM platform" CV.