Domain-specialised LLM via LoRA + LLM-as-judge eval

Goal

Take an open base model (Llama-3.1-8B, Qwen2.5-7B, or Mistral-7B-v0.3) and fine-tune it for a specific domain (code review, healthcare summaries, legal Q&A, financial reports, your own niche). Evaluate rigorously with rule-based + LLM-as-judge. Ship a deployable adapter.

The "I shipped a real fine-tune" project. The first one most ML hiring managers ask for.

Time: 2-3 weeks part-time.

Prerequisites

• 04_posttraining/01_sft.md
• 07_eval_and_interpretability/01_evaluation.md

Tech stack

• transformers, trl, peft, bitsandbytes
• datasets, unsloth (optional; faster training)
• lm-evaluation-harness for capability sanity-check
• gpt-4o-mini or claude-haiku-4-5 as judge
• vLLM for serving

Pick a domain

Pick one that you find interesting; depth > breadth.

Examples:

1. Python code-review: (diff, review_comment) pairs from open-source PRs.
2. Medical Q&A: MedMCQA + MedQA + clinical guidelines.
3. Indian legal: legal Q&A with citations to the relevant act.
4. Finance: 10-K summaries, earnings-call sentiment.
5. Customer support for X: synthetic from product docs.

Data pipeline

1. Collect

• 5-50k high-quality (prompt, completion) examples.
• Include diversity: short/long, easy/hard, with/without context.

2. Format

Use the model's chat template (Lesson 4.1). Each example becomes:

{"messages": [
   {"role": "system", "content": "You are an expert ..."},
   {"role": "user", "content": "..."},
   {"role": "assistant", "content": "..."}
]}

3. Quality-filter

• Length filter (drop too-short / too-long).
• Run a small dedup (MinHash) — see Lesson 3.2.
• Sample 50 random examples and read them; reject if 10%+ are noisy.

4. Holdout eval set

• Carve out 200 examples as a held-out test set. Never train on these.

Training

LoRA SFT (Lesson 4.1) with QLoRA for memory:

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype=torch.bfloat16,
                         bnb_4bit_use_double_quant=True)
peft_cfg = LoraConfig(r=32, lora_alpha=64, lora_dropout=0.05, bias="none",
                      target_modules=[...all 7 in-block linears...],
                      task_type="CAUSAL_LM")
sft_cfg = SFTConfig(num_train_epochs=2, per_device_train_batch_size=2,
                    gradient_accumulation_steps=8, learning_rate=2e-4,
                    lr_scheduler_type="cosine", warmup_ratio=0.03,
                    bf16=True, packing=True, assistant_only_loss=True,
                    max_seq_length=4096)

Train for 1-3 epochs. Save adapter.

Evaluation

A. Capability sanity (no regressions)

Run lm-evaluation-harness on mmlu, arc_challenge, gsm8k. Compare base vs fine-tuned. Should be within 1-2%.

B. Domain quality

Custom eval script that, for each held-out example:

1. Generates an answer (vLLM serving).
2. Computes rule-based scores: contains required keywords, parses as JSON, length within bounds.
3. Has gpt-4o-mini judge against the gold answer with a domain-specific rubric.
4. Aggregates: % rule pass, % judge prefers fine-tune over base, % judge prefers fine-tune over gold.

RUBRIC = """
You are a senior {domain} reviewer. Score each response from 1-5 on:
  factuality, helpfulness, citation, format adherence.
Return JSON: {"factuality": 1-5, "helpfulness": 1-5, "citation": 1-5, "format": 1-5, "comments": "..."}.
"""

C. Adversarial / safety

Quick XSTest (50 prompts) — your fine-tune should not break refusal behaviour.

D. Cost / latency

• Tokens/sec at vLLM serving.
• Average tokens per response.
• $/1k requests at on-demand and at saturation.

Serving

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-lora \
  --lora-modules my-lora=path/to/adapter \
  --max-model-len 8192 \
  --enable-prefix-caching

Now you can route requests to base or LoRA adapter via the model field. Showcase this in your README.

Acceptance criteria

• Public dataset card on HuggingFace describing your data.
• LoRA adapter on HuggingFace (your-name/llama3-8b-XYZ-lora).
• README: domain, training config, eval table, sample outputs, cost.
• Eval table compares base vs fine-tuned on domain quality, capability, safety.
• vLLM serving instructions reproducible from a single docker compose up.
• Adversarial section showing where the model still fails.

Stretch goals

• Add a second stage: DPO with synthetic preference pairs (Project 4).
• Compare against full fine-tune at the same compute budget.
• Try Unsloth for 2× faster training.
• Add streamlit demo with side-by-side base vs fine-tune.
• Evaluate cost-per-task vs an API call to GPT-4o-mini and discuss when fine-tuning wins.

Common pitfalls

• Wrong chat template → silent regression.
• Training loss looks great, eval is mediocre → likely overfitting; reduce epochs / shrink LoRA r.
• assistant_only_loss=False → model imitates user style.
• Mismatched tokenizer between data prep and model.
• LoRA adapter trained with one model, served with another quantised version → adapter rank-mismatch errors. Always serve with the same base.

Story / portfolio

• Title: "I built a {domain}-specialised LLM with $50 of compute."
• Section: "What I learned tuning hyperparameters."
• Charts: judge win-rate vs base, capability deltas, cost vs API.
• Demo gif: side-by-side responses.
• Final: "When I would and would not use fine-tuning."

This project demonstrates the complete workflow every fine-tuning team runs: data, training, eval, serving. It is the most directly applicable project for a hands-on engineer role.