Preference alignment with DPO + synthetic data

Goal

Take an SFT model (your Project 3 LoRA, or any open instruct model) and improve it with Direct Preference Optimisation using synthetic preference data generated by a stronger judge model. Document the full pipeline so a hiring manager sees you can run a complete post-training loop.

This is the project that proves you can align a model — the highest-leverage single skill in modern LLM engineering.

Time: 2-3 weeks part-time.

Prerequisites

• 04_posttraining/03_dpo_orpo_kto.md
• 04_posttraining/04_rlaif_constitutional.md
• Project 3 finished (you'll improve that model, or start from a public instruct model).

Tech stack

• trl.DPOTrainer, peft, transformers
• vllm for batch sampling
• distilabel (optional) for the data pipeline
• gpt-4o-mini or claude-haiku-4-5 as judge
• lm-eval-harness, alpaca_eval

End-to-end pipeline

flowchart LR
    PROMPTS[1k diverse prompts] -->|sample 4 cands @ varying T| GEN[Candidate generations]
    GEN --> JUDGE[LLM judge: rank A vs B]
    JUDGE --> PAIRS[(chosen, rejected) pairs]
    PAIRS --> DPO[DPOTrainer LoRA]
    DPO --> CHECK[eval suite]
    CHECK --> SHIP[adapter on HF]

Step-by-step

1. Prompt set

Build 1k-2k diverse prompts:

• 30% domain-specific (your fine-tune target).
• 30% general chat / instructions.
• 20% safety-adjacent (XSTest-style benign).
• 10% adversarial (jailbreak / off-topic).
• 10% multi-turn fragments.

Sources: Magpie extractions from your model, OpenHermes subsets, your own collected user prompts.

2. Candidate generation

For each prompt, sample 4 candidates with vLLM at T = [0.3, 0.7, 1.0, 1.3]. ~5-10 seconds per prompt; parallelise with --num-prompts and OpenAI-style API calls.

Save as prompt, candidates[4].

3. Judging

For each pair (A, B) (6 pairs per prompt × 2 orientations = 12 calls/prompt), ask the judge:

You are evaluating two assistant responses to the same user prompt.
Score each on:
- Helpfulness (does it solve the task?)
- Accuracy (any factual errors?)
- Format/Brevity (appropriate length and structure?)
- Safety (refuses if harmful, helps if benign).

Then pick the better response.

Output JSON: {"better": "A"|"B"|"tie", "reasons": "...".}

Cost: ~$5-30 per 1k prompts (using gpt-4o-mini or claude-haiku-4-5).

Important controls:

• Position-randomise (50% A first / 50% B first).
• Drop pairs where the judge says "tie."
• Keep only pairs with consistent preference across both orientations.

4. Aggregate to (chosen, rejected) pairs

For each prompt, pick the most-preferred candidate as chosen and the least-preferred as rejected. Yields ~1k high-quality pairs from 1k prompts (one pair per prompt).

(Aggressive variant: keep all robust pairs; ~3-5k pairs per 1k prompts.)

5. DPO training

Per Lesson 4.3:

DPOConfig(
    beta=0.1, learning_rate=5e-7,
    num_train_epochs=1,
    per_device_train_batch_size=2, gradient_accumulation_steps=8,
    bf16=True, max_length=2048, max_prompt_length=1024,
    loss_type="sigmoid",   # try "ipo" and "simpo" too
)

Track in WandB: rewards/chosen, rewards/rejected, loss, kl, mean_length.

6. Evaluation

A. Capability (no regressions)

lm-eval-harness: MMLU, ARC, GSM8K, MBPP. Within 1-2% of pre-DPO.

B. AlpacaEval 2 / MT-Bench

• AlpacaEval 2 LC win-rate vs gpt-4-turbo (use alpaca_eval).
• MT-Bench score (use the official judge prompt with gpt-4o).

Compare to pre-DPO and to your base instruct model.

C. Safety

• XSTest over-refusal.
• AdvBench refusal rate.
• Garak quick scan.

Show that DPO did not make the model over-cautious or unsafe.

D. Length-bias check

• Average response length pre vs post DPO.
• Run a length-controlled win-rate ("LC" mode in AlpacaEval) to confirm gains are not just from being longer.

7. Optional online iteration

Re-generate candidates with the DPO model and judge again. Train another short DPO step. This "online DPO" closes the gap to PPO-quality results.

Acceptance criteria

• Synthetic preference dataset on HF Hub (with dataset card detailing prompts, judges, biases).
• DPO LoRA adapter on HF Hub.
• Eval table: capability, AlpacaEval LC, MT-Bench, XSTest, AdvBench, length stats.
• At least +5% AlpacaEval LC win-rate vs pre-DPO.
• No safety regression (refusal rate within 2% of pre-DPO).
• README walks through the full pipeline; make data && make train && make eval reproducible.
• Cost analysis: $ spent on data, $ on training.

Stretch goals

• Compare DPO vs SimPO vs ORPO vs KTO at fixed compute.
• Add rule-based signals (code passes, math correct) for a subset of prompts.
• Try online DPO with two iterations.
• Replace LLM judge with a small trained reward model for cheaper at-scale judging.
• Use a panel of 3 judges (e.g., GPT-4o, Claude, Llama-3 70B) and majority vote.

Common pitfalls

• LR too high (use ~5e-7 to 5e-6).
• Pairs are not robustly preferred → noisy signal.
• Forgetting position randomisation → judge bias dominates.
• Training too many epochs → catastrophic forgetting.
• Length explodes; mitigate with SimPO or length-norm.
• Skipping safety eval → "happier" model that is also more compliant with harmful asks.

Story / portfolio

• Title: "Aligning an open LLM with $20 of synthetic preferences."
• Pipeline diagram (mermaid).
• Win-rate plot before vs after.
• Hand-picked example pairs (chosen vs rejected) showing the model's improvement.
• Honest "what we did not improve" section.

A successful Project 4 puts you in the top 5% of "I have done this end-to-end" candidates. It demonstrates SFT-> DPO -> eval competence — the exact workflow open-model labs run.