DPO, ORPO, KTO, IPO, SimPO

Why this matters

In late 2023, Direct Preference Optimisation (DPO) showed you can do RLHF without an RL loop, without a separate reward model, with one cross-entropy-like loss. Every open-source post-training pipeline since (Tülu, Zephyr, Llama-3 Instruct, Mistral Instruct, Qwen Instruct) uses DPO or one of its descendants. Knowing them is now table stakes.

This is the lesson that turns SFT engineers into post-training engineers.

Learning objectives

Derive the DPO loss intuition.
Distinguish DPO, IPO, KTO, ORPO, SimPO — when to use each.
Run DPO on a real preference dataset with trl.DPOTrainer.
Diagnose common DPO failures (reward hacking, length bias, format collapse).
Combine SFT + DPO into a complete post-training recipe.

1. DPO — Direct Preference Optimisation (Rafailov et al., 2023)

Key insight

The optimal RLHF policy has a closed-form relationship to the reward model. By inverting that relationship, you can train directly on preference data without ever fitting an explicit RM, and without RL.

The loss (intuition)

For a preference pair (x, y_w, y_l) (chosen vs rejected):

loss_DPO = -log σ(β * (log π_θ(y_w|x) - log π_θ(y_l|x)
                     - log π_ref(y_w|x) + log π_ref(y_l|x)))

Read it as: "increase the policy's log-probability of the chosen response relative to the rejected, and relative to the reference (SFT) model's ratio."

The implicit reward is r(x,y) = β log(π_θ(y|x) / π_ref(y|x)). The policy is encouraged to produce chosen completions and discouraged from rejected ones, all while staying close to the reference.

Why it works

No RM training. The pairwise data directly trains the policy.
No rollouts / RL. Just a forward pass and a logit comparison.
Stable. It is essentially a contrastive cross-entropy.
Reuses the SFT setup. Same DataLoader, same trainer.

Implementation with `trl`

python

from trl import DPOConfig, DPOTrainer
from peft import LoraConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

base = AutoModelForCausalLM.from_pretrained("my-sft-model", torch_dtype="bfloat16",
                                            attn_implementation="flash_attention_2")
ref  = AutoModelForCausalLM.from_pretrained("my-sft-model", torch_dtype="bfloat16",
                                            attn_implementation="flash_attention_2")
tok  = AutoTokenizer.from_pretrained("my-sft-model")

ds = load_dataset("HuggingFaceH4/ultrafeedback_binarized", split="train_prefs").select(range(20_000))

cfg = DPOConfig(
    output_dir="out/dpo",
    beta=0.1,                   # KL strength; 0.05-0.3 typical
    learning_rate=5e-7,         # MUCH lower than SFT
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    num_train_epochs=1,
    warmup_ratio=0.05,
    bf16=True,
    max_length=2048,
    max_prompt_length=1024,
    loss_type="sigmoid",        # default DPO; "ipo" for IPO; "kto_pair" for KTO
)

peft = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                  target_modules=["q_proj","k_proj","v_proj","o_proj",
                                  "gate_proj","up_proj","down_proj"],
                  task_type="CAUSAL_LM")

trainer = DPOTrainer(model=base, ref_model=ref, args=cfg, peft_config=peft,
                     train_dataset=ds, processing_class=tok)
trainer.train()

DPO is essentially "another SFT job," memory-wise; you just hold the reference copy in addition.

Hyperparameters

beta = 0.05 – 0.3. Higher β = stay closer to reference; lower β = more aggressive preference learning.
learning_rate ≈ 5e-7 to 5e-6 (much lower than SFT). DPO can quickly destroy SFT capabilities.
epochs = 1-2 typically.
LoRA r = 16-64 works well.

2. The DPO failure modes (and fixes)

DPO is not magic — it has its own pathologies.

Length bias

DPO often makes the model longer because longer answers usually win in preference data. Mitigations:

Length-normalise in the loss (loss / response_length).
Use SimPO (next).

"Out-of-distribution" reward hacking

The model can drift to outputs unlike anything in SFT, where neither chosen nor rejected was sampled. The implicit reward there is unreliable.

Fix: keep beta reasonably high; iterate online DPO (regenerate preference pairs from the current model, not the SFT model).

Capability regression

DPO sometimes breaks math/code skills the SFT had. The "alignment tax."

Fix: include capability-preserving prefs (chosen = correct math, rejected = wrong) and run mix DPO + SFT loss (NCA / RPO objectives).

3. The DPO family — what each fixes

IPO — Identity Preference Optimisation (Azar et al., 2023)

Replaces the sigmoid+log with a squared loss on the implicit-reward gap. Less prone to overfitting when many preference pairs have near-equal preferences. Set loss_type="ipo" in DPOConfig.

KTO — Kahneman-Tversky Optimisation (Ethayarajh et al., 2024)

Trains on single-sided signals (just "this output is good" or "this is bad"), not pairs. Fits real-world data where you have feedback on individual outputs but no pairwise judgements. loss_type="kto_pair" (uses pairs) or use KTOTrainer for unpaired data.

ORPO — Odd-Ratio Preference Optimisation (Hong et al., 2024)

Combines SFT loss + a preference odds-ratio term. Single-stage: skip the SFT step entirely; ORPO does both at once. Surprisingly effective; used in some 2024 open releases (e.g., Llama-3-OpenORPO).

python

from trl import ORPOTrainer, ORPOConfig
cfg = ORPOConfig(output_dir="out/orpo", beta=0.1, learning_rate=5e-6,
                 max_length=2048, num_train_epochs=2, bf16=True)
trainer = ORPOTrainer(model=base, args=cfg, processing_class=tok, train_dataset=ds)

SimPO — Simple Preference Optimisation (Meng et al., 2024)

Drops the reference model entirely. Length-normalises the implicit reward. Often outperforms DPO with simpler infrastructure.

r_simpo(x, y) = (β/|y|) sum_t log π(y_t | x, y_<t)  -  γ
loss = -log σ(r(y_w) - r(y_l))

NCA / DRO / RSO / Step-DPO / R-DPO

A continually expanding zoo. The 2025 state of the art is generally DPO/IPO/SimPO with online generation + capability mixing.

Online DPO

Iteratively: generate completions with the current model → judge with a strong RM (or LLM) → train DPO step → repeat. Essentially "soft RL" without the RL infrastructure.

trl.OnlineDPOTrainer is the official implementation.

4. The complete 2026 post-training recipe

The Tülu-3 / Zephyr / Llama-3-Instruct-style pipeline:

Base LM
   │
   │ 1. SFT on 100k-1M instructions (Tülu mix, OpenHermes, internal)
   ▼
SFT model
   │
   │ 2. DPO / SimPO / ORPO on 100k preference pairs
   │    (UltraFeedback or your own preference data)
   ▼
DPO model
   │
   │ 3. (Optional) Online DPO with self-generated pairs judged by GPT-4o
   ▼
Final aligned model
   │
   │ 4. (Optional) RLVR on math/code with rule-based rewards
   ▼
Reasoning-enhanced model

Each stage adds a few % on benchmarks. Combined with good SFT data, this matches closed-source quality on chat tasks.

5. Building preference data when you don't have it

Three practical paths:

A. Use public sets

HuggingFaceH4/ultrafeedback_binarized — 60k high-quality pairs.
argilla/distilabel-intel-orca-dpo-pairs — Orca-style.
Anthropic/hh-rlhf — older but classic.

B. LLM-as-judge

Take a list of prompts.
Generate 4 candidates from your SFT model (varied temperatures).
Have GPT-4o or Claude pick the best and worst.
Use that as (chosen, rejected).

C. Rule-based / executable signal

For code: chosen = test-passing solution, rejected = test-failing.
For math: chosen = numerically correct, rejected = incorrect.
For format-following: chosen = parses, rejected = does not parse.

This is the cheapest, scalable preference source — and the one closed labs lean on most.

Hands-on lab (4 hours)

dpo_lab.ipynb:

Take your SFT model from Lesson 4.1. Run DPO with loss_type="sigmoid" on ultrafeedback_binarized (10k subset). Track rewards/chosen, rewards/rejected, and loss.
Compare base vs SFT vs DPO on 30 prompts using LLM-as-judge.
Try loss_type="ipo" and compare. Discuss which wins on your prompts.
Implement length-normalised DPO manually and verify length distribution improves.
Generate your own preference pairs: ask gpt-4o-mini to compare 4 candidate responses you generate. Use those for DPO.
Bonus: run ORPO starting from the base (skip SFT). Compare end quality.

Common pitfalls

Same learning_rate as SFT in DPO → catastrophic. Drop ~10×.
Wrong beta — too low, the model drifts; too high, it learns nothing.
Skipping SFT before DPO when your base is not already instruct-tuned — DPO struggles.
Forgetting attn_implementation="flash_attention_2" → painfully slow.
Mismatched chat templates between dataset and tokenizer.
Using peft with ref_model=None (auto-builds the ref from the merged base) when your LoRA target modules are wrong → ref_model shares trainable params. Print parameter counts to check.

Self-check

What does DPO eliminate from the RLHF pipeline?
What does the beta parameter control?
When would you choose SimPO over DPO?
How does ORPO collapse SFT and DPO?
Name three sources of preference data when you do not have human labels.

References

Rafailov et al. (2023), "Direct Preference Optimization: Your Language Model is Secretly a Reward Model."
Azar et al. (2023), "A General Theoretical Paradigm to Understand Learning from Human Preferences" (IPO).
Ethayarajh et al. (2024), "Model Alignment as Prospect Theoretic Optimization" (KTO).
Hong et al. (2024), "ORPO: Monolithic Preference Optimization without Reference Model."
Meng et al. (2024), "SimPO: Simple Preference Optimization with a Reference-Free Reward."
Tunstall et al. (2023), "Zephyr: Direct Distillation of LM Alignment."
AI2 (2024), "Tülu 3 Technical Report."
HuggingFace TRL DPO docs.