RLHF (the InstructGPT recipe)

Why this matters

SFT teaches the model what to say. RLHF teaches the model what humans prefer. RLHF — Reinforcement Learning from Human Feedback — is the technique that made ChatGPT feel "polished" rather than just "instructed." Every major closed model (GPT-4o, Claude, Gemini) and most strong open models go through some preference-optimisation step.

This lesson explains the original RLHF recipe (InstructGPT, 2022) using PPO. The next lesson covers DPO/ORPO/KTO — newer, simpler alternatives that are now more common. You should understand the original even if you never run it; it is the conceptual ancestor of every preference method.

Learning objectives

1. Explain the 3-stage RLHF pipeline (SFT → reward model → PPO).
2. Train a small reward model from preference data.
3. Understand the PPO loss in the language-model setting.
4. Reason about KL constraints and reward hacking.
5. Recognise why DPO largely replaced PPO for open-source post-training.

1. The InstructGPT pipeline (2022)

[base LM]
   │
   ▼  Stage 1: SFT
[SFT model]
   │
   ▼  Stage 2: Reward model — train on (prompt, chosen, rejected) pairs
[Reward model RM]
   │
   ▼  Stage 3: PPO — RL fine-tune SFT policy with reward = RM(prompt, response) - β KL(policy || ref)
[RLHF model — your chat assistant]

This pipeline produced ChatGPT in 2022. Every later technique tweaks Stage 3.

2. Stage 2 — training a reward model

Data format

Each example is a pair of completions for the same prompt with a human-judged preference:

{"prompt": "Write a haiku about coffee.",
 "chosen":  "Mug warm in my hand / steam rises like whispered words / morning has begun.",
 "rejected": "Coffee good. Drink coffee. Yes."}

Public datasets: Anthropic/hh-rlhf, HuggingFaceH4/ultrafeedback, openbmb/UltraFeedback, lmsys/chatbot_arena_conversations.

Architecture

The reward model is the base or SFT model with a scalar head replacing the LM head:

class RewardModel(nn.Module):
    def __init__(self, base):
        super().__init__()
        self.body = base
        self.score = nn.Linear(base.config.hidden_size, 1, bias=False)
    def forward(self, ids, mask):
        h = self.body(ids, attention_mask=mask).last_hidden_state[:, -1, :]
        return self.score(h).squeeze(-1)

Loss — Bradley-Terry preference

P(chosen > rejected) = σ(r_chosen - r_rejected)
loss = - log σ(r_chosen - r_rejected)

Train to maximise this likelihood. After ~1 epoch on 100k pairs, you have a usable RM.

loss = -F.logsigmoid(r_chosen - r_rejected).mean()

trl.RewardTrainer does this.

3. Stage 3 — PPO

The crux: optimise the SFT policy to produce responses with high reward, without drifting too far from the SFT model (otherwise the policy will exploit the RM's flaws — "reward hacking").

The objective per token:

J(θ) = E_{x,y~π_θ} [ r(x,y) ]   -   β * KL( π_θ(y|x) || π_ref(y|x) )

π_ref is the frozen SFT model. β is the KL coefficient (~0.05-0.2).

Why PPO?

PPO (Proximal Policy Optimisation; Schulman 2017) is the workhorse RL algorithm: keeps update steps "proximal" to the current policy via a clipped ratio. Stable, off-the-shelf.

PPO loss:

ratio    = π_θ(y|x) / π_old(y|x)
clip     = min(ratio * A, clip(ratio, 1-ε, 1+ε) * A)         # ε=0.2
J        = E[clip] - β * KL

Where A is the advantage (typically reward - value baseline; we estimate the value with a critic head).

Practical components

1. Actor — your trainable policy (SFT model).
2. Critic — value head; estimates expected return for a given prefix.
3. Reward model — frozen.
4. Reference model — frozen copy of SFT for the KL term.
5. Roll-outs — sample completions from the actor.
6. Update — backprop PPO loss.

You hold four copies of a 7B model in memory. PPO is expensive.

trl.PPOTrainer minimal example

from trl import PPOConfig, PPOTrainer

cfg = PPOConfig(model_name="my-sft", learning_rate=1e-6, batch_size=64,
                mini_batch_size=8, kl_coef=0.05, init_kl_coef=0.05,
                target_kl=6.0)

trainer = PPOTrainer(cfg, policy=actor, ref_policy=ref, reward_model=rm,
                     value_model=critic, train_dataset=prompts_ds, processing_class=tok)
trainer.train()

4. The hard parts (and why DPO won)

Reward hacking

The policy finds adversarial responses the RM scores highly but humans dislike — verbose preambles ("Certainly! Here is a detailed answer..."), refusing borderline questions to be "safe," excessive politeness, sycophancy.

Mitigations:

• KL constraint (the β term).
• Better RM (more diverse data, larger RM).
• Reward shaping (combine with rule-based rewards: format, length penalty).
• Composite RMs (separate RMs for helpfulness, harmlessness).

Stability

PPO LM training is fragile. Tuning learning rate, KL coefficient, clip ε, and rollout temperature consumes most of an engineer's time.

Compute

You need 4× the model in memory and run RL roll-outs (slow autoregressive sampling) every step. RLHF on 70B is a serious infrastructure project.

This is why DPO (next lesson) — which eliminates the RM and the RL loop — exploded in 2023-2024. But:

• The biggest closed labs still use PPO (and variants like REINFORCE-style updates) at scale.
• Reasoning RL (DeepSeek-R1, o1) uses PPO/GRPO, not DPO.

5. RLHF variants you should know

• PPO — original; standard at OpenAI/Anthropic/Google for years.
• REINFORCE / RLOO (REINFORCE Leave-One-Out) — simpler; just gradient × reward. Re-emerged because PPO's clipping is overkill at language-modelling scale.
• GRPO (DeepSeek 2024) — Group Relative Policy Optimisation; estimate advantage by comparing within a group of N samples for the same prompt instead of a critic. No critic → 25% memory saving. Used in DeepSeek-R1.
• REINFORCE++ / RAFT — reject low-reward samples, fine-tune on the rest with cross-entropy. Cheap "best-of-N" SFT.
• SimPO / DPO / IPO / KTO / ORPO — preference methods that skip explicit RL (next lesson).

6. Mental model — RL vs SFT

SFT and RLHF differ in who knows the right answer:

RL becomes essential when you cannot demonstrate the right answer (hard reasoning tasks) but you can verify or rank outputs after the fact.

Hands-on lab (4 hours)

rlhf_lab.ipynb (use small models — Llama-3.2-1B is fine):

1. Train a reward model on Anthropic/hh-rlhf using trl.RewardTrainer. 1 epoch, 30k pairs. Plot accuracy on a held-out set (target: 65-70%).
2. Sanity-check the RM: feed a list of 5 candidate completions for a prompt and rank them. Discuss.
3. Run a tiny PPO on trl against your RM. Track KL divergence, reward, and mean_response_length.
4. Show the length bias: PPO with no length penalty produces verbose answers. Add a length penalty to the reward and rerun.
5. Run GRPO instead of PPO for the same model+RM. Compare wall-time and final reward.
6. Bonus: replace the RM with a rule-based reward (regex for format, exact match for short factuals) and run RL. This is the "RLVR" mode used in modern reasoning training.

Common pitfalls

1. No KL constraint → policy collapses into degenerate high-reward outputs.
2. Untrained / weak RM → RLHF amplifies the RM's biases.
3. Too high LR in PPO → instant divergence.
4. Sampling temperature = 0 during roll-outs → no exploration.
5. Reward range explosion → standardise rewards (subtract group mean, divide by std).
6. Forgetting to detach ref_policy → trainable params include the reference model. Painful debug.

Self-check

1. State the three stages of RLHF.
2. What does the KL term in the PPO loss prevent?
3. What is reward hacking? Give two mitigations.
4. Difference between PPO and GRPO?
5. Why is RLHF more expensive than SFT?

References

• Christiano et al. (2017), "Deep Reinforcement Learning from Human Preferences."
• Stiennon et al. (2020), "Learning to Summarize from Human Feedback."
• Ouyang et al. (2022), "Training Language Models to Follow Instructions" (InstructGPT).
• Schulman et al. (2017), "Proximal Policy Optimization Algorithms."
• Bai et al. (2022), "Training a Helpful and Harmless Assistant with RLHF" (Anthropic).
• Shao et al. (2024), "DeepSeekMath" — introduces GRPO.
• HuggingFace TRL PPO docs.