Reasoning RL (o1, R1, GRPO, RLVR)

Why this matters

In late 2024, OpenAI's o1 and DeepSeek's R1 redefined what an "LLM" can be. Instead of producing an answer in one shot, these models think — generate long chains of reasoning, backtrack, self-correct — before answering. The crucial recipe: RL with verifiable rewards (math correct/incorrect, code passes/fails) at massive scale, on top of a strong base model.

This lesson explains what reasoning RL is, how to build a tiny version, and why it has split LLMs into two families ("fast" chat models and "deep think" reasoning models).

Learning objectives

1. Define RL with verifiable rewards (RLVR).
2. Explain GRPO and why DeepSeek used it.
3. Understand "test-time compute" and chains of thought as policy outputs.
4. Build a tiny GRPO loop on GSM8K-style math problems.
5. Recognise the trade-offs between reasoning models and standard chat models.

1. The shift — verifiable rewards

For most tasks ("write a haiku") there is no single correct answer; you need preferences (Lesson 4.3) or human raters. But for math, code, logic puzzles, retrieval:

You can check whether the answer is right.

So you can give the model:

• +1 if the final answer matches the gold,
• 0 otherwise (or a graded partial credit),

and run RL directly. No reward model needed.

This unlocks training on hard reasoning at scale because the supervision signal is automatic.

def gsm8k_reward(generation: str, gold_number: float) -> float:
    pred = extract_final_number(generation)
    return 1.0 if pred is not None and abs(pred - gold_number) < 1e-6 else 0.0

That's it. The signal is sparse but unbiased.

2. GRPO — DeepSeek's algorithm

PPO needs a value/critic model (a second copy of the policy with a regression head). For 70B+ policies that doubles memory.

GRPO (Group Relative Policy Optimisation) removes the critic by computing the advantage within a group of N samples for the same prompt:

For each prompt x:
  Sample {y_1, y_2, ..., y_N} with the current policy.
  r_i = reward(x, y_i)
  A_i = (r_i - mean(r)) / std(r)              # group-relative advantage
loss_i = -A_i * log π_θ(y_i | x)  +  β * KL( π_θ || π_ref )

Effectively: "responses better than the average for this prompt are pushed up; those below are pushed down." Cheaper than PPO; works at scale.

DeepSeek-R1 trained on hundreds of thousands of math + coding prompts with GRPO until reasoning emerged.

3. The R1 / o1 recipe (as best understood publicly)

DeepSeek-R1-Zero (the surprise)

Start from base DeepSeek-V3-Base. Run pure GRPO with rule-based rewards on math and code. No SFT. After ~thousands of steps:

• The model spontaneously produces longer chains of reasoning.
• "Aha moments" emerge — backtracking, self-checking, multiple-method exploration.
• Performance on AIME / MATH / Codeforces approaches GPT-4-class.

DeepSeek-R1 (the polished version)

Same base, but pipeline:

1. Cold-start SFT on a small ~1k curated reasoning trace.
2. GRPO with rule-based rewards on math + code (the bulk of training).
3. RLAIF using the R1-Zero model to label preference pairs for general chat.
4. Final SFT pass to clean format.

The result is a model that thinks for ~100-10000 tokens before answering and matches o1 quality on many reasoning benchmarks.

OpenAI o1 / o3 (less detail public)

• Uses RL to learn long internal chains of thought.
• The visible "reasoning summary" is a scaled-down version of the actual hidden CoT.
• Test-time compute scales: the more tokens it thinks, the better it does.

Test-time compute scaling

Reasoning models exhibit a new scaling law: performance increases with the number of reasoning tokens generated. You can spend more inference compute (longer thinking, multiple samples + best-of-N or majority vote) to get higher accuracy.

This is why an o1 query can cost $0.10-$1.00 — the model truly does compute more.

4. Building a tiny reasoning RL — GRPO on GSM8K

You can run a minimal GRPO loop on a small model (Qwen2.5-Math-1.5B-Instruct, Llama-3.2-1B-Instruct) on a single 24 GB GPU.

# Simplified pseudo-code; use trl.GRPOTrainer for real runs
from trl import GRPOConfig, GRPOTrainer
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Qwen/Qwen2.5-Math-1.5B-Instruct"
tok = AutoTokenizer.from_pretrained(model_id)
m   = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16")

ds = load_dataset("openai/gsm8k", "main", split="train")

def reward_func(prompts, completions, **kwargs):
    rewards = []
    golds = kwargs["answer"]                 # numeric gold answers
    for c, gold in zip(completions, golds):
        pred = extract_number(c)
        rewards.append(1.0 if pred == gold else 0.0)
    return rewards

cfg = GRPOConfig(
    output_dir="out/grpo-gsm8k",
    learning_rate=5e-6,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    num_generations=8,                       # samples per prompt for group relative
    max_prompt_length=512,
    max_completion_length=1024,
    num_train_epochs=1,
    beta=0.04,                               # KL strength
    bf16=True,
)

trainer = GRPOTrainer(
    model=m, processing_class=tok, args=cfg,
    train_dataset=ds, reward_funcs=[reward_func],
)
trainer.train()

After ~1000 steps you should see GSM8K accuracy climb from ~70% to ~85%+ on the small model. Watch the average reasoning length grow — that is the model learning to think.

trl.GRPOTrainer (>=0.13) is officially supported.

5. Reward design — the real hard part

For math: correct/incorrect works.

For code: tests pass / fail (run in a sandbox).

For multi-step tasks you might combine:

• Format reward: parses correctly (e.g., <answer>...</answer>).
• Step reward: each intermediate sub-answer correct.
• Length penalty: discourage degenerate filler.
• Outcome reward: final correctness.

Frontier reasoning training combines several reward channels with a weighted sum or hierarchical schedule. Bad reward design = reward hacking (model learns to fool the format checker).

6. Reasoning models vs chat models — when each wins

In 2026 most providers offer both: a fast model and a reasoning model, with seamless mode switching (e.g., Claude's "extended thinking," GPT-4 / o3 routing).

7. Open frontier reasoning models

Public recipes / models you can study:

• DeepSeek-R1 / R1-Distill — full pipeline open; distilled into Qwen and Llama bases.
• Qwen-QwQ-32B — open reasoning model with extensive CoT.
• Llama-Nemotron (NVIDIA) — Llama-based reasoning fine-tunes.
• Open-R1 (HuggingFace) — fully open reproduction of the R1 recipe.
• rStar-Math (Microsoft) — small reasoning models on math.
• K1.5 / Kimi K2 — Moonshot's reasoning + agentic models.

Each ships a paper or blog with reproducible details.

Hands-on lab (full day)

grpo_gsm8k.ipynb:

1. Implement extract_final_number for GSM8K (#### 42 style).
2. Run GRPOTrainer on Qwen2.5-1.5B-Instruct with the reward above. 500 steps.
3. Plot mean reward, mean completion length, and KL.
4. Sample completions before and after. Show the reasoning depth grew.
5. Add a format reward ("must contain <reasoning> and <answer>"). Re-train. Observe more disciplined output.
6. Bonus: switch to a coding task — execute generated code in a sandbox and reward by tests passing.

Common pitfalls

1. No KL constraint → model learns degenerate "answer always 42" tricks.
2. Tiny num_generations → noisy advantage estimates → unstable RL.
3. Using a base that is too weak → no reasoning emerges, only memorisation.
4. Format reward too dominant → the model produces empty reasoning blocks but correct format.
5. Forgetting to evaluate on a held-out test set — train accuracy can soar via memorisation.

Self-check

1. What is RLVR?
2. How does GRPO differ from PPO?
3. What does "test-time compute scaling" mean?
4. Why do reasoning models cost 10× more at inference?
5. Why was R1-Zero so surprising?

References

• DeepSeek-AI (2025), "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning."
• OpenAI (2024), "Learning to Reason with LLMs" (o1 system card).
• Shao et al. (2024), "DeepSeekMath: Pushing the Limits of Mathematical Reasoning" (introduces GRPO).
• HuggingFace, "Open-R1."
• Wang et al. (2024), "rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking."
• Snell et al. (2024), "Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters."
• HuggingFace TRL GRPO docs.