RLAIF and Constitutional AI

Why this matters

Human preference data is expensive: $5-50 per pair, slow to collect, hard to scale, inconsistent. RLAIF (RL from AI Feedback) and Constitutional AI (CAI) replace human preferences with carefully prompted LLM judges. This is how Anthropic trains Claude, how the open-source community produces preference datasets like UltraFeedback, and how synthetic-data pipelines now generate millions of pairs in a weekend.

Knowing this material is important because: (1) you almost certainly will use AI feedback in your projects, and (2) it raises subtle questions of bias propagation and alignment that interviewers love.

Learning objectives

1. Understand RLAIF as a drop-in for RLHF.
2. Describe Anthropic's Constitutional AI two-step pipeline.
3. Build a small RLAIF data pipeline using GPT-4o or Claude as the judge.
4. Reason about the strengths and risks of LLM-as-judge.
5. Combine self-instruct and CAI for fully synthetic post-training.

1. RLAIF — RL from AI Feedback (Lee et al., 2023)

The recipe is identical to RLHF except the labeller is another LLM instead of a human.

Pipeline

1. Generate K candidate completions for each prompt using your SFT model.
2. For each pair, ask a strong "judge" model (Claude / GPT-4o / Llama-3 70B) which is preferred.
3. Use those pairs in DPO/PPO/SimPO exactly as if they were human pairs.

A judge prompt template (works in 2026)

You will be shown a user prompt and two assistant responses. Decide which response
is more helpful, accurate, and follows the user's instructions. Respond ONLY with
"A" or "B".

Prompt:
{prompt}

Response A:
{a}

Response B:
{b}

Answer:

For better calibration:

• Position-randomise A and B (judges have a position bias).
• Ask the judge to think step-by-step before answering ("CoT-judge").
• Use a panel of 3 judges and majority vote.
• For verifiability tasks (math, code) use rule-based judges instead.

Quality of LLM judgments

Empirically, GPT-4-class judges agree with human judges ~80% of the time — comparable to inter-annotator agreement. They are biased toward:

• Length (longer = better, all else equal).
• Their own writing style.
• Refusals on borderline content.
• Format markers (bullet points, headings).

You must mitigate or your fine-tune will inherit those biases.

2. Constitutional AI (Bai et al., 2022)

Anthropic's signature alignment technique. Two phases:

Phase 1 — Critique & Revise (CAI-SFT)

1. Generate a response to a potentially harmful prompt.
2. Ask the model: "Critique your response according to this principle: [principle]. Identify ways it might violate the principle."
3. Ask: "Now revise the response to remove those violations."
4. Use the revised response as SFT training data.

The model effectively self-supervises against a written constitution (a list of principles like "be helpful," "avoid bias," "do not provide instructions for harm").

Phase 2 — RLAIF on the constitution

After CAI-SFT, run preference optimisation where the judge LLM evaluates pairs against the same constitution. The result is a model trained to align with the constitution without per-example human labels.

Why it works

• Scales: you can generate millions of (harmful_prompt, harmless_response) pairs from one human-written principle.
• Auditable: the constitution is a transparent artefact, unlike a learned RM.
• Steerable: change the constitution, retrain, get a model with different values.

Key principles you might write

1. Be helpful, harmless, and honest.
2. Refuse to assist with illegal activity, weapons of mass destruction,
   or self-harm enabling.
3. Avoid bias or stereotyping; respect diverse perspectives.
4. Acknowledge uncertainty rather than fabricating facts.
5. Follow user instructions when they conflict with no other principle.
6. Be polite and concise.

Anthropic's published Claude constitution is much longer and includes references to UN Declarations, Apple privacy policy, etc.

3. Self-Instruct — generating prompts (Wang et al., 2022)

The other half of synthetic post-training: where do the prompts come from?

Self-Instruct: seed an LLM with ~150 hand-written tasks, then iteratively ask it to:

1. Generate new tasks similar in style.
2. Generate inputs for those tasks.
3. Generate outputs.

You end up with ~50k diverse instruction examples. Quality-filter, dedup, and you have an SFT corpus.

This is how Alpaca (52k examples) was built in 2023, and how every modern open SFT mixture is augmented today.

Modern variants

• Evol-Instruct (WizardLM) — repeatedly mutate prompts to make them harder ("rewrite this to require more reasoning steps"). Used by WizardLM-2, Microsoft Wizard-Math.
• Magpie (2024) — exploit the chat template: prompt the model with only the user-turn marker; whatever it generates is a free instruction. Scales massively.
• Distilabel — Argilla's framework for data-pipeline synthesis (steps: generate, judge, ranking).
• Reflexion-style critique — generate, critique, revise; use the revised version.

4. The fully synthetic recipe (Tülu 3 / Phi style)

[base LM]
   │
   │ Synthetic SFT
   │
   │   1. Self-Instruct / Magpie generate 100k-1M prompts
   │   2. Strong LLM (GPT-4o / Claude / Llama-3 70B) generates responses
   │   3. Filter low-quality / unsafe ones
   │   4. SFT on the result
   ▼
[SFT model]
   │
   │ Synthetic DPO / RLAIF
   │
   │   1. Generate 4 candidates per prompt with the SFT model at high T
   │   2. Have a panel of judges rank
   │   3. DPO/SimPO on (chosen, rejected)
   ▼
[DPO model]
   │
   │ (Optional) CAI revision / safety pass
   │
   ▼
[Aligned model]

This is, in 2026, the dominant open-source recipe. Llama-3-Instruct, Qwen-2.5-Instruct, Mistral-Instruct, Gemma-2-Instruct, Phi-3.5, Tülu-3 — all use heavy synthetic post-training.

5. Risks and mitigations

Bias amplification

The judge bakes its own biases into your model. Use diverse judges, swap positions, and run held-out human evals.

Mode collapse

Too much synthetic data of a single style → model becomes monotone (e.g., always uses bullet points, always opens with "Certainly!"). Mix in real conversations.

Refusal over-fit

LLM judges over-reward "safe" refusals. Models trained on this become overly cautious. Mitigation: explicitly add prompts where helpfulness > caution and reward direct answers.

Capability laundering

A small open model trained only on GPT-4o outputs can score artificially well on benchmarks judged by GPT-4o. Always evaluate on independent benchmarks too.

6. Distilabel — the practical framework

from distilabel.pipeline import Pipeline
from distilabel.steps.tasks import TextGeneration, UltraFeedback
from distilabel.llms import OpenAILLM, vLLM

with Pipeline(name="rlaif-pairs") as p:
    gen = TextGeneration(llm=vLLM(model="my-sft"), num_generations=4)
    judge = UltraFeedback(llm=OpenAILLM(model="gpt-4o-mini"))
    gen >> judge

dataset = p.run(parameters={"input_batch_size": 8, "max_concurrency": 32})
dataset.push_to_hub("you/my-rlaif-pairs")

That's a complete RLAIF data pipeline. ~$200 buys you a meaningful preference dataset.

Hands-on lab (4 hours)

rlaif_lab.ipynb:

1. Build a 1k-prompt seed list (mix coding, writing, math, common-sense).
2. For each, generate 4 candidates with gpt-4o-mini at temperatures [0.3, 0.7, 1.0, 1.3].
3. Have gpt-4o rank them with a CoT-judge prompt. Save the top vs bottom as DPO pairs.
4. Evaluate position bias: run with (A, B) and (B, A); what % flip? Mitigate.
5. Run a small DPO using these pairs (your SFT base).
6. Bonus: implement a CAI critique-and-revise step on a list of harmful-style prompts and add the revised pairs.

Common pitfalls

1. Cheap-judge mismatch — using gpt-3.5-turbo style judges; they introduce more bias. Use the strongest judge you can afford.
2. Forgetting to shuffle position when judging A vs B.
3. Not deduplicating synthetic prompts → model overfits to a tiny prompt distribution.
4. Mixing CAI safety data with too many "refusal" examples → over-cautious model.
5. Treating LLM-judge benchmarks as ground truth — always run a few human evals.

Self-check

1. Difference between RLHF and RLAIF in one sentence each.
2. What are the two phases of Constitutional AI?
3. What does Self-Instruct generate?
4. Name three known biases of LLM-as-judge.
5. Why might a model trained purely on synthetic data score well on benchmarks but feel "off" in real use?

References

• Bai et al. (2022), "Constitutional AI: Harmlessness from AI Feedback."
• Lee et al. (2023), "RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback."
• Wang et al. (2022), "Self-Instruct: Aligning Language Models with Self-Generated Instructions."
• Xu et al. (2024), "Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing."
• Cui et al. (2023), "UltraFeedback: Boosting Language Models with High-quality Feedback."
• Argilla (2024), "Distilabel."
• Anthropic, "Claude's Constitution."