Supervised Fine-Tuning (SFT)

Why this matters

A pretrained base LLM is a powerful next-token predictor — but it does not "follow instructions." Ask Llama-3 base "What is the capital of France?" and you may get a quiz, a Wikipedia paragraph, or another question; it has no concept that you want a response. Supervised fine-tuning (SFT) turns a base model into an instruction-following assistant by training on (prompt, response) pairs in a chat format.

SFT is the single most important post-training step. Every chat model — GPT-4o, Claude, Gemini, Llama-3 Instruct — starts here. You will use SFT in nearly every fine-tuning project.

Learning objectives

1. Format a chat dataset correctly with chat templates.
2. Mask the loss so we only train on assistant turns.
3. Use HuggingFace trl.SFTTrainer and peft LoRA.
4. Run a real SFT pipeline end-to-end.
5. Evaluate the resulting model.

1. The SFT objective in one line

Same loss as pretraining (causal LM), but only on the assistant's tokens.

That is it. Mathematically nothing new. The art is in:

• The data (instruction quality, diversity, length distribution).
• The format (chat template, system prompt).
• The masking (don't train on the user's tokens — the model already speaks like a user).

2. Chat templates

Each instruct model expects a specific conversation format. Use the model's built-in tokenizer.apply_chat_template — never hand-roll it.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
messages = [
    {"role": "system",    "content": "You are a helpful assistant."},
    {"role": "user",      "content": "What is the capital of France?"},
    {"role": "assistant", "content": "Paris."},
]
chat_text = tok.apply_chat_template(messages, tokenize=False)
print(chat_text)

Common templates you'll see:

# Llama-3 / 3.1
<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>

What is the capital of France?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Paris.<|eot_id|>

# ChatML (Qwen, OpenAI)
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What is the capital of France?<|im_end|>
<|im_start|>assistant
Paris.<|im_end|>

Pick the right tokenizer and trust its template.

3. Loss masking — the must-do

If you train on the whole conversation, the model learns to imitate the user as well as the assistant. Always mask everything that is not the assistant's content:

# Make assistant token positions get a real label; the rest get -100 (ignore)
def make_labels(input_ids, assistant_mask):
    labels = input_ids.clone()
    labels[~assistant_mask] = -100
    return labels

F.cross_entropy(..., ignore_index=-100) skips those positions.

trl.SFTTrainer does this for you when you set assistant_only_loss=True.

4. Datasets — what good SFT data looks like

Public starter datasets:

• HuggingFaceH4/ultrachat_200k — 200k cleaned ChatGPT conversations.
• teknium/OpenHermes-2.5 — 1M+ diverse instructions.
• allenai/tulu-3-sft-mixture — AI2's open SFT mixture (the open-source SOTA in early 2025).
• argilla/distilabel-intel-orca-dpo-pairs (DPO pairs, for next lesson).
• cognitivecomputations/dolphin-2.9 — uncensored instruct mix.

Quality matters more than quantity. A clean 10k examples beat a noisy 1M. Modern recipes:

5. The recipe — trl.SFTTrainer + LoRA

LoRA (Low-Rank Adapters; Hu et al. 2021) replaces full fine-tuning with a tiny low-rank update:

W' = W + α/r * A B^T,   A ∈ R^{d × r}, B ∈ R^{r × d}, r ≪ d

Train only A and B (often r=8 to 64). Memory footprint drops 10-100×, and quality is usually within 1-2% of full FT.

QLoRA = quantize the base model to 4-bit + LoRA on top — fits a 70B model on a single H100.

Working SFT script

import torch
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model
from trl import SFTConfig, SFTTrainer

model_id = "meta-llama/Llama-3.1-8B-Instruct"
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype=torch.bfloat16,
                         bnb_4bit_use_double_quant=True)

tok = AutoTokenizer.from_pretrained(model_id)
tok.pad_token = tok.eos_token

base = AutoModelForCausalLM.from_pretrained(
    model_id, quantization_config=bnb, device_map="auto", torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2"
)

peft_cfg = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05, bias="none",
    target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"],
    task_type="CAUSAL_LM",
)

ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft").select(range(20_000))

cfg = SFTConfig(
    output_dir="out/llama3-sft",
    num_train_epochs=1,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    bf16=True,
    logging_steps=20,
    save_strategy="epoch",
    report_to="wandb",
    max_seq_length=2048,
    packing=True,                         # concatenate short examples for efficiency
    assistant_only_loss=True,             # mask user tokens
    dataset_text_field="messages",
)

trainer = SFTTrainer(model=base, args=cfg, train_dataset=ds, peft_config=peft_cfg, tokenizer=tok)
trainer.train()
trainer.model.save_pretrained("out/llama3-sft/lora")

This is the starter recipe in 2026. Memorise it.

6. Hyperparameters — what works empirically

For LoRA SFT on a 7-13B base:

• learning_rate: 1e-4 to 2e-4 (LoRA tolerates higher LR than full FT).
• epochs: 1 (large datasets) to 3 (small datasets).
• r: 16-64; bigger r ≈ closer to full FT.
• lora_alpha: typically 2 × r.
• target_modules: all linear layers in attention + MLP (above).
• gradient_accumulation: aim for an effective batch ≥ 64-128 examples.
• weight_decay: 0 (LoRA already regularises).
• warmup_ratio: 0.03.
• bf16: yes.

For full FT on a 7B (1× H100, ZeRO-3 or FSDP):

• learning_rate: 5e-6 to 2e-5 (much lower).
• Otherwise similar.

7. Packing — the throughput win

Real instructions are 50-3000 tokens. Padding short examples wastes compute.

Packing concatenates multiple examples up to max_seq_length, separated by EOS, with proper attention masks. SFTTrainer(packing=True) enables it. ~3-5× throughput on small instructions.

8. Evaluating an SFT'd model

Quick:

• Sample 50 prompts spanning use cases. Eyeball outputs vs base.
• LLM-judge (GPT-4o or Claude) win-rate vs base.
• Run MT-Bench or AlpacaEval 2 (LLM judge benchmarks).

Robust:

• A held-out test set of (prompt, gold) and a custom judge prompt.
• Run targeted evals (math, code, instruction-following, refusal).
• Capture regressions on capabilities the base had.

We will go deep on this in Phase 7. For now, "spot-check 50 prompts and run AlpacaEval" gets you 80% of the value.

Hands-on lab (4 hours, GPU recommended)

sft_lab.ipynb:

1. Run the script above on Llama-3.2-1B-Instruct (smaller GPU friendly) with 1k Ultrachat examples.
2. Compare to the base model on 20 hand-picked prompts.
3. Now turn off assistant_only_loss. Retrain. Show qualitatively that the model now sometimes "completes" user turns.
4. Increase LoRA r from 16 to 64. Compare quality and memory.
5. Save the LoRA adapter; merge it into the base with peft.merge_and_unload() to produce a deployable single model.
6. Bonus: build a synthetic SFT set: ask GPT-4o-mini to answer 200 of your own prompts, fine-tune on that, evaluate.

Common pitfalls

1. Wrong chat template — silently degrades quality. Always use tokenizer.apply_chat_template.
2. Training on the whole conversation — model learns to play user. Always mask.
3. Too high LR for full FT — base capabilities collapse ("alignment tax").
4. Padding without packing for short data → 5× wasted compute.
5. Forgetting pad_token — tokenizer raises errors mid-train.
6. Mixing chat templates (system / user different across rows) — tokenizer breaks.

Self-check

1. What part of the conversation is the loss computed on?
2. What is LoRA and why is it used?
3. Why does QLoRA fit a 70B model on one GPU?
4. What is data packing and what does it speed up?
5. Why would you fine-tune at LR=1e-4 with LoRA but 1e-5 with full FT?

References

• Wei et al. (2022), "Finetuned Language Models Are Zero-Shot Learners" (FLAN).
• Ouyang et al. (2022), "Training Language Models to Follow Instructions" (InstructGPT — the SFT + RLHF paper).
• Hu et al. (2021), "LoRA: Low-Rank Adaptation of Large Language Models."
• Dettmers et al. (2023), "QLoRA: Efficient Finetuning of Quantized LLMs."
• AI2 (2024), "Tülu 3: Pushing Frontiers in Open Language Model Post-Training."
• HuggingFace trl SFT docs.
• HuggingFace PEFT docs.