Fine-tuning open models with LoRA, QLoRA, and DPO

Why this matters

Fine-tuning is back in vogue in 2026 because:

• Open models (Llama 3.3, Qwen 3, DeepSeek, Mistral) now match GPT-4-class quality on narrow tasks once tuned.
• LoRA / QLoRA make tuning a 70B model on a single consumer GPU feasible in hours.
• Self-hosted finetuned models on vLLM (next lesson) can be 10-50x cheaper per token at scale.

You will not fine-tune for every project. But "I shipped a LoRA-tuned 8B that replaced a 70B prompt at 1/30 the cost" is a senior badge.

Learning objectives

1. Decide when fine-tuning is the right answer.
2. Build a clean training dataset.
3. Run QLoRA with Unsloth on Llama 3.1/3.3.
4. Apply DPO (preference tuning) and know when it helps.
5. Export to GGUF for Ollama or safetensors for vLLM.

1. Should you fine-tune?

Fine-tune when:

• Prompt + RAG cannot reach quality after honest tuning.
• The task is narrow and repeatable (style, format, domain language).
• You have ≥ 500 high-quality examples (more is better).
• Self-hosting beats per-token API economics at your scale.
• Latency or privacy demands on-prem.

Do not fine-tune when:

• Frontier reasoning ability is required (use frontier API).
• You have < 100 examples (RAG + prompt almost always wins).
• The base model already passes evals (waste of money).
• You expect frequent task changes (refactor prompts, not weights).

2. Dataset hygiene

The single biggest factor in success.

• At least 500 examples for SFT, ideally 2-10k.
• High quality > volume. 1000 hand-curated > 100k noisy.
• Diversity: cover edge cases, different lengths, different intents.
• Deduplicate aggressively (exact and near dedup).
• Train/eval split 90/10 with no leakage.
• Format consistency: pick a chat template (Llama-3 chat) and stick to it.

Common formats (use datasets library):

# Conversation:
[{"role":"system","content":"You are a SQL assistant."},
 {"role":"user","content":"Top 5 cities by revenue"},
 {"role":"assistant","content":"```sql\nSELECT ...\n```"}]

Apply the chat template:

from unsloth.chat_templates import get_chat_template
tokenizer = get_chat_template(tokenizer, chat_template="llama-3.1")
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

3. QLoRA with Unsloth (the path of least pain)

Unsloth patches HuggingFace internals to be 2-5x faster and use 30-70% less VRAM. It is the de-facto consumer-GPU fine-tuning tool in 2026.

uv add unsloth bitsandbytes accelerate trl peft transformers

from unsloth import FastLanguageModel
from trl import SFTTrainer, SFTConfig

# Load 4-bit Llama 3.1 8B (or 70B if you have a beefier GPU)
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit",
    max_seq_length=2048,
    dtype=None,                          # auto: bf16 on Ampere+, fp16 older
    load_in_4bit=True,                   # QLoRA
)

# Attach LoRA adapters (the only trainable weights)
model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"],
    lora_alpha=16,
    lora_dropout=0,                       # Unsloth needs 0
    bias="none",
    use_gradient_checkpointing="unsloth",
    random_state=3407,
)

# Format dataset
from datasets import load_dataset
ds = load_dataset("json", data_files="train.jsonl", split="train")

def format_fn(rows):
    texts = []
    for msgs in rows["messages"]:
        texts.append(tokenizer.apply_chat_template(msgs, tokenize=False, add_generation_prompt=False))
    return {"text": texts}

ds = ds.map(format_fn, batched=True)

# Train
trainer = SFTTrainer(
    model=model, tokenizer=tokenizer, train_dataset=ds,
    dataset_text_field="text", max_seq_length=2048, packing=False,
    args=SFTConfig(
        output_dir="out",
        num_train_epochs=2,
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_ratio=0.03,
        learning_rate=2e-4,
        bf16=True,
        logging_steps=10,
        optim="adamw_8bit",
        weight_decay=0.01,
        lr_scheduler_type="linear",
        seed=3407,
        report_to="wandb",                # or "none"
    ),
)
trainer.train()

# Save LoRA only (small)
model.save_pretrained("lora_out"); tokenizer.save_pretrained("lora_out")

# Save merged model (optional, large)
model.save_pretrained_merged("merged_model", tokenizer, save_method="merged_16bit")

# Save GGUF for Ollama
model.save_pretrained_gguf("gguf_out", tokenizer, quantization_method="q4_k_m")

That is the entire QLoRA flow. On a free Colab T4 with Llama 3.1 8B and 1k examples it takes ~2 hours.

Key hyperparameters to know

• r (LoRA rank): 8 (cheap), 16 (default), 32 (more capacity).
• lora_alpha: usually equal to r.
• lr: 2e-4 typical for QLoRA, 1e-4 for 70B.
• num_train_epochs: 1-3. More = overfitting risk.
• max_seq_length: 2048 default; bump for long-context tasks (more VRAM).
• packing: True packs short examples for speed; False is safer for chat templates.

4. DPO — preference tuning

After SFT, DPO (Direct Preference Optimization) teaches the model what to prefer. Dataset: (prompt, chosen, rejected) triples.

from trl import DPOTrainer, DPOConfig

dpo_ds = [
    {"prompt": "...", "chosen": "...", "rejected": "..."},
    ...
]

trainer = DPOTrainer(
    model=model, tokenizer=tokenizer,
    train_dataset=dpo_ds,
    args=DPOConfig(
        output_dir="dpo_out",
        num_train_epochs=1,
        per_device_train_batch_size=1,
        gradient_accumulation_steps=8,
        learning_rate=5e-6,
        beta=0.1,
        optim="adamw_8bit",
        bf16=True,
    ),
)
trainer.train()

DPO is great when you can produce or collect preference data (your existing eval set + LLM-judged "better" answer is a fast way).

Newer alternatives — ORPO (combines SFT + DPO in one pass) and KTO (binary "good/bad" instead of pairs) — are supported in trl and worth trying for production polish.

5. Export and deploy

Three artifact options:

• LoRA only (~50MB): fastest to share, requires the base model at runtime.
• Merged safetensors: full model. Use with vLLM / TGI.
• GGUF: quantised for llama.cpp and Ollama — easy to ship to laptops.

Push merged or GGUF to HuggingFace Hub for sharing:

model.push_to_hub_merged("yourname/llama-3.1-8b-sql-v1", tokenizer, save_method="merged_16bit")

Then load with vLLM (next lesson) or:

ollama create my-sql-llm -f Modelfile

6. Evaluating your fine-tuned model

Run the same eval suite before and after:

• Domain accuracy (your golden set).
• Out-of-domain regression (general benchmarks like MMLU subset, HumanEval if coding).
• Latency / cost (vs API baseline).
• Refusal / safety (Llama Guard).

Write a "fine-tune card" — title, base model, dataset stats, hyperparams, eval results, intended use, limits, license. Anyone receiving the model artefact should be able to read this card and use it responsibly.

Hands-on lab (1-2 days)

Pick a narrow task:

• SQL generator for your own DB schema.
• Customer-support reply writer in your company's tone.
• Markdown to JSON note structurer.

1. Build a 1k-example dataset from real or synthetic data (LLM-generated, then human-reviewed).
2. SFT with Unsloth on Llama 3.1 8B (Colab Pro / a single 4090).
3. Add DPO on 200 (chosen, rejected) pairs (use GPT-4.1 to grade).
4. Eval against your golden set + general regression.
5. Export to merged safetensors + GGUF.
6. Ship a Modelfile so someone can ollama run yourname/your-model.
7. Compare cost against the Anthropic / OpenAI baseline at 100k requests/month.

This single project produces a portfolio piece almost no candidate has.

Common pitfalls

1. Bad data > tiny model issues. Spend 80% of the time on data.
2. Training too long. 2 epochs is usually enough; more = catastrophic forgetting.
3. Forgetting chat template. Different from base model → broken output.
4. No eval against the base. Always show "we improved over the unfinetuned baseline."
5. Forgetting license. Llama license has restrictions; check before commercial use.

Self-check

1. Why is QLoRA cheaper than full fine-tuning?
2. What is r in LoRA and how does it affect capacity?
3. When is DPO useful but SFT is not enough?
4. What is the difference between merged, LoRA-only, and GGUF artifacts?
5. How do you check for catastrophic forgetting?

References

• Unsloth notebooks
• TRL (SFTTrainer, DPOTrainer, ORPOTrainer, KTOTrainer)
• PEFT (LoRA library)
• QLoRA paper
• DPO paper
• Llama license
• Axolotl (alternative trainer)
• LLaMA-Factory (alternative GUI/CLI trainer)