Positional encodings (sinusoidal → RoPE → YaRN)

Why this matters

Attention is permutation invariant by itself — it sees a set of tokens, not a sequence. To make a Transformer language model, we have to inject position information somehow. The technique chosen has enormous downstream effects:

• Whether a model can generalise to longer contexts than it was trained on.
• Whether you can extend context from 8k to 1M with a few lines of code.
• Whether long-context retrieval works.

Modern LLMs almost universally use RoPE (Rotary Positional Embeddings) plus stretching tricks (NTK-aware, YaRN, longRoPE) to scale context. This lesson teaches them all.

Learning objectives

1. Distinguish absolute, relative, and rotary positional encodings.
2. Implement RoPE in ~20 lines of PyTorch.
3. Explain why RoPE generalises better than sinusoidal absolute encodings.
4. Apply NTK-aware / YaRN scaling to extend a model's context length.
5. Reason about the failure modes ("lost in the middle," needle-in-a-haystack).

1. Why we need positional encodings

Recall scaled dot-product attention:

attention(Q, K, V) = softmax(Q K^T / sqrt(d_k)) V

If we permute Q, K, V along the time axis, the output permutes the same way — attention itself does not "know" position. We must:

1. Add something position-dependent to the embeddings, OR
2. Bias the attention scores with position, OR
3. Rotate Q and K by a position-dependent rotation.

Option 1 is the original (sinusoidal/learned). Option 2 is ALiBi / T5 relative bias. Option 3 is RoPE — and is what you will see in 95% of modern code.

2. Sinusoidal absolute (Vaswani 2017)

PE(pos, 2i)   = sin(pos / 10000^(2i/d))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d))

Add this to the token embedding before layer 1.

Pros: deterministic; can extrapolate slightly past training length. Cons: absolute — token "the" at position 5 has different vector from "the" at position 6 in every layer; bad for semantic similarity. Falls apart past ~2× training length.

3. Learned absolute (GPT-2, GPT-3)

A learned embedding for each position 0..T_max. Simple, works, but cannot go past T_max — your max context is locked at training time.

GPT-2 has 1024 learned positions. Cannot extend (without retraining).

4. ALiBi (Press 2022)

Attention with Linear Biases. No positional encoding added to embeddings; instead, subtract a position-dependent penalty from attention scores:

scores[i, j] -= m_h * (i - j)        # per-head slope m_h, fixed

Each head has its own slope; far-away keys get penalised. Works surprisingly well; allows context length extrapolation.

Used by BLOOM and earlier MPT; less common today (RoPE won mindshare).

5. RoPE — Rotary Positional Embeddings (Su et al., 2021)

The dominant choice in 2024-2026.

The intuition

Treat each pair of dimensions in Q and K as a 2D vector. Rotate that vector by an angle that depends on the position:

θ_i = 10000^(-2i/d)                             # base frequencies
R_θ(pos) = block-diagonal 2x2 rotation matrices, where block i rotates by pos * θ_i

Q' = R_θ(pos_q) Q
K' = R_θ(pos_k) K

Then the inner product Q' . K' depends only on the difference pos_q - pos_k. So RoPE is a clever way to inject relative position information into attention while keeping the form softmax(Q K^T).

In code (Llama-style)

import torch
def precompute_freqs(dim, end, theta=10000.0):
    freqs = 1.0 / (theta ** (torch.arange(0, dim, 2).float() / dim))   # (dim/2,)
    t = torch.arange(end).float()
    freqs = torch.outer(t, freqs)                                       # (end, dim/2)
    cos, sin = freqs.cos(), freqs.sin()
    return cos, sin

def apply_rope(x, cos, sin):
    # x: (B, h, T, d_head); even/odd pairing
    x1, x2 = x[..., 0::2], x[..., 1::2]
    rotated = torch.stack([x1 * cos - x2 * sin,
                           x1 * sin + x2 * cos], dim=-1)
    return rotated.flatten(-2)

You apply RoPE to Q and K inside attention, just before the dot product. V is left unrotated — only the score depends on position.

Why it generalises

Because the rotation angle scales linearly with position, RoPE has a clean frequency structure. Low-dimensional pairs rotate slowly (capture long-range info); high-dimensional pairs rotate quickly (capture local info). Like Fourier features — well-understood mathematically.

6. Extending RoPE — NTK-aware, YaRN, longRoPE

A model trained at T_max=8192 will struggle past that. But because RoPE is just rotation by pos * θ_i, you can change θ_i at inference and recover (most) quality at much longer contexts. Three increasingly sophisticated tricks:

Position interpolation (PI) — Chen et al., 2023

Pretend the new context is the same as old by scaling positions: pos_new = pos / scale. Effectively divides each rotation angle by scale. Cheap, simple — but loses high-frequency detail.

NTK-aware scaling (bloc97, 2023)

Apply a non-uniform base scaling: leave high-frequency dims alone (preserve local detail), spread low-frequency dims (extend long-range). Better quality.

def ntk_scale_theta(theta, scale, dim):
    return theta * scale ** (dim / (dim - 2))

YaRN — Peng et al., 2023

Best of both: piecewise interpolation that combines NTK-aware and linear, with attention-temperature compensation. Used in Mistral 7B 32k, Yi 200k, Qwen long-context. Extends 4k → 128k+ with a few hundred steps of fine-tuning.

longRoPE / longRoPE-2 — Microsoft 2024

Searches for per-dimension rescale factors. Pushes Phi-3 and others to 2M tokens with quality.

How to use this in HF

from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    rope_scaling={"type": "yarn", "factor": 4.0, "original_max_position_embeddings": 8192},
)

7. The "lost in the middle" problem

Even with long context, models often perform worst on info in the middle of the context (Liu et al., 2023). Empirically, attention concentrates on the start and end. Mitigations:

• Long-context fine-tuning on retrieval-style tasks.
• Needle-in-a-haystack evals (insert a fact in the middle, ask about it).
• Architectural fixes (MoE-attn, attention sinks) and post-training tricks (better data mix).

You will hear "long-context-pretrained" vs "long-context-extended" — the former is much better. Llama-4, Gemini 2.5 Pro, Claude 4.x are pretrained on long sequences from the start.

8. ALiBi vs RoPE — when to choose what

In 2026, defaults are RoPE + sliding window or full attention; ALiBi is a curiosity.

Hands-on lab (3 hours)

rope_lab.ipynb:

1. Implement precompute_freqs and apply_rope from scratch.
2. Verify rotational property: (R_θ(p) Q) . (R_θ(p+k) K) = Q . (R_θ(k) K) (relative-position).
3. Add RoPE to your MultiHeadAttention from Lesson 2.2; train a tiny LM on tiny_shakespeare. Compare PPL with no positional encoding (should be terrible) and with sinusoidal.
4. Load Llama-3.2-1B-Instruct from HF. Compute attention to a 16k input. Now apply YaRN scaling (factor=4) and re-evaluate the same prompt at 32k. Discuss perplexity behaviour.
5. Run a needle-in-a-haystack experiment on Qwen2.5-7B-Instruct (or any open model with ≥32k context): place "the secret code is FROG" at depth 10%, 50%, 90% of a 32k buffer; ask the model to retrieve it. Plot accuracy vs depth.
6. Bonus: implement ALiBi as a per-head slope mask and compare PPL on the toy LM.

Common pitfalls

1. Applying RoPE to V — only Q and K should be rotated.
2. Mixing even/odd pairing conventions (Llama vs RoPE original) — pick one and check decode against the reference model.
3. Forgetting to scale cos/sin to device/dtype of the input.
4. Extending context with PI without re-fine-tuning when the model needs it (PI alone hurts quality past ~2× original length).
5. Believing claims of "1M context" without running needle-in-haystack — many models advertise context they cannot actually use.

Self-check

1. Why is attention permutation invariant without positional encodings?
2. RoPE encodes absolute or relative position?
3. Why is RoPE preferred over learned absolute encodings?
4. What is YaRN and when do you need it?
5. What is the "lost in the middle" phenomenon?

References

• Su et al. (2021), "RoFormer: Enhanced Transformer with Rotary Position Embedding."
• Press et al. (2022), "Train Short, Test Long: ALiBi."
• Chen et al. (2023), "Extending Context Window of Large Language Models via Positional Interpolation."
• Peng et al. (2023), "YaRN: Efficient Context Window Extension of Large Language Models."
• Liu et al. (2023), "Lost in the Middle: How Language Models Use Long Contexts."
• Microsoft (2024), "LongRoPE."
• EleutherAI blog on RoPE — accessible deep dive.