Modern transformer variants (what changed since 2017)

Why this matters

If you read the original Transformer paper and stopped, you would build something that trains 30% slower, costs 2× more memory, and tops out at 8k tokens. Modern open-source LLMs (Llama, Mistral, Qwen, DeepSeek, Gemma, Phi) all share a standard 2024-2026 stack. This lesson is the changelog.

After this lesson you can read any frontier-model paper and instantly identify which of the standard tricks are used.

Learning objectives

1. List the standard architectural changes vs the 2017 paper.
2. Explain why each one exists.
3. Recognise model-specific quirks (Mistral's sliding window, Llama's GQA, Qwen's QK-Norm, DeepSeek's MLA, Gemma's logit softcap).
4. Read a HuggingFace config.json for any modern LLM and explain every field.

1. The 2026 standard stack

A "modern" decoder-only LLM almost always includes:

Now the variants.

2. Llama family (Meta)

Llama-3.1 / 3.2 / 3.3 (2024-2025)

• 32k-128k context (3.1) up to long-context-pretrained.
• GQA (8 KV heads), RoPE base θ=500_000.
• 128k token vocab (vs 32k in Llama 2).
• Fully open weights including the largest 405B.

Llama-4 (2025)

• Native multimodal (text + image + video).
• 1M+ token context window.
• Mixture-of-Experts (MoE) at the largest size — Phase 6.

Read its config:

from transformers import AutoConfig
cfg = AutoConfig.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
print(cfg)

Key fields: hidden_size, intermediate_size, num_hidden_layers, num_attention_heads, num_key_value_heads, rope_theta, rope_scaling, max_position_embeddings.

3. Mistral family

Mistral-7B (2023, the model that "broke" academia)

• Sliding window attention (W=4096): each token attends to last 4k. Cheap long-context.
• GQA, RoPE.
• Set the open-source quality bar by beating Llama 2 13B at half the size.

Mixtral 8×7B (2023) and 8×22B (2024)

• Sparse Mixture-of-Experts in the FFN. 8 experts; top-2 routed per token. ~13B active params, ~47B total.
• Phase 6 covers MoE in detail.

Mistral Large 2, Codestral, Pixtral (2024-2025)

• Tekken tokenizer (variant of tiktoken).
• Pixtral adds a vision encoder + cross-attention adapter.

4. Qwen family (Alibaba)

Qwen 2.5 / Qwen 3 (2024-2026)

• QK-Norm — apply RMSNorm to Q and K before dot product. Stabilises training at very long context.
• 32k - 128k context with YaRN-style scaling.
• Chinese + English balanced training.
• Strong Qwen-Coder and Qwen-Math specialised checkpoints.

Qwen-VL / Qwen2-VL / Qwen-Audio

• Vision/audio adapters in front of the LLM. Phase 6.

5. DeepSeek family (DeepSeek)

DeepSeek-V2 / V3 / R1 (2024-2025)

• Multi-head Latent Attention (MLA): compress K, V into a low-rank latent space (d_c << d); decompress lazily. Drastically reduces KV cache (~10× vs MHA). The most aggressive KV-savings architecture in production.
• DeepSeek-MoE: very fine-grained MoE (256 experts, top-8 routing). Far more experts than Mixtral.
• DeepSeek-R1: post-training emphasising RL for reasoning (Phase 4). The model that first made open-weight "o1-style" reasoning a thing.

MLA in one paragraph

Project x to a low-rank latent c_t (e.g., 512-d). Cache only c_t. At attention time, project c_t up to K, V on the fly. Trades a little compute for ~10× KV memory savings. Combined with decoupled RoPE (separate small dims that do hold position info), MLA achieves both compression and quality.

6. Gemma family (Google)

Gemma 2 / Gemma 3 (2024-2025)

• Logit softcap: cap logits with tanh(x / cap) * cap before softmax. Prevents extreme spikes.
• Alternating local + global attention layers.
• Gemma 3 adds vision via SigLIP-based encoder; multimodal.

7. Phi family (Microsoft)

Phi-3 / Phi-3.5 / Phi-4 (2024-2025)

• "Textbooks are all you need" thesis: high-quality synthetic + curated data > raw scale. Phi-4 (14B) competes with much larger general models.
• Vanilla architecture (RoPE + GQA + SwiGLU). The novelty is in the data and training, not the architecture.

8. Other architectural quirks worth knowing

• Parallel attention/FFN (PaLM, GPT-J): compute attention and FFN in parallel and sum, instead of sequentially. Saves a tiny bit of latency.
• NoPE (Kazemnejad 2023): skip positional encoding entirely; relies on causal masking + decoder structure for implicit position. Surprisingly competitive on small scales; less common at frontier scale.
• RoPE base θ scaling: training with θ=500_000 (instead of 10_000) yields better long-context quality from the start (Llama 3).
• Hybrid attention: mix global attention with local attention layers (Gemma 3, Mistral Mamba hybrids). Phase 6.
• Memory layers / explicit recurrence: experimental (Memory3, Sparse Memory).

9. The interview cheat sheet

If asked "describe a modern LLM architecture":

"Decoder-only transformer with pre-norm RMSNorm, SwiGLU FFN at ~3× hidden size, RoPE positional encoding, GQA with around 8 KV heads, no biases, byte-level BPE tokenizer, weight-tied input embedding and LM head. Trained in bf16 with AdamW, cosine LR with warmup, ~6 tokens per parameter (Chinchilla-optimal) or 15-20 tokens per parameter (over-trained for inference efficiency). Inference uses FlashAttention with a paged KV cache."

If asked "name a non-standard architecture":

"DeepSeek-V3 uses Multi-head Latent Attention (MLA) and a fine-grained MoE FFN — they trade extra compute for ~10× less KV memory and improved per-active-parameter quality."

If asked "what's the difference between Llama and Mistral":

"Llama 3 uses standard full attention with GQA; Mistral 7B used sliding-window attention. At the 7-13B scale they are very close in architecture; differences are mostly in data, tokenizer (Llama 3's 128k vs Tekken), and post-training."

Hands-on lab (3 hours)

variants_lab.ipynb:

1. Print and parse the config.json for: meta-llama/Llama-3.1-8B-Instruct, mistralai/Mistral-7B-Instruct-v0.3, Qwen/Qwen2.5-7B-Instruct, deepseek-ai/DeepSeek-V2-Lite. Tabulate n_layers, d, n_q_heads, n_kv_heads, d_ff, rope_theta.
2. Compute parameter count for each; verify against the model card.
3. Compute KV cache size at T=32k, B=1, bf16 for each. Note DeepSeek's MLA win.
4. Instantiate each tokenizer; compare token counts on the same English/code/Chinese paragraph.
5. Bonus: read modeling_qwen2.py in transformers and identify QK-Norm.

Common pitfalls

1. Believing "a transformer is a transformer" — the variants matter for memory, latency, context length.
2. Choosing MHA in your architecture today — almost always wrong; GQA-8 is a strict win.
3. Forgetting to set rope_theta=500000 (or larger) when training a long-context model from scratch.
4. Picking SwiGLU and forgetting it has 3 weight matrices instead of 2 → param count off by 50%.
5. Comparing models without normalising tokens-per-byte — Llama vs DeepSeek vs Qwen tokenize Chinese very differently.

Self-check

1. Why is RMSNorm preferred over LayerNorm?
2. Trade-off of SwiGLU vs GELU FFN?
3. What is QK-Norm and which family uses it?
4. What does MLA compress and what is the KV cache benefit?
5. Sliding-window attention saves what cost?

References

• Touvron et al. (2024), "The Llama 3 Herd of Models."
• Jiang et al. (2023), "Mistral 7B."
• Bai et al. (2024), "Qwen2 / Qwen2.5 Technical Report."
• DeepSeek-AI (2024), "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model."
• DeepSeek-AI (2025), "DeepSeek-V3 Technical Report."
• Google (2024), "Gemma 2 Technical Report."
• Microsoft (2024), "Phi-3 / Phi-4 Technical Reports."