Scaling laws (Chinchilla and beyond)

Why this matters

The single most important paper in modern LLM economics is Chinchilla (Hoffmann et al., 2022). It tells you, given a compute budget C (FLOPs), how to spend it: how many parameters N, on how many tokens D. Misallocating C between N and D can waste 30-50% of the budget. Every lab now plans pretraining around scaling laws — yet most engineers cannot derive them.

After this lesson you can: estimate the FLOPs of any training run, predict its loss, and reason about whether to over-train, scale up, or improve data.

Learning objectives

1. Describe the Kaplan and Chinchilla scaling laws and their differences.
2. Compute compute-optimal N, D for a given budget C.
3. Reason about the trade-off between training FLOPs and inference cost (over-training).
4. Predict loss before launching a training run.
5. Interpret real frontier-model recipes (Llama-3, Mistral, DeepSeek) through this lens.

1. The Kaplan laws (2020)

OpenAI's first major scaling paper. Empirically:

L(N) ~ N^(-α_N)        # holding data fixed, loss decreases as a power of params
L(D) ~ D^(-α_D)        # holding params fixed, loss decreases as a power of data
L(C) ~ C^(-α_C)

Conclusion (which became famous): "increase parameters, hold data roughly fixed." This led to GPT-3 (175B) trained on ~300B tokens — a tokens/parameter ratio of about 2.

Two years later, Chinchilla showed this was wrong.

2. The Chinchilla law (Hoffmann et al., 2022)

DeepMind retrained hundreds of models systematically and fit a joint law:

L(N, D) ≈ E + A / N^α + B / D^β

Minimising this subject to compute C ≈ 6 N D gives the compute-optimal allocation:

N_opt ∝ C^a              (a ≈ 0.5)
D_opt ∝ C^(1-a)          (1-a ≈ 0.5)

In practice: D ≈ 20 N. Roughly 20 training tokens per parameter is compute-optimal.

Worked example

Suppose you have C = 1e22 FLOPs (a ~$1M training run on H100s).

6 N D = 1e22
D ≈ 20 N
6 * N * 20 N = 1e22
N^2 = 1e22 / 120 ≈ 8.3e19
N   ≈ 9.1e9        (~ 9B parameters)
D   ≈ 1.8e11       (~ 180B tokens)

So a $1M-equivalent budget should buy you roughly a 9B model on 180B tokens. (In practice, see "over-training" below.)

3. The compute formula C ≈ 6 N D

Where does the factor 6 come from?

For a transformer forward pass on one token:

• Each parameter is involved in roughly 2 multiply-adds (1 for the weight, 1 for the activation gradient during backward).

Counting:

• Forward: 2 N D FLOPs.
• Backward: 4 N D FLOPs (gradient w.r.t. weights AND activations).
• Total: 6 N D.

This is a rule of thumb; attention's O(T^2) term is non-trivial for very long contexts and is sometimes added separately.

Why this matters

You can estimate a training run's cost in 30 seconds:

FLOPs = 6 * N * D
GPU-hours ≈ FLOPs / (peak_GPU_FLOPs_per_sec * 3600 * MFU)

MFU (Model FLOPs Utilization) is typically 35-55% for well-tuned distributed training on H100s.

For 9B params, 180B tokens, on H100s (~989 TFLOPS in BF16, MFU=0.45):

FLOPs = 6 * 9e9 * 1.8e11 ≈ 9.7e21
GPU-hours = 9.7e21 / (989e12 * 3600 * 0.45) ≈ 6,060

~6k H100-hours. At ~$3/hr, ~$18k. Plus data costs, plus exploration runs (typically 5-10× the final), realistic budget ~$200k.

4. Over-training (the modern trick)

Chinchilla minimises training compute for a given loss. But you also pay for inference every time someone uses the model. If a 9B model serves 1B requests, the inference compute dwarfs training. So labs increasingly over-train: train a smaller model on much more than 20D tokens, accepting a slightly worse loss-per-FLOP but a permanently cheaper inference model.

Llama 3 8B was trained on 15 trillion tokens — that is D / N = 15e12 / 8e9 ≈ 1900, almost 100× the Chinchilla optimum. The 70B was trained on the same 15T (~215× over-trained).

Why? At Meta's scale, inference cost dominates by orders of magnitude. Saving 20% of inference compute forever is worth months of extra training.

Modern recipe (open-source, 2024-2026)

5. Data-scaling vs model-scaling

If you cannot get more high-quality data, scaling parameters past a certain point gives diminishing returns. This is the data wall people worry about.

Mitigations:

• Synthetic data (math, code, reasoning chains generated by stronger models).
• Multi-epoch training (re-using data 2-4× — works at small scale; degrades at large scale).
• Better curriculum / mid-training (annealing on best data subset).
• New data modalities (video transcripts, structured docs).

6. The loss-prediction trick

Once you know the law, you can extrapolate:

# Fit a, b, c, d to your scan of small models
def predict_loss(N, D, a, b, c, alpha, beta, E):
    return E + a / (N ** alpha) + b / (D ** beta)

Lab practice:

1. Train ~30 small models at varying N, D (e.g., 100M-1B params, 1B-30B tokens).
2. Fit the law on (N, D, loss).
3. Extrapolate the loss of the planned big run.
4. Decide go/no-go.

This is how Anthropic / OpenAI / DeepMind avoid 10M-USD surprises.

7. Beyond Chinchilla — emergent abilities and the limits

Two open debates you should be aware of:

Emergent abilities (Wei et al., 2022 → Schaeffer et al., 2023)

Some capabilities (chain-of-thought arithmetic, multi-step reasoning) appear suddenly at certain scales. Schaeffer's reanalysis showed many "emergences" are artefacts of thresholded metrics. Smooth metrics (log-prob, partial credit) usually show gradual improvement.

The data wall

Frontier labs have likely consumed most high-quality public English text. Future scale may rely on synthetic data + multimodal data + RL gains.

8. Real-world recipes through the scaling-law lens

Notice: smaller open models are massively over-trained; the largest models stay near Chinchilla-optimal because inference savings on the very largest models still need to be balanced against the (gigantic) extra training cost.

Hands-on lab (3 hours)

scaling_lab.ipynb:

1. Train your tiny GPT (Lesson 2.4) at three sizes: ~3M, ~10M, ~30M params, on the same corpus, same compute. Plot final loss vs N. Fit a power law.
2. Now fix model size at 10M and vary D (1B / 2B / 4B tokens). Plot loss vs D. Fit.
3. Compute and compare the FLOPs per second you achieve on your GPU (target: >10% of peak).
4. Estimate the FLOPs of training Llama 3 8B (15T tokens). Convert to H100-hours and dollars.
5. Bonus: read FineWeb-Edu's blog post and recreate its loss-vs-tokens plot for a 360M model.

Common pitfalls

1. Confusing parameters (the model's weight count) with active parameters (MoE) — scaling laws apply to active parameters per token.
2. Forgetting the embedding layer in N for very small models — it dominates parameter count.
3. Ignoring the attention T^2 term at long context — 6ND underestimates.
4. Over-fitting the law on too few points → wildly wrong extrapolations.
5. Treating a scaling law as universal — it depends on architecture, data, optimiser. Refit when anything changes.

Self-check

1. State the compute-optimal tokens-per-parameter ratio.
2. Why is C ≈ 6 N D a rule of thumb?
3. Why do labs over-train small models?
4. What is MFU and a typical good value?
5. What is the "data wall"?

References

• Kaplan et al. (2020), "Scaling Laws for Neural Language Models."
• Hoffmann et al. (2022), "Training Compute-Optimal Large Language Models" (Chinchilla).
• Wei et al. (2022), "Emergent Abilities of Large Language Models."
• Schaeffer et al. (2023), "Are Emergent Abilities of Large Language Models a Mirage?"
• Touvron et al. (2024), "The Llama 3 Herd of Models" — recipes through the scaling lens.
• Hoffmann et al. (2022) supplementary material — derivation of the compute formula.
• Epoch AI's training compute trends — keep these bookmarked.