The pretraining data pipeline

Why this matters

"We didn't train a smarter model. We trained on smarter data."
— every senior pretraining engineer, ever.

For ~5 years the pretraining recipe was "more parameters." Since 2023 it has been "more, cleaner, more diverse tokens." The single biggest predictor of a model's quality is the data pipeline. This lesson teaches it.

Learning objectives

1. List the public pretraining corpora (CommonCrawl, FineWeb, Dolma, RedPajama, etc.).
2. Build a pipeline: download → text-extract → language ID → quality filter → dedup → tokenize.
3. Implement near-duplicate detection (MinHash + LSH).
4. Reason about data mixtures — code vs web vs books vs multilingual.
5. Avoid contamination of evaluation sets.

1. The corpus landscape (2025-2026)

For self-study: HuggingFaceFW/fineweb-edu (15B sample) is the modern-default starter corpus.

2. The pipeline (end-to-end)

[CommonCrawl WARC files]
        │
        ▼  trafilatura / extractus  (HTML → plain text)
[raw text]
        │
        ▼  fastText langid
[language-tagged text]
        │
        ▼  quality classifier (FastText, Reddit-trained, or LLM judge)
[high-quality docs]
        │
        ▼  MinHash + LSH dedup (across the corpus)
[unique docs]
        │
        ▼  PII / safety filter, decontamination vs eval sets
[ready text]
        │
        ▼  tokenize → uint16/uint32 binary shards
[mmap-ready tokens.bin]
        │
        ▼  data mixer (proportions per domain)
[training stream]

We will walk each step.

3. Text extraction

CommonCrawl WARC files contain HTML. Extracting clean text is harder than it looks:

import trafilatura
html = open("page.html").read()
text = trafilatura.extract(html, include_comments=False, include_tables=False)

trafilatura outperforms readability and BeautifulSoup on web boilerplate removal. For PDFs use LlamaParse or Unstructured.io.

4. Language identification

import fasttext
ft = fasttext.load_model("lid.176.bin")        # 176-language model
lang, conf = ft.predict("Hello world")

Reject docs with low confidence or unwanted languages, depending on the data mix you want.

5. Quality filtering

Two main approaches:

Heuristic filters (Llama / Gopher style)

• Reject docs <50 or >100k words.
• Reject if mean line length < 10 chars.
• Reject if symbol-to-word ratio > 0.1.
• Reject if perplexity-vs-Wikipedia is too high (use a small Wikipedia LM).
• Reject if too many bullet points / boilerplate signatures.

Model-based filters

• Train a classifier on "is this educationally valuable text" vs random web (FineWeb-Edu's recipe). Score every doc; keep top X%.
• Use an LLM (e.g., Mixtral 8×7B) to score 5M random docs, fine-tune a small classifier on those, then run the classifier across the entire corpus.

This is the highest-leverage step. FineWeb-Edu's 5T tokens beats a 15T-token raw web corpus on most benchmarks.

6. Deduplication (must do)

CommonCrawl has 30-70% duplicate or near-duplicate content. Training on duplicates is a strict harm — it causes verbatim memorisation and wastes compute.

Two scales:

Exact dedup

Hash each document; drop matches.

Near-duplicate dedup with MinHash + LSH

For each document, compute a small signature (e.g., 128 hash values of shingles). Documents with high Jaccard similarity have similar signatures.

from datasketch import MinHash, MinHashLSH

def shingles(text, k=5):
    tokens = text.split()
    return {" ".join(tokens[i:i+k]) for i in range(len(tokens)-k+1)}

def minhash(text, num=128):
    m = MinHash(num_perm=num)
    for sh in shingles(text):
        m.update(sh.encode())
    return m

lsh = MinHashLSH(threshold=0.8, num_perm=128)
for i, doc in enumerate(docs):
    lsh.insert(f"d{i}", minhash(doc))

# duplicates of doc 42:
print(lsh.query(minhash(docs[42])))

Use text-dedup (HuggingFace) or datasketch for production. At scale (T-tokens), ship dedup on Spark / Ray.

7. PII and safety filtering

• Use Microsoft Presidio to redact emails, SSNs, phone numbers.
• Use perspective-API or a small classifier to drop toxic / NSFW pages.
• Document everything for compliance and reproducibility.

8. Decontamination vs evaluation sets

If 13-grams from MMLU / GSM8K / HumanEval appear in training data, your benchmark scores are meaningless. Build a 13-gram filter against your eval set.

eval_grams = set(extract_13grams(eval_corpus))
clean = [doc for doc in train_docs
         if not any(g in eval_grams for g in extract_13grams(doc))]

OpenAI, Meta, and Anthropic all publish "decontamination diff" tables in technical reports.

9. Tokenization at scale

import tiktoken
enc = tiktoken.get_encoding("o200k_base")

def tokenize_shard(input_path, output_path):
    with open(input_path) as f, open(output_path, "wb") as out:
        for line in f:
            ids = enc.encode_ordinary(line.rstrip("\n"))
            ids.append(enc.eot_token)              # end-of-doc separator
            np.array(ids, dtype=np.uint16).tofile(out)

For a 10T-token corpus, run this in parallel across 1000+ workers. Use numpy.memmap for training-time random access:

data = np.memmap("tokens.bin", dtype=np.uint16, mode="r")

Most production stacks split tokens into shards of ~1B tokens each for fault-tolerant parallel reads.

10. The mixture

Pretraining quality is sensitive to what fraction of tokens come from each domain. Public Llama-style mixtures:

Llama 3 added much more code (~17%) and code reasoning data, plus far more synthetic data.

Modern frontier-style mixtures (2025+):

• 25-40% code (heavily de-duplicated, high-quality + bug fixes + reasoning).
• 30-50% high-quality web (FineWeb-Edu style).
• 10-20% math + reasoning (synthetic + curated).
• 5-15% multilingual.
• 5-10% books / textbooks.

The exact mix is the most-guarded recipe in every lab.

11. Curriculum and schedule

Some labs (DeepSeek, Phi, Llama 3) do multi-stage pretraining:

1. Stage 1: huge but messier corpus.
2. Stage 2 ("annealing" / "midtraining"): smaller, cleaner, higher-quality (often heavy math + code + instruction data) at a much lower learning rate.
3. Stage 3: long-context extension (sequences ≥ 32k) for the last ~5% of tokens.

The annealing stage gives outsized gains per token; this is one of the secrets of modern strong open models.

Hands-on lab (4 hours)

data_pipeline.ipynb — work on a small sample, the techniques are the same:

1. Download HuggingFaceFW/fineweb-edu 10k-row sample via datasets.load_dataset(..., streaming=True, split="train", name="sample-10BT").
2. Apply heuristic quality filters (length, symbol ratio). Report % kept.
3. Run MinHash dedup at threshold 0.8. Report % kept.
4. Tokenize with tiktoken o200k_base. Save as uint16 binary.
5. Compute total tokens; verify shape with np.memmap.
6. Train your nano-GPT from Lesson 2.4 on this corpus for 1 hour. Compare PPL to training on tiny_shakespeare.
7. Bonus: implement a 13-gram contamination check against MMLU.

Common pitfalls

1. Skipping dedup — verbatim memorisation, wasted compute, eval inflation.
2. Tokenizing on the fly during training — bottlenecks the GPU. Pre-tokenize.
3. Forgetting EOS between docs — model never learns when to stop generating.
4. Quality filter that disproportionately keeps a single domain — you accidentally specialise the model.
5. Storing tokens as int64 when uint16 (vocab ≤ 65k) suffices — 4× more disk and bandwidth than needed.

Self-check

1. Why is dedup so important?
2. What is FineWeb-Edu and why does it outperform raw FineWeb?
3. Outline a quality filter for web text.
4. Why store tokens as uint16 instead of int64?
5. Why might a lab do an "annealing" final stage?

References

• Together AI, "RedPajama-Data."
• HuggingFace, "FineWeb-Edu" + the FineWeb blog post.
• Penedo et al. (2024), "FineWeb: Decanting the Web for the Finest Text Data at Scale."
• Soldaini et al. (2024), "Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining."
• Together AI, "The Stack v2."
• Holtzman et al. (2019), "The Curious Case of Neural Text Degeneration" (motivates dedup).
• HuggingFace text-dedup library.