BPE tokenizer from scratch — Large Language Models

Build a byte-level BPE tokenizer in pure Python, train it on a small corpus, and compare against tiktoken.

Goal

Build a byte-level BPE tokenizer in pure Python. Train it on a small corpus. Compare it against tiktoken on tokens-per-byte. Ship a clean library API.

This project teaches: text encoding, the bytes type in Python, BPE training algorithm, encoding/decoding round-trip safety. The single best test of your "actually understands LLM internals" credibility on a resume.

Time: 3-7 days.

Prerequisites

00_foundations/03_classic_nlp.md
Reading: Karpathy's "Let's build the GPT tokenizer" video.

Tech stack

Python 3.11+
regex (Unicode-aware)
numpy (optional, for arrays)
pytest for tests
rich for pretty CLI

No tokenizers, no tiktoken for the implementation (you'll use them only for comparison).

Architecture

mermaid

flowchart LR
    CORPUS[corpus.txt] --> PRET[GPT-4 split regex]
    PRET --> BYTES[utf-8 bytes per pretoken]
    BYTES --> BPE[BPE train: count pairs, merge top, repeat]
    BPE --> VOCAB[(vocab.json + merges.txt)]
    VOCAB --> ENC[encoder]
    VOCAB --> DEC[decoder]
    ENC --> IDS[token ids]
    IDS --> DEC --> TXT[recovered text]

Step-by-step

1. Pre-tokenization regex

Use GPT-4's cl100k_base style regex to split text into "atoms" before BPE. (Improves coding-text handling and avoids merging across natural boundaries.)

python

import regex as re
GPT4_PATTERN = re.compile(r"""'(?i:[sdmt]|ll|ve|re)|[^\r\n\p{L}\p{N}]?+\p{L}+|\p{N}{1,3}|
                              ?[^\s\p{L}\p{N}]++[\r\n]*|\s*[\r\n]|\s+(?!\S)|\s+""", re.X)

2. Train (greedy merges)

Start: every byte of every pretoken is its own token (256 base tokens).
Loop until vocab size = target:
- Count adjacent token pairs across the corpus.
- Take the most frequent pair (a, b).
- Add a new token (a, b) -> id.
- Replace every (a, b) in the corpus with the new id.

Implement efficiently with a dict of pretokens → list of token ids.

3. Encode

For input text:

Split into pretokens.
Convert each pretoken to bytes → ids (256-base alphabet).
Apply learned merges in order until none apply.

Output: a list of integers.

4. Decode

Look up each id → its byte sequence (via the merge tree).
Concatenate; decode as UTF-8.

5. Special tokens

Add <|endoftext|>, <|im_start|>, <|im_end|> as out-of-band ids. They never participate in merges; the encoder treats them as escape sequences in the input.

6. Tests

Round-trip on 1000 random Unicode strings (including emojis, CJK, RTL).
Determinism: same training data → same tokenizer.
Compatibility: small corpus, train two ways, verify same vocab ordering.

7. Compare

On tinyshakespeare, enwik8, a code corpus, and a multilingual corpus, compute:

Average tokens per byte (your tokenizer vs cl100k_base vs o200k_base).
Average tokens per English word.

Report a table and a short discussion.

Acceptance criteria

Library API: bpe.train(corpus, vocab_size), bpe.encode(text), bpe.decode(ids), bpe.save(path), bpe.load(path).
CLI: bpe train ..., bpe encode ..., bpe decode ....
Round-trip test passes on 10k random Unicode strings.
README shows comparison table vs tiktoken.
>= 90% test coverage.
Trains an 8000-vocab tokenizer on a 100MB corpus in < 15 minutes single-threaded.
Implements special tokens correctly (encoded only when explicitly requested).

Stretch goals

Multi-process training with multiprocessing.Pool for the pair-count step.
BPE-Dropout for training-time regularisation.
Add tokenizer alignment metric: tokens_per_byte per language.
Re-export to a HuggingFace tokenizers JSON file so it can be loaded by transformers.

Common pitfalls

Splitting on Unicode codepoints instead of UTF-8 bytes — multilingual breaks.
Not handling whitespace prefix tokens correctly (the leading space "_" issue).
Losing merge order on save/load.
Forgetting to add a fallback for OOV bytes.

Story / portfolio

Title: "I built tiktoken, but worse — here is what I learned."

Code link, README, ~1500-word post.
A graph of tokens_per_byte over training steps.
A side-by-side encoding of the same paragraph with your tokenizer vs o200k_base.

This is a small project but a very high-signal one. Reviewers immediately know you understand what most engineers do not.