Classic NLP (the part that still matters)

Why this matters

LLMs did not appear out of thin air. The decisions baked into modern models — how they tokenize, what they predict, how their embeddings are organised — come straight from 30 years of NLP. You do not need n-gram theory to ship a chatbot, but if you ever debug a tokenizer, train a custom model, or interview at OpenAI / Anthropic / Google DeepMind, you are expected to fluently discuss tokenization, language modelling, and word embeddings.

This lesson covers the bridge between classic NLP and the modern Transformer era.

Learning objectives

1. Understand what a language model is, mathematically.
2. Implement and use Byte-Pair Encoding (BPE) tokenization.
3. Train and inspect simple word embeddings (word2vec).
4. Use HuggingFace tokenizers and tiktoken correctly.
5. Compute perplexity and reason about tokenizer choice.

1. What is a language model?

A language model assigns a probability to any sequence of tokens.

p(w_1, w_2, ..., w_T) = ∏_t p(w_t | w_1, ..., w_{t-1})

That product over conditional probabilities is the autoregressive factorisation — the same one GPT uses today. The only difference between an n-gram LM in 1990 and Llama-4 in 2026 is the function class used to estimate p(w_t | context):

The training objective is identical: maximise the log-likelihood of the next token given the context. Everything you do in this course — pretraining, SFT, RLHF — eventually reduces to manipulating that conditional distribution.

2. Tokenization — the silent giant

Models do not see characters or words. They see token ids. The choice of tokenizer determines:

• How many tokens an input/output costs (= money, latency).
• Whether the model can spell (yes, this is a tokenization issue).
• Whether non-English languages are first- or second-class citizens.
• Whether the model handles code, emojis, and rare names well.

Word-level (obsolete)

Vocab = every distinct word. Problem: out-of-vocabulary words and a 500k+ vocab.

Character-level

Tiny vocab (~256), but sequences become 5-7x longer → expensive attention, weak token semantics. Some research models still use it.

Subword tokenization (the winner)

Byte-Pair Encoding (BPE) — start with bytes, repeatedly merge the most frequent adjacent pair until you hit a target vocab size. Trained once on a large corpus.

Initial: ["t", "h", "e", " ", "q", "u", "i", "c", "k"]
Merge "th" -> ["th", "e", " ", "q", "u", "i", "c", "k"]
Merge "the" -> ["the", " ", "q", "u", "i", "c", "k"]
...

After training, encoding any string is greedy: apply learned merges left-to-right.

Variants you will meet:

• Byte-level BPE (GPT-2/3/4, Llama-3) — operate on UTF-8 bytes, so any Unicode is representable.
• SentencePiece (Unigram) (T5, mT5, Mistral first version) — probabilistic alternative, often better for multilingual.
• WordPiece (BERT) — merges chosen by likelihood ratio.
• TikToken (cl100k_base, o200k_base) — OpenAI's fast Rust BPE used by GPT-4 / 4o.
• Tekken (Mistral) — Tekken is a tiktoken-style tokenizer used by Mistral Large / Pixtral.

Why this matters in interviews

You will be asked: "Why does the model say there are two rs in 'strawberry'?" Answer: because to the model "strawberry" is one token, not 10 letters. It never sees letters. (Modern models have begun mitigating with character-aware training, byte fallback, and tool use, but the underlying issue is real.)

Hands-on with tiktoken

import tiktoken
enc = tiktoken.get_encoding("o200k_base")     # GPT-4o tokenizer
ids = enc.encode("Hello, world! 안녕하세요 🌍")
print(ids)
print(enc.decode(ids))
print([enc.decode([i]) for i in ids])         # see each token

Train your own BPE with tokenizers

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import ByteLevel

tok = Tokenizer(BPE(unk_token="<unk>"))
tok.pre_tokenizer = ByteLevel()
trainer = BpeTrainer(vocab_size=8000, special_tokens=["<pad>", "<s>", "</s>", "<unk>"])
tok.train(files=["corpus.txt"], trainer=trainer)
tok.save("my_bpe.json")

Project 1 (in projects/01_tokenizer_from_scratch.md) has you build BPE in pure Python.

3. Word embeddings — the quiet revolution

Before transformers, the big idea was that words can be represented as dense vectors where geometry encodes meaning.

word2vec (Mikolov, 2013)

Trains a tiny network on a huge corpus to predict either:

• CBOW: predict word from its neighbours.
• Skip-gram: predict neighbours from a word.

After training, each word has a learned vector (~300 dims). Famous result:

vec("king") - vec("man") + vec("woman") ≈ vec("queen")

This is not magic — it is a side-effect of co-occurrence patterns. But it gave NLP its first usable distributional semantics.

GloVe (Stanford, 2014)

Optimises a different objective: factorise the global co-occurrence count matrix. Roughly equivalent in quality to word2vec.

fastText (Facebook, 2016)

word2vec but with subword features. Robust to typos and rare words.

Why this matters today

The token embedding table at the bottom of every Transformer is the direct descendant of word2vec — it learns a vector per token from co-occurrence patterns during pretraining. The geometry observations (king - man + woman ≈ queen) still hold inside Llama-3's embedding matrix.

Modern sentence embeddings (used in RAG) extend this idea: train a model so that whole sentences map to vectors where semantic similarity = cosine similarity. Examples: text-embedding-3-large, bge-m3, voyage-3, nomic-embed-v2.

Try it

from sentence_transformers import SentenceTransformer
import numpy as np

m = SentenceTransformer("BAAI/bge-small-en-v1.5")
vecs = m.encode(["a king sits on a throne", "a queen reigns over a kingdom",
                 "the stock market crashed today"])
def cos(a,b): return float(np.dot(a,b) / (np.linalg.norm(a)*np.linalg.norm(b)))
print(cos(vecs[0], vecs[1]))   # high
print(cos(vecs[0], vecs[2]))   # low

4. Perplexity — the intrinsic score

Perplexity = exp(cross_entropy). Intuitively, "the model is as confused as if it had to choose uniformly among PPL options at each step."

import math, torch.nn.functional as F
loss = F.cross_entropy(logits.view(-1, V), targets.view(-1))
ppl = math.exp(loss.item())
print(f"PPL = {ppl:.2f}")

Why care:

• Training health: PPL should go down monotonically (after warmup).
• Tokenizer comparison: Be careful — PPL depends on the tokenizer's average token length. Comparing across tokenizers is meaningless without normalising to bits-per-byte.
• Bits-per-byte (BPB) = loss * tokens_per_byte / ln(2). This is the fair cross-tokenizer metric.

5. The bridge to deep learning NLP

Pre-2017 architecture lineage you should recognise:

• N-gram LMs (with Kneser-Ney smoothing) — fast, brittle.
• Bengio 2003 — first neural LM with embeddings + MLP.
• RNN LMs (Mikolov 2010) — stateful, capture longer context but suffer from vanishing gradients.
• LSTM / GRU — gates fix vanishing gradients; SOTA from 2014-2017.
• Seq2Seq with attention (Bahdanau 2014) — encoder-decoder with the first attention mechanism. Direct ancestor of the Transformer.
• Transformer (Vaswani 2017) — drops recurrence, all-attention. Phase 2 of this course is dedicated to it.

You do not need to implement an LSTM from scratch (we will at a high level in Lesson 1.2). You should know the names and roughly what problem each one solved.

Hands-on lab (3 hours)

classic_nlp.ipynb:

1. Load tiny_shakespeare.txt (1MB). Train a BPE tokenizer with tokenizers lib, vocab=2000.
2. Encode and decode several lines; print round-trip is exact.
3. Compute average tokens per line for your BPE vs cl100k_base vs o200k_base. Which is most efficient on Shakespeare?
4. Encode "café 北京 🚀" with o200k_base. Show the byte-level ids of the emoji.
5. Use sentence-transformers (bge-small-en-v1.5). Compute cosine similarity for these triples and report which is highest:
   • ("dog", "puppy", "skyscraper")
   • ("Paris is the capital of France", "France's capital is Paris", "An apple a day")
6. Compute perplexity of gpt2 on a 5-line snippet using HuggingFace.
7. Bonus: train word2vec on a 10k-line corpus with gensim. Show 5 nearest neighbours for "king".

Common pitfalls

1. Comparing PPL across tokenizers — meaningless without BPB normalisation.
2. Forgetting BOS/EOS tokens — many models (Llama, Mistral) require BOS prepended; missing it silently degrades quality.
3. Lowercasing before tokenizing modern models — most byte-level BPEs are case-sensitive on purpose.
4. Long emojis / CJK characters turning into many tokens — measure your real-world cost.
5. tokenizer.encode(text, add_special_tokens=False) — defaults differ between libraries. Print and verify.

Self-check

1. Why did the field move from word-level to subword tokenization?
2. Explain why "strawberry" produces tokenization-related errors.
3. What is the relationship between cross-entropy loss and perplexity?
4. Why is byte-level BPE preferred for multilingual + code models?
5. What problem did the attention mechanism solve in seq2seq before transformers?

References

• Jurafsky & Martin, Speech and Language Processing, 3rd ed. (free online) — Chapters 2-7.
• Mikolov et al. (2013), "Efficient Estimation of Word Representations" (word2vec).
• Sennrich et al. (2016), "Neural Machine Translation of Rare Words with Subword Units" (BPE).
• Karpathy's Let's build the GPT tokenizer (YouTube).
• HuggingFace Tokenizers chapter.
• tiktoken on GitHub.