Sequence models before Transformers (RNN, LSTM, seq2seq)

Why this matters

Transformers won, but the vocabulary and intuitions of modern LLMs come from RNNs and seq2seq:

• "Hidden state" → KV cache.
• "Encoder–decoder" → original Transformer architecture.
• "Attention" → originated as an add-on to seq2seq RNNs.
• "Teacher forcing," "exposure bias," "beam search" — all from this era.

Spend 4 hours here and you will read transformer papers with much more confidence.

Learning objectives

1. Explain how an RNN consumes a sequence and what its hidden state represents.
2. Understand vanishing gradients and how LSTM gates solve them.
3. Describe seq2seq with encoder-decoder.
4. Trace the Bahdanau attention mechanism that birthed the Transformer.
5. Build a tiny LSTM language model and use it for sampling.

1. Recurrent neural networks (RNN)

The model maintains a hidden state h_t updated as each token arrives:

h_t = tanh(W_xh x_t + W_hh h_{t-1} + b_h)
y_t = softmax(W_hy h_t + b_y)

Unrolled, an RNN is a deep network where the same weights are reused at every step → trains via backprop-through-time (BPTT).

Why it kind-of worked

It is the simplest model that handles variable-length sequences. From 2010-2015 it set SOTA on language modelling, MT, and speech recognition.

Why it did not scale

• Vanishing gradients: h_t depends on h_1 through ~T multiplications. Gradients shrink exponentially → cannot learn long-range dependencies.
• Exploding gradients: opposite issue; mitigated with clipping but unstable.
• No parallelism: must process tokens sequentially in time → expensive on GPUs.

Both issues motivated LSTMs (gates) and eventually Transformers (no recurrence).

2. LSTM and GRU — gating to the rescue

LSTM (Hochreiter & Schmidhuber 1997) introduces a cell state c_t and three gates: input, forget, output. The forget gate decides what to keep from the previous step, the input gate decides what to add. That additive structure gives gradients a clean highway through time.

class LSTMCell(nn.Module):
    def __init__(self, d_in, d_h):
        super().__init__()
        self.i = nn.Linear(d_in + d_h, d_h)   # input gate
        self.f = nn.Linear(d_in + d_h, d_h)   # forget gate
        self.g = nn.Linear(d_in + d_h, d_h)   # candidate
        self.o = nn.Linear(d_in + d_h, d_h)   # output gate
    def forward(self, x, hc):
        h, c = hc
        z = torch.cat([x, h], -1)
        i = torch.sigmoid(self.i(z))
        f = torch.sigmoid(self.f(z))
        g = torch.tanh(self.g(z))
        o = torch.sigmoid(self.o(z))
        c = f * c + i * g
        h = o * torch.tanh(c)
        return h, c

PyTorch ships nn.LSTM (multi-layer, bidirectional, dropout). Use it.

GRU (Cho 2014) is a simpler 2-gate variant. Slightly faster, similar quality.

LSTM-based models held SOTA on most NLP tasks until 2018, when transformers ate them.

3. Encoder–decoder seq2seq (the architecture that learned MT)

Sutskever, Vinyals, Le (2014). The trick: use one RNN to read the source sentence into a vector, another to generate the target.

ENCODER: source tokens → h_enc (a single vector summary)
DECODER: starts from h_enc, autoregressively emits target tokens

It worked surprisingly well, but compressing a 50-word sentence into a single vector is a bottleneck for long inputs.

4. Attention — the missing piece (Bahdanau 2014)

Bahdanau's insight: instead of compressing the source into a single vector, let the decoder look up the relevant source words at each output step.

For each decoder step t:

score(t, j) = MLP(h_dec_t, h_enc_j)           # alignment score
α(t, j)     = softmax_j(score(t, j))          # attention weights
context_t   = sum_j α(t, j) * h_enc_j         # weighted sum of source states

The decoder uses [h_dec_t ; context_t] to predict the next token.

Two profound consequences:

1. Translation quality jumped dramatically.
2. The decoder could now attend to any source token at any time → no more bottleneck.

Three years later (Vaswani 2017), the Transformer asked: what if we drop the RNN entirely and use only attention? That is Phase 2.

Variants of attention you will encounter

• Additive (Bahdanau) — the MLP-based score above.
• Multiplicative (Luong) — score = h_dec . h_enc.
• Scaled dot-product (Vaswani) — score = (Q . K) / sqrt(d) — the basis of transformers.

5. Tiny LSTM language model (working code)

import torch, torch.nn as nn, torch.nn.functional as F

class LSTMLM(nn.Module):
    def __init__(self, V, d=128, n_layers=2):
        super().__init__()
        self.emb  = nn.Embedding(V, d)
        self.lstm = nn.LSTM(d, d, num_layers=n_layers, batch_first=True, dropout=0.1)
        self.head = nn.Linear(d, V)
    def forward(self, ids, hidden=None):
        x = self.emb(ids)               # (B, T, d)
        out, hidden = self.lstm(x, hidden)
        logits = self.head(out)         # (B, T, V)
        return logits, hidden

# train on next-token prediction
@torch.no_grad()
def sample(model, start_ids, n=50, T=1.0):
    ids = start_ids.clone()
    h = None
    for _ in range(n):
        logits, h = model(ids[:, -1:], h)
        probs = F.softmax(logits[:, -1] / T, dim=-1)
        nxt = torch.multinomial(probs, 1)
        ids = torch.cat([ids, nxt], dim=1)
    return ids

This tiny model, trained for 30 minutes on tiny-shakespeare, will produce plausible Shakespearean nonsense. It is the same training objective as GPT — only the architecture differs.

6. Concepts that survived into LLMs

You will not train an LSTM in production again — but nn.LSTM will appear in homework, embeddings ablations, and small components (e.g., audio frontends).

Hands-on lab (3 hours)

lstm_drills.ipynb:

1. Build the LSTMLM above. Train on tiny_shakespeare (~1MB) for 5 epochs.
2. Plot training perplexity per epoch.
3. Sample 200 tokens at temperatures [0.5, 1.0, 1.5]. Compare outputs.
4. Add gradient clipping at max_norm=5. Observe stability.
5. Replace LSTM with GRU. Compare PPL.
6. Bonus: add Bahdanau-style attention as if you were doing seq2seq translation between two language pairs in Multi30k. (HF datasets has it.)

Common pitfalls

1. Forgetting batch_first=True in nn.LSTM → shape (T, B, d) not (B, T, d). Both are valid; pick one and be consistent.
2. Computing loss on padded tokens. Use ignore_index=PAD_ID in cross_entropy.
3. Calling .detach() on hidden state when continuing training across batches (truncated BPTT).
4. Using a too-small hidden size on a complex sequence task.

Self-check

1. Why do RNNs suffer from vanishing gradients?
2. What gate in an LSTM solves the long-range memory problem?
3. What problem in seq2seq did attention solve?
4. Difference between additive and dot-product attention?
5. Why don't transformers use RNNs at all?

References

• Hochreiter & Schmidhuber (1997), "Long Short-Term Memory."
• Bahdanau et al. (2014), "Neural Machine Translation by Jointly Learning to Align and Translate."
• Sutskever et al. (2014), "Sequence to Sequence Learning with Neural Networks."
• Karpathy, "The Unreasonable Effectiveness of Recurrent Neural Networks."
• Olah, "Understanding LSTM Networks" — the canonical visual guide.