Large Language Models

The focused math kit (linear algebra, probability, calculus, information theory), PyTorch essentials, and the classic NLP that still matters.

• Math for LLMs (only the parts you really need)
• PyTorch essentials for LLM work
• Classic NLP (the part that still matters)

Neural networks that survived, sequence models before Transformers, and where attention actually came from.

• Neural networks (the parts that survived)
• Sequence models before Transformers (RNN, LSTM, seq2seq)
• Attention, intuitively

The architecture that changed everything: anatomy, a deep attention dive, positional encodings, building GPT from scratch, and modern variants.

• Transformer anatomy
• Attention deep dive (MHA → MQA → GQA → FlashAttention)
• Positional encodings (sinusoidal → RoPE → YaRN)
• Build a GPT from scratch
• Modern transformer variants (what changed since 2017)

How a foundation model is born: objectives, the data pipeline, scaling laws, distributed training, and infrastructure.

• Pre-training objectives
• The pretraining data pipeline
• Scaling laws (Chinchilla and beyond)
• Distributed training (DDP, ZeRO, FSDP, TP, PP)
• Training infrastructure (mixed precision, optimisers, monitoring)

Turning a raw pretrained model into a helpful assistant: SFT, RLHF, DPO family, RLAIF, and reasoning-focused RL.

• Supervised Fine-Tuning (SFT)
• RLHF (the InstructGPT recipe)
• DPO, ORPO, KTO, IPO, SimPO
• RLAIF and Constitutional AI
• Reasoning RL (o1, R1, GRPO, RLVR)

Serving a model fast and cheap: sampling and decoding, the KV cache, speculative decoding, quantization, and long context.

• Sampling and decoding
• KV cache and PagedAttention
• Speculative decoding
• Quantization (INT8 / INT4 / FP8 / AWQ / GPTQ / GGUF)
• Long context (engineering 100k+ tokens)

Where the field is heading: mixture of experts, state-space models, multimodality, and native agentic models.

• Mixture of Experts (MoE)
• State-space models (Mamba, S4, hybrids)
• Multimodal LLMs (vision, audio, video)
• Agentic / native-tool-use models

Judge a model honestly: benchmarks and leaderboards, safety and bias evaluation, and mechanistic interpretability.

• Evaluating LLMs (benchmarks, leaderboards, judges)
• Safety, harmfulness, and bias evaluation
• Interpretability (mechanistic + behavioural)

LLM interview prep: system-design templates for training, serving, and eval stacks, plus live-coding drills for attention, KV cache, and RoPE.

• Interview prep — LLM system design templates
• Interview prep — LLM coding drills