Multimodal LLMs (vision, audio, video)

Why this matters

By 2026, "LLM" is a misnomer — the strongest models are multimodal. GPT-4o, Claude 4, Gemini 2.5, Llama-4, Qwen-2.5-VL all natively accept text, images, audio, and video. Knowing how a vision encoder gets bolted onto a transformer (CLIP, LLaVA, Flamingo, native), what tokens an image becomes, and how to fine-tune a multimodal model is now table stakes for ML interviews.

This lesson covers vision-language models in depth and audio/video at high level.

Learning objectives

1. Describe CLIP and contrastive image-text training.
2. Compare LLaVA-style adapter, Flamingo cross-attention, and native multimodal training.
3. Explain how images become tokens for an LLM.
4. Build a small multimodal pipeline with HuggingFace.
5. Recognise modern multimodal models (Qwen-VL, Pixtral, LLaVA-NeXT, Idefics3, Llama-4, GPT-4o).

1. CLIP — the prerequisite (Radford et al., 2021)

CLIP trains a vision encoder and a text encoder jointly so that matching (image, caption) pairs end up close in a shared space.

embed_img = vision_encoder(image)
embed_txt = text_encoder(caption)
loss = symmetric_contrastive(embed_img, embed_txt)         # InfoNCE

Trained on 400M pairs scraped from the web. Outputs:

• A vision tower (ViT-L/14 was original; SigLIP, EVA, DINOv2 followed).
• A text tower (transformer encoder).
• A shared embedding space.

CLIP is the vision frontend of nearly every multimodal LLM, even the "native" ones.

from transformers import CLIPProcessor, CLIPModel
proc = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
m    = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
inputs = proc(text=["a cat", "a dog"], images=[img], return_tensors="pt", padding=True)
out = m(**inputs)
print(out.logits_per_image.softmax(-1))   # per-class probabilities

CLIP alone is great for retrieval, classification, and as a frozen vision encoder for VLMs.

2. Three integration patterns

Pattern A — Adapter (LLaVA, MiniGPT-4, BLIP-2)

[image] -> [vision encoder, frozen] -> [adapter / projector] -> [LLM as if text tokens]

The adapter is a small MLP or Q-Former that maps N vision tokens (e.g., 576 patches from CLIP-ViT-L) into the LLM's embedding space. The LLM then sees [image_tokens] + [text_tokens] and treats them uniformly.

Train in two stages:

1. Pretrain projector on image-caption data.
2. Visual instruction tuning with image-text-instruction triples (LLaVA-style).

LLaVA-1.5 / 1.6, LLaVA-NeXT, Idefics2/3, MiniCPM-V follow this pattern. Cheap to train (the LLM is mostly frozen). Strong on most benchmarks.

Pattern B — Cross-attention (Flamingo, Idefics)

LLM block: [self-attention] -> [cross-attention to vision tokens] -> [FFN]

The LLM attends separately to image features. Visual tokens are not concatenated into the text stream; they live in a parallel encoder.

Flamingo, Idefics-1, OpenFlamingo. Strong few-shot multimodal but more parameters and complexity.

Pattern C — Native multimodal (GPT-4o, Gemini, Llama-4, Qwen-VL latest)

Pretrain the LLM from scratch on interleaved text + vision (+ audio + video) tokens. No frozen encoder, no separate adapter — it is a single model that natively handles modalities.

Typically the model has:

• A patch tokenizer for images (split into 14×14 patches → linear projection → tokens).
• An audio tokenizer (often EnCodec / Whisper encoder + VQ).
• The same transformer body for everything.

This is the strongest pattern but the most expensive to train. It produces the most coherent cross-modal reasoning (e.g., GPT-4o's voice + screen-share + text).

3. How images become tokens

A vision encoder (ViT) processes an image as patches:

image (224×224×3) -> 16×16=256 patches of 14×14 -> Linear projection to d -> [CLS] + patches

This produces ~256 + 1 vectors per image. For a 1024×1024 image, modern VLMs use dynamic resolution: tile the image into multiple 336×336 sub-images, run each through the encoder, concatenate. Result: 1000-3000 visual tokens for one HD image.

→ Images are token-expensive. A 4-image RAG-with-images query at 1500 tokens/image = 6000 tokens before the user's text. Manage this cost.

Compressing visual tokens

• Q-Former (BLIP-2) — a small transformer with learnable queries that compress 576 patches → 32 tokens.
• Resampler / Perceiver (Flamingo) — similar idea.
• Honeybee, M-Resampler — newer compressors.
• MoVQ-style discrete tokens — quantize patches to a small codebook; the LLM treats them as ordinary tokens.

Most modern VLMs use a simple linear projector for quality, paying the token cost.

4. Modern VLMs to know (2025-2026)

For self-study, Qwen2.5-VL-7B-Instruct is the best balance of quality and runnability.

5. Building with HuggingFace VLMs

from transformers import AutoProcessor, AutoModelForVision2Seq
from PIL import Image
import torch

processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct")
model = AutoModelForVision2Seq.from_pretrained(
    "Qwen/Qwen2.5-VL-7B-Instruct",
    torch_dtype=torch.bfloat16, device_map="auto",
    attn_implementation="flash_attention_2",
)

img = Image.open("chart.png")
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": img},
        {"type": "text",  "text": "Summarise the trend in this chart."},
    ],
}]
text   = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[img], return_tensors="pt").to(model.device)
out    = model.generate(**inputs, max_new_tokens=300)
print(processor.batch_decode(out, skip_special_tokens=True)[0])

For multi-image / video: pass a list of images / a list of frames.

6. Audio LLMs

Two main families:

Speech-to-text + LLM pipeline

Whisper / NeMo Parakeet / Faster-Whisper transcribe; LLM consumes transcript. Mature; non-trivial latency. Used in most production "voice agents."

Native audio LLMs

• Whisper-style encoder + LLM — feed continuous Whisper features to an LLM (Qwen-Audio, Phi-4-Multimodal).
• Discrete audio tokens — EnCodec / SoundStream tokenize audio into ~50 tokens/sec; LLM treats them as another modality. Used by GPT-4o (its low-latency voice mode), Moshi, AudioLM.
• Output audio — additional decoder transforms generated tokens back to waveform.

Latency target: ~300ms. Real-time conversational audio LLMs are an active production frontier (Sesame, OpenAI Realtime API, ElevenLabs Conversational AI).

7. Video LLMs

Effectively "many-frame VLM." Sample N frames, encode each, concatenate tokens. Token cost is enormous → most models cap at 8-32 frames per minute and use compression (Q-Former, time-pooling).

Strong models:

• Gemini 2.5 — multi-hour video, native.
• GPT-4.x with video API.
• Qwen2.5-VL — handles up to ~minutes natively.
• InternVideo2 / Video-LLaVA / LLaVA-Video — open research.

8. Multimodal evaluation

Standard benchmarks:

• MMMU — multimodal reasoning across disciplines.
• MathVista — math + visual.
• MMBench / MMStar — broad VLM eval.
• DocVQA / ChartQA — documents and charts.
• OCRBench — text-in-image extraction.
• VideoMME / Perception Test — video.

Always also run a small internal eval on your own use case (e.g., "given a UI screenshot, identify the button to click").

9. Fine-tuning multimodal models

• Stage-1 projector training — freeze vision and LLM; train the projector on captions.
• Stage-2 visual instruction tuning — unfreeze LLM; train on (image, instruction, response) triples.
• LoRA on LLM only — cheap fine-tune for new domains.

Datasets: liuhaotian/LLaVA-Instruct-150k, lmms-lab/LLaVA-NeXT-Data, your own task-specific.

Hands-on lab (4 hours)

vlm_lab.ipynb:

1. Run CLIP zero-shot classification on 5 of your images vs 10 candidate labels.
2. Use Qwen2.5-VL-7B-Instruct to answer questions about a complex chart and a UI screenshot.
3. Compare token counts: pass the same image at 224×224 vs 1024×1024. Note explosion.
4. Fine-tune the projector of llava-hf/llava-onevision-qwen2-0.5b-ov-hf on a tiny custom set with LoRA.
5. Build a small RAG-with-images: index 100 images by CLIP embeddings; retrieve top-3 for a query; send them to a VLM for answer.
6. Bonus: try a video model (Qwen2.5-VL-7B-Instruct accepts videos) on a 30-second clip; explore frame-sampling controls.

Common pitfalls

1. Image token cost surprises — always print the input token count.
2. Wrong processor — VLMs require model-specific AutoProcessor. The text tokenizer alone won't work.
3. Freezing vision tower wrong during fine-tune — most VLMs need the vision tower frozen except for late layers.
4. Resolution mismatch — passing 4k images to a model trained at 336×336 → poor quality.
5. Mixing modalities in a chat template the model wasn't trained for.

Self-check

1. What does CLIP train two encoders to do?
2. Compare adapter, cross-attention, and native multimodal patterns.
3. What is a Q-Former for?
4. How are images turned into LLM tokens?
5. Why is video so expensive to process?

References

• Radford et al. (2021), "CLIP: Learning Transferable Visual Models From Natural Language Supervision."
• Alayrac et al. (2022), "Flamingo: a Visual Language Model for Few-Shot Learning."
• Li et al. (2023), "BLIP-2: Bootstrapping Language-Image Pre-training" (Q-Former).
• Liu et al. (2023), "Visual Instruction Tuning" (LLaVA).
• Bai et al. (2024), "Qwen2-VL Technical Report" / Qwen2.5-VL release.
• Beyer et al. (2024), "PaliGemma."
• Meta (2024-2025), "Llama 3.2 Vision" / "Llama 4."
• Défossez et al. (2024), "Moshi: a speech-text foundation model for real-time dialogue."