Quantization (INT8 / INT4 / FP8 / AWQ / GPTQ / GGUF)

Why this matters

A 70B model in bf16 weighs 140 GB — too big for any single consumer GPU. Quantize to INT4 and it fits in 35 GB — runs on a single H100 or even a 40 GB A6000. Quality drop: typically 1-2% on benchmarks. Quantization is the single biggest lever for shipping LLMs at low cost or running them locally.

This lesson teaches the active formats (INT8, INT4 with AWQ/GPTQ, FP8 native, GGUF, BitNet) and how to choose between them.

Learning objectives

1. Distinguish weight-only, activation-quantized, and full-quantized formats.
2. Use AWQ, GPTQ, FP8, and GGUF correctly.
3. Reason about quality trade-offs.
4. Quantize a model end-to-end with each major library.
5. Pick the right format for a given deployment target.

1. Why quantization works

Modern LLM weights are very redundant. After training:

• Most weight values cluster near zero.
• A few outlier channels carry disproportionate magnitude.
• Activations have similar structure (some channels much larger than others).

Quantization approximates float weights with low-bit integers. As long as we capture outlier behaviour, we can drop precision dramatically without hurting quality.

The basic operation

For per-channel symmetric quantization:
  scale = max(|w|) / 127         # for INT8
  q     = round(w / scale)       # int values in [-127, 127]
  dq    = q * scale              # dequantized approximation

For block-wise (every 32-128 weights share a scale): better quality at the cost of 1-2 extra bits per block.

2. Weight-only quantization (the easy win)

Quantize only the weights; activations stay in fp16/bf16. Cheap to apply, dominant for inference today.

INT8 / FP8 weight-only

• 2× smaller model.
• Negligible quality loss.
• Drop-in for most engines.

INT4 weight-only

• 4× smaller model.
• 1-3% quality loss with naive uniform.
• With AWQ or GPTQ: ≤1% loss.

3. AWQ — Activation-aware Weight Quantization (Lin et al., 2023)

Insight: not all weights are equal. The weights connected to large-activation channels matter much more. AWQ keeps those channels at higher precision (or scales them up before quantizing).

Algorithm:

1. Profile activations on a small calibration set.
2. Identify the top-1% "important" channels.
3. Apply per-group quantization with a learned scaling that protects them.

Strengths: very fast to apply, no fine-tuning, INT4 with negligible loss for most models.

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model = AutoAWQForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
model.quantize(tok, quant_config={"zero_point": True, "q_group_size": 128,
                                   "w_bit": 4, "version": "GEMM"})
model.save_quantized("Qwen2.5-7B-AWQ")

vLLM serves AWQ with --quantization awq_marlin.

4. GPTQ — Optimal Brain Quantization (Frantar et al., 2022)

Frame quantization as: "given a calibration set of activations, find the int4 weights that minimise the reconstruction error after activation."

Solves it with a clever Hessian-based one-pass algorithm. Uses a small calibration set (~128 sequences) to fit per-channel quantization.

Result similar to AWQ; sometimes better, sometimes worse, model-dependent. Both are widely deployed.

from gptqmodel import GPTQModel, QuantizeConfig
m = GPTQModel.from_pretrained("Qwen/Qwen2.5-7B-Instruct",
    quantize_config=QuantizeConfig(bits=4, group_size=128))
m.quantize(calibration_dataset=calib_examples)
m.save("Qwen2.5-7B-GPTQ")

5. FP8 — native low-precision (Hopper/Blackwell)

H100 has dedicated FP8 tensor cores. Two formats:

• E4M3 (4 exponent, 3 mantissa) — wider range; used for weights and forward activations.
• E5M2 (5 exponent, 2 mantissa) — narrower mantissa; used for gradients in training.

Used in:

• Training: NVIDIA Transformer Engine, DeepSpeed FP8.
• Inference: vLLM --quantization fp8. ~2× speedup at near-bf16 quality on H100.

vllm serve meta-llama/Llama-3.1-70B-Instruct --quantization fp8

For Hopper / Blackwell deployments, FP8 is rapidly becoming the default.

6. GGUF / llama.cpp (CPU and Apple Silicon)

The llama.cpp ecosystem uses GGUF files — a binary format storing quantized weights, tokenizer, and metadata in one file. Quant methods: Q2_K, Q3_K, Q4_K_M, Q5_K_M, Q6_K, Q8_0 (the _K are k-quants, very efficient).

# Convert HF -> GGUF
python convert_hf_to_gguf.py meta-llama/Llama-3.2-3B --outtype f16 -o llama32.gguf

# Quantize
./quantize llama32.gguf llama32-q4_k_m.gguf Q4_K_M

# Run
./llama-cli -m llama32-q4_k_m.gguf -p "Explain attention."

Use GGUF when:

• You need to run on CPU.
• You need to run on Apple Silicon (Metal).
• You want to ship a model with Ollama / LM Studio / Jan / GPT4All.

GGUF is the consumer-facing inference format. Production server-side uses AWQ/GPTQ/FP8.

7. Bitsandbytes — the convenience option

bitsandbytes provides 8-bit and 4-bit quantization integrated with HuggingFace Transformers via a few flags:

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype=torch.bfloat16,
                         bnb_4bit_use_double_quant=True)
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct",
    quantization_config=bnb, device_map="auto")

This is slower than AWQ/GPTQ at inference but is the standard for fine-tuning quantized models (QLoRA — Lesson 4.1).

8. Activation quantization and SmoothQuant

If you also quantize activations (full INT8/INT4 inference), you double the speedup. But activation outliers cause big quality drops.

SmoothQuant (Xiao et al., 2022): scale weights up and activations down by the same factor before quantization, "smoothing" the activation distribution. Enables full W8A8 (weight 8-bit, activation 8-bit).

Used in vLLM with --quantization compressed-tensors for W8A8 / W4A16 / W4A8 deployments.

9. Format selection — the practical decision tree

Are you running on Apple/CPU?
  └─ GGUF Q4_K_M or Q5_K_M.

Hopper or newer GPU and serving at scale?
  └─ FP8 weights + FP8 KV cache (vLLM).

Older GPU (A100, A6000, RTX 4090) at scale?
  └─ AWQ INT4 (or GPTQ).

Tight memory budget on 70B+ on a single GPU?
  └─ AWQ INT4 weight-only.

Need to fine-tune cheaply?
  └─ bitsandbytes 4bit + LoRA (QLoRA).

Need ultimate compression (1.58-bit)?
  └─ BitNet — works only with models *trained* for it.

10. Quality measurement

Always measure the delta:

1. Run a small benchmark (MMLU, HellaSwag, your domain eval) on the bf16 model.
2. Run the same benchmark on the quantized model.
3. Look at perplexity on a held-out corpus.

Acceptable losses:

• INT8 weight-only: ~0%.
• FP8: ~0-0.5%.
• AWQ/GPTQ INT4 group-128: ~0.5-1%.
• INT4 naive: 1-3%.
• Q4_K_M (GGUF): ~0.5-1%.
• Q3_K: 2-4%.
• Q2: noticeable; only for memory-starved cases.

Run a small adversarial set too (long context, code, multilingual). Quantization tends to fail first on these.

Hands-on lab (3 hours, GPU helpful)

quant_lab.ipynb:

1. Quantize Qwen2.5-7B-Instruct with AWQ to INT4. Save and serve with vLLM.
2. Compare perplexity on wikitext-2 between bf16 and AWQ.
3. Convert to GGUF Q4_K_M with llama.cpp. Run with Ollama on your laptop. Verify outputs are similar.
4. Spin up FP8 vLLM (Hopper required). Measure tokens/sec vs bf16.
5. Quantize with bitsandbytes 4-bit + LoRA fine-tune for 200 steps on a small dataset. Confirm training works.
6. Bonus: implement a tiny per-channel symmetric INT8 quantizer from scratch on a single Linear and verify <0.1% perplexity change.

Common pitfalls

1. Quantizing without calibration data — AWQ/GPTQ need a few hundred sequences from the target distribution.
2. Mixing quant formats: don't swap the chat template / tokenizer — keep the original.
3. Forgetting the LM head — quantizing the output projection often hurts. Most libraries skip it by default; verify.
4. Quantizing a fine-tuned model without re-running calibration — calibration mismatch.
5. Running INT4 on hardware without INT4 kernels (older A100s have INT8, not native INT4 matmul) → GPU does it slowly via lookup tables.

Self-check

1. Difference between weight-only and full quantization.
2. What insight powers AWQ?
3. When would you choose GGUF over AWQ?
4. What hardware unlocks native FP8?
5. Acceptable quality loss for INT4 quantization?

References

• Frantar et al. (2022), "GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformers."
• Lin et al. (2023), "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration."
• Xiao et al. (2022), "SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models."
• Dettmers et al. (2022, 2023), "LLM.int8" and "QLoRA."
• Wang et al. (2024), "BitNet b1.58: 1-bit LLMs Future."
• llama.cpp GGUF spec.
• vLLM quantization docs.