Self-hosted serving with vLLM (and Ollama, TGI, SGLang)

Why this matters

Once you fine-tune (Lesson 5.4) or want to host an open model for cost/privacy, you need a high-throughput serving engine. vLLM is the de-facto open-source choice in 2026, with PagedAttention, continuous batching, and an OpenAI-compatible API. It is the reason the open-model economy is now viable — typically 5-20x throughput of generic transformers serving.

Learning objectives

1. Run a model with vllm serve and call it via the OpenAI client.
2. Pick the right quantization (FP8, AWQ, GPTQ, INT4).
3. Use tensor parallelism across multiple GPUs.
4. Add an API key and rate limit.
5. Know when to pick Ollama, TGI, SGLang, or LMDeploy instead.

1. Install and serve in 60 seconds

# WSL2 / Linux box with NVIDIA GPU
uv pip install vllm
vllm serve meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 8000

In another shell:

from openai import OpenAI
c = OpenAI(base_url="http://localhost:8000/v1", api_key="anything")
r = c.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role":"user","content":"Hi"}],
)
print(r.choices[0].message.content)

vLLM is OpenAI-compatible — your existing LangChain / LangGraph / OpenAI Agents SDK code works unchanged.

2. Docker (the production default)

docker run --gpus all --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --gpu-memory-utilization 0.9 \
  --max-model-len 8192 \
  --api-key sk-yourkey-rotate-me

Key flags:

• --gpu-memory-utilization 0.9 — reserve 10% headroom.
• --max-model-len — hard cap on context window.
• --api-key — accept multiple via repeated flags or VLLM_API_KEY.
• --tensor-parallel-size 2 — split a single model across 2 GPUs.
• --quantization fp8 (or awq, gptq).
• --enable-prefix-caching — caches identical prefixes across requests (huge win for chat).
• --enable-chunked-prefill — improves multi-tenant latency.

3. Quantization choices

Rule of thumb: try FP8 if you are on H100/L40S/B200, otherwise AWQ-INT4 for the best quality-per-GB.

vllm serve TheBloke/Llama-3.1-70B-Instruct-AWQ --quantization awq --gpu-memory-utilization 0.92

A 70B AWQ model fits in 2× L40S 48GB comfortably. FP8 needs roughly the same with marginally better quality.

4. Multi-GPU and multi-node

vllm serve mistralai/Mistral-Large-Instruct-2411 \
  --tensor-parallel-size 4 \
  --pipeline-parallel-size 2 \
  --max-num-seqs 256

Tensor parallel splits each layer across GPUs (lower latency, all GPUs talk every step). Pipeline parallel splits layers across GPUs (higher throughput, less inter-GPU traffic). Combine for very large models.

For Kubernetes use vLLM Production Stack (a Helm chart):

helm repo add vllm https://vllm-project.github.io/production-stack
helm install vllm vllm/vllm-stack -f values.yaml

It gives you autoscaling pods, request routing across replicas, and Prometheus metrics out of the box.

5. Hardening for production

TLS + reverse proxy

Run vLLM behind nginx / Caddy / Traefik. Terminate TLS, do per-API-key rate limits, and proxy to vLLM on localhost.

Multi-tenant API keys

Pass several keys in --api-key and route per-key with the gateway. Or front it with LiteLLM Proxy (next paragraph).

LiteLLM Proxy (model gateway)

If you serve multiple models or want one base URL for all:

# config.yaml
model_list:
  - model_name: llama-3-8b
    litellm_params: {model: openai/meta-llama/Llama-3.1-8B-Instruct, api_base: http://vllm-host:8000/v1, api_key: sk-...}
  - model_name: claude
    litellm_params: {model: anthropic/claude-sonnet-4-6, api_key: env:ANTHROPIC_API_KEY}
general_settings: {master_key: sk-master, database_url: postgres://...}

litellm --config config.yaml

LiteLLM adds: per-team rate limits, usage tracking, key rotation, virtual keys, and detailed logs — the production standard "AI gateway" in 2026.

Observability

vLLM exposes Prometheus metrics on /metrics:

• vllm:num_requests_running
• vllm:num_requests_waiting
• vllm:e2e_request_latency_seconds
• vllm:gpu_cache_usage_perc

Three Grafana panels (queue depth, p95 latency, KV-cache utilisation) tell you 90% of operational truth.

6. Alternative serving engines (when not vLLM)

Default pick: vLLM for self-hosted serving, LiteLLM as the gateway in front of everything (self-hosted + paid APIs).

7. Cost math worked example

Suppose your app sends 10M input tokens + 2M output tokens / day to a chat feature.

• GPT-4.1-mini API: 10M × $0.40 + 2M × $1.60 = $4.00 + $3.20 = $7.20 / day (~$216/month).
• Self-hosted Llama 3.1 8B AWQ on a 4090 at ~3000 tok/s: easily fits the workload.
  • Hardware: ~$2/hour cloud GPU = $1440/month, or $0 if on-prem.
  • At 100M tokens/day, the API line jumps; self-hosting wins decisively.

The crossover is typically around 30-100M tokens/day depending on quality requirements and model choice. Below that, paid APIs win on simplicity. Above that, self-hosting wins on cost.

8. Useful runtime tricks

• Speculative decoding with a small draft model: --speculative-model facebook/opt-125m --num-speculative-tokens 5. 1.5-3x speedup on chat workloads.
• Prefix caching is a free win for any chat app — same system prompt across users gets cached.
• Structured outputs via xgrammar or guidance integration — JSON Schema constraints enforced during decoding.
• Tool-calling parsers for popular models (--enable-auto-tool-choice --tool-call-parser hermes).
• Batch requests for embeddings via /v1/embeddings (vLLM serves embedding models too).

Hands-on lab (8 hours)

Take your fine-tuned 8B model from Lesson 5.4 and serve it:

1. Build the model into a Docker image based on vllm/vllm-openai:latest.
2. Add nginx with API-key auth and per-key rate limiting (10 req/sec).
3. Front with LiteLLM Proxy; alias your model + the OpenAI/Anthropic ones behind it.
4. Configure Prometheus scraping vLLM /metrics. Build a Grafana dashboard with queue depth, p95 latency, KV-cache usage, requests/sec.
5. Load-test with locust at 50 concurrent users; record throughput and p95.
6. Run your LangGraph agent against the new local model and re-run the eval suite.
7. Publish a compose.yaml that spins up vLLM + LiteLLM + nginx + Prometheus + Grafana with one docker compose up.

Common pitfalls

1. Underestimating VRAM. A 70B Q4 needs ~40GB; 70B FP8 ~80GB. Plan first.
2. Not setting --max-model-len. Memory blows up.
3. No queue alarms. A backlog goes unnoticed and timeouts cascade.
4. Single replica behind a single nginx. Plan for at least 2 for rolling updates.
5. Forgetting the gateway. Direct vLLM exposure means no auth, no rate-limit, no logs.

Self-check

1. Why is PagedAttention key to vLLM throughput?
2. Difference between tensor parallelism and pipeline parallelism.
3. When does prefix caching save the most?
4. Why use LiteLLM Proxy in front of vLLM?
5. What is the typical break-even point (tokens/day) for self-hosting vs API?

References

• vLLM docs
• vLLM production stack (Helm)
• LiteLLM proxy docs
• HuggingFace TGI
• SGLang
• Ollama
• vLLM quantization guide