AskAiml
Sign in Get started

Deployment, CI/CD, and cloud platforms for agents

Containers, staged deploys, promotion gates, and the CI patterns that catch prompt and model regressions.

🚀 Module 5 8 min read Not started

Why this matters

A production agent is an HTTP service like any other — plus a model, a vector store, an observability surface, a guardrail layer, and a long-running stateful workflow runtime. This lesson teaches the bones of shipping that stack: Docker, Compose, FastAPI, Kubernetes, eval gates in CI, and the new cloud-native agent platforms (AWS Bedrock AgentCore, Azure AI Foundry, Vertex AI Agent Builder, LangGraph Platform).

Learning objectives

  1. Containerise a LangGraph agent with FastAPI.
  2. Use Docker Compose for local prod-mirroring stack.
  3. Push to Kubernetes with Helm.
  4. Add CI/CD with eval gates blocking bad merges.
  5. Pick the right managed agent platform for your case.

1. The reference stack

[client]
   ↓ HTTPS
[nginx / ALB / Cloud Run] ── TLS, rate limit
   ↓
[FastAPI app] ── auth, JSON
   ↓
[LangGraph runtime + workers] ── checkpointer, store
   ↓ tools/MCP                ↓ retrieval
[MCP servers]      [pgvector / Chroma / Qdrant]
   ↓ guardrails
[Llama Guard / Presidio / NeMo]
   ↓ models
[OpenAI / Anthropic API]   AND/OR   [vLLM cluster]
   ↓ telemetry
[LangSmith / OTel / Phoenix / Prometheus / Grafana]

Each box is a Docker container in dev, a deployment in K8s in prod.

2. FastAPI wrapper for a LangGraph agent

python
# app/main.py
from fastapi import FastAPI, HTTPException, Header
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
from .agent import graph                  # your compiled LangGraph
from .config import settings
import uuid, json

app = FastAPI(title="Agent API")

class Ask(BaseModel):
    question: str
    thread_id: str | None = None

@app.post("/v1/ask")
def ask(body: Ask, x_api_key: str = Header(None)):
    if x_api_key not in settings.api_keys:
        raise HTTPException(401, "bad api key")
    config = {"configurable": {"thread_id": body.thread_id or str(uuid.uuid4())}}

    def gen():
        for ev in graph.stream({"question": body.question}, stream_mode="messages", config=config):
            yield f"data: {json.dumps(ev, default=str)}\n\n"
        yield "data: [DONE]\n\n"
    return StreamingResponse(gen(), media_type="text/event-stream")

@app.get("/healthz")
def health():
    return {"ok": True}

Run: uvicorn app.main:app --host 0.0.0.0 --port 8080.

For production:

  • Use gunicorn + uvicorn workers (multi-process), --workers=4 --worker-class=uvicorn.workers.UvicornWorker.
  • Read pool sizing from env (Postgres, Redis).
  • Liveness vs readiness endpoints.
  • Graceful shutdown that drains in-flight requests.

3. Dockerfile (small image, fast cold start)

dockerfile
# syntax=docker/dockerfile:1
FROM python:3.12-slim AS base
ENV UV_LINK_MODE=copy PYTHONUNBUFFERED=1 PIP_NO_CACHE_DIR=1
WORKDIR /app

# uv for fast deterministic installs
COPY --from=ghcr.io/astral-sh/uv:0.5 /uv /uvx /usr/local/bin/

COPY pyproject.toml uv.lock ./
RUN uv sync --frozen --no-dev

COPY app ./app
ENV PATH="/app/.venv/bin:$PATH"
EXPOSE 8080

CMD ["gunicorn","-w","4","-k","uvicorn.workers.UvicornWorker","-b","0.0.0.0:8080","app.main:app","--timeout","60"]

Build & test:

bash
docker build -t my-agent:dev .
docker run --rm -p 8080:8080 --env-file .env my-agent:dev

4. docker-compose for full local prod-mirror

yaml
services:
  api:
    build: .
    env_file: .env
    ports: ["8080:8080"]
    depends_on: [postgres, redis, vllm]

  postgres:
    image: pgvector/pgvector:pg17
    environment:
      POSTGRES_PASSWORD: secret
    volumes: ["pgdata:/var/lib/postgresql/data"]
    ports: ["5432:5432"]

  redis:
    image: redis:7-alpine
    ports: ["6379:6379"]

  vllm:
    image: vllm/vllm-openai:latest
    runtime: nvidia
    command: ["--model","meta-llama/Llama-3.1-8B-Instruct","--api-key","sk-vllm","--gpu-memory-utilization","0.9"]
    ports: ["8000:8000"]

  prometheus:
    image: prom/prometheus
    volumes: ["./prometheus.yml:/etc/prometheus/prometheus.yml:ro"]
    ports: ["9090:9090"]

  grafana:
    image: grafana/grafana
    environment: { GF_SECURITY_ADMIN_PASSWORD: admin }
    ports: ["3000:3000"]

volumes:
  pgdata:

docker compose up and your full stack runs locally.

5. Kubernetes (the production setup)

A minimum production helm chart includes:

  • Deployment (api) with livenessProbe (/healthz) and readinessProbe.
  • HorizontalPodAutoscaler on CPU + custom metric (e.g., vllm:num_requests_waiting).
  • Service (ClusterIP) and Ingress with TLS.
  • ConfigMap + Secret for env.
  • ServiceMonitor (Prometheus operator).
  • NetworkPolicy (egress to model providers, ingress only via gateway).
  • PodDisruptionBudget (minAvailable: 1).

For LangGraph specifically, run multiple stateless API replicas with a shared PostgresSaver so checkpoints survive pod restarts.

For vLLM, see Lesson 5.5's vLLM production stack Helm chart.

6. CI/CD: tests + eval gates

GitHub Actions example:

yaml
# .github/workflows/ci.yml
name: ci
on:
  pull_request:
  push: { branches: [main] }
jobs:
  unit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v3
      - run: uv sync
      - run: uv run ruff check .
      - run: uv run mypy app
      - run: uv run pytest -q

  eval:
    runs-on: ubuntu-latest
    needs: unit
    if: github.event_name == 'pull_request'
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v3
      - run: uv sync
      - run: uv run pytest tests/eval -q
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          LANGSMITH_API_KEY: ${{ secrets.LANGSMITH_API_KEY }}

  docker:
    runs-on: ubuntu-latest
    needs: [unit, eval]
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4
      - uses: docker/login-action@v3
        with: { registry: ghcr.io, username: ${{ github.actor }}, password: ${{ secrets.GITHUB_TOKEN }} }
      - uses: docker/build-push-action@v5
        with:
          push: true
          tags: ghcr.io/${{ github.repository }}:${{ github.sha }}

The eval job is the senior signal. A PR that drops faithfulness or task-success below threshold fails the build. No more "ship and pray."

7. Deployment patterns

  • Rolling updates (default in K8s) — fine for stateless API replicas.
  • Blue/green — run v2 alongside v1; flip at the load balancer.
  • Canary — 5% → 25% → 100% of traffic to v2 with eval-driven rollback.
  • Feature flags (LaunchDarkly / OpenFeature) — enable new prompts/models per cohort.
  • Shadow traffic — duplicate requests to v2 silently for offline comparison.

For agents, canary with online eval gating is the highest-confidence pattern: if task_success on the canary cohort falls below v1, auto-rollback.

8. Managed agent platforms (2026 picks)

PlatformWhen
AWS Bedrock AgentCoreAWS shop, Bedrock models, IAM-tight
Vertex AI Agent Builder / Agent EngineGCP shop; Gemini-friendly
Azure AI Foundry (Agents)Azure / Microsoft 365 ecosystem
LangGraph Platform (LangChain)LangGraph-native; want HIL + cron + studio + traces
Cursor / Devin / Lindy / Replit AgentCoding-centric agents

Decision rule: if your team already lives in one cloud, prefer that cloud's agent runtime — IAM, billing, support, compliance all line up. Otherwise LangGraph Platform is portable.

The non-managed alternative — roll your own with Kubernetes + LangGraph + your gateway — gives the most flexibility but the most ops cost. Many teams run a hybrid: dev on LangGraph Platform, prod on their cloud.

9. Compliance and identity

For regulated deployments add:

  • SSO (SAML/OIDC) for users.
  • Tenant isolation at the DB level (row_level_security in Postgres).
  • Encryption at rest (KMS-managed keys).
  • Audit logging of every tool call with who/what/when.
  • Data residency (region-locked vector stores; pick eu-west or ap-south etc).
  • Data deletion (DSAR / right-to-be-forgotten) — implemented per tenant_id and user_id.

If you can describe these for your project, you are interviewing as a senior.

Hands-on lab (1 day)

Take your full Phase 5 stack and ship it:

  1. Add a FastAPI front and Dockerfile (gunicorn + uvicorn workers).
  2. Add docker-compose.yml with Postgres+pgvector, Redis, vLLM, Prometheus, Grafana.
  3. Build a Helm chart with HPA, probes, ServiceMonitor.
  4. Add a GitHub Actions workflow with unit → eval → docker push → helm upgrade for main.
  5. Add a canary deployment via Argo Rollouts or a simple two-deployment pattern with a 5% Ingress weight.
  6. Demonstrate a rollback when the eval gate fails on a deliberately-bad PR.
  7. Add a terraform module that provisions the cluster + namespaces (or pick AWS Bedrock AgentCore as an alternative path).

Acceptance:

  • One command (make deploy) goes from clean repo to running canary on a real cluster.
  • README has architecture diagram, runbook, and on-call playbook.

Common pitfalls

  1. Stateful agent in a stateless deployment without a checkpointer. State lost on every restart.
  2. No eval gate. PR breaks faithfulness; nobody notices for a week.
  3. Public LLM keys committed. Rotate immediately; use sealed secrets / external-secrets-operator.
  4. No graceful shutdown. In-flight tool calls die mid-flight.
  5. HPAs based only on CPU. Use queue-depth / latency / custom metrics.

Self-check

  1. Why use gunicorn workers in addition to uvicorn?
  2. Why must LangGraph replicas share a PostgresSaver?
  3. What is the difference between blue/green and canary?
  4. What is one custom metric you would scale vLLM pods on?
  5. Pick a managed platform and justify it in 3 lines for your project.

References

Sign in to save your progress and earn badges.