Securing MCP servers in production

Cross-ref (Microsoft stack): In Entra + Agent 365 you get a tenant-scoped MCP catalogue (Lesson 10.8) plus Global Secure Access (also 10.8) for outbound MCP egress filtering. The catalogue enforces the "agents can only call approved MCP servers" pattern natively — no home-grown allow-list required for Copilot Studio + Agent 365 SDK-based agents. Combine with the sandboxing + signing patterns here for defence in depth.

Why this matters

The MCP authorization spec is necessary but not sufficient. Real MCP servers run user data, ship plugins, sandbox subprocesses, and live in supply chains you don't fully control. This lesson covers the production hardening: per-tool authz, output sanitisation, sandboxing, signing, and operational playbooks. Skipping these is how marketplaces get news-headline incidents.

Learning objectives

1. Enforce per-tool authorization with policies.
2. Sanitise tool inputs and outputs.
3. Sandbox MCP server execution (Firecracker, gVisor, Wasm, namespaces).
4. Sign + verify MCP servers (cosign, Sigstore).
5. Build an "MCP allow-list" for an enterprise.

1. Defense in depth: layers on top of OAuth

OAuth gets the right token to the server. Production MCP needs more:

Network → TLS / mTLS / private link
  ↓
Transport → OAuth bearer + DPoP
  ↓
Protocol → JSON-RPC validation, rate limit, JSON-schema enforcement
  ↓
Authorization → per-tool policy (OPA / Cedar)
  ↓
Execution → sandbox (Wasm / gVisor / Firecracker)
  ↓
Output → sanitise / classify / redact
  ↓
Audit → decision logs, replayable trail

Every layer fails sometimes; together they keep the blast radius small.

2. Per-tool authorization with OPA

Authentication tells you who; authorization tells you whether they can do this specific thing right now. Centralise the decision.

import requests

def authz(decision_input: dict) -> bool:
    r = requests.post("http://opa:8181/v1/data/mcp/allow",
                      json={"input": decision_input}, timeout=0.2)
    return r.json().get("result", False)

@mcp.tool()
def delete_repo(owner: str, repo: str, *, ctx):
    if not authz({
        "subject": ctx.user, "actor": ctx.actor,
        "tenant": ctx.tenant, "tool": "delete_repo",
        "args": {"owner": owner, "repo": repo},
        "now": time.time(),
    }):
        raise PermissionError("denied")
    ...

Rego policy:

package mcp

default allow := false

# Repo deletion: only owner-tier roles, only in business hours, with HITL approval
allow {
    input.tool == "delete_repo"
    input.subject_roles[_] == "tenant_owner"
    business_hours(input.now)
    input.approval.granted == true
    input.args.owner == input.tenant
}

business_hours(t) {
    h := time.weekday(t * 1000000000)
    h != "Saturday"; h != "Sunday"
    hour := time.clock(t * 1000000000)[0]
    hour >= 9; hour < 18
}

Policy lives in git, ships through CI, deploys with signed bundles. Decisions are logged (lesson 7.1).

For relationship-heavy access ("can this agent acting for Alice see doc X owned by Bob's team?"), prefer OpenFGA or AWS Cedar (ReBAC and ABAC respectively). Pick one and stick with it.

3. Input validation

JSON-RPC takes any payload. Each tool must have a strict schema and reject anything else.

from pydantic import BaseModel, Field, ValidationError, ConfigDict

class DeleteRepoArgs(BaseModel):
    model_config = ConfigDict(extra="forbid", str_max_length=200)
    owner: str = Field(pattern=r"^[a-zA-Z0-9_-]{1,39}$")
    repo:  str = Field(pattern=r"^[a-zA-Z0-9_.-]{1,100}$")

@mcp.tool()
def delete_repo(**raw):
    args = DeleteRepoArgs(**raw)
    ...

The MCP Python SDK uses Pydantic by default; lock it down with extra="forbid" and constrained types. Don't trust the LLM to give you well-formed input.

Sanitise text inputs for indirect prompt injection: strip unusual unicode (zero-width joiners, RTL marks), normalise (NFKC), cap length, reject control characters. (Deeper coverage in lesson 8.1.)

4. Output sanitisation

The output of a tool flows back to the LLM as context. If the tool returns attacker-controlled text (e.g., a webpage, an email body, a database row inserted by a different user), it can carry an indirect prompt injection payload — instructions that hijack the agent.

Defenses:

• Provenance markers: wrap untrusted content in tagged blocks (<untrusted_source>...</untrusted_source>) and instruct the LLM not to follow instructions inside.
• Strip dangerous markup: HTML, Markdown image/script tags, JavaScript URIs.
• Content classifiers: run output through a small jailbreak/injection detector (Llama Guard, Prompt Guard, vendor moderation APIs).
• Length caps: prevent prompt-stuffing attacks.
• Format enforcement: tools that should return JSON must return JSON only.

For data-classification reasons, redact PII before returning when the agent doesn't need it. Use Microsoft Presidio or your own tagging pipeline.

5. Sandboxing execution

Untrusted code or tools that touch the filesystem should not share the agent process's address space.

Tools, from lightest to heaviest isolation:

• Process namespaces + seccomp + read-only FS — bubblewrap, firejail. Cheap; reasonable for trusted-but-quarantined tools.
• gVisor (Google) — user-space kernel; strong syscall isolation; small overhead. Excellent for general-purpose tool sandboxing.
• Firecracker / Kata Containers — micro-VMs; near-bare-metal isolation; AWS Lambda uses Firecracker. The right answer for multi-tenant, code-running tools (Python REPL, R, browsers).
• Wasm (WASI) — perfect for portable plugin tools you trust the runtime to enforce.
• Browser-based tools (Chromium-in-container) — needed for "agent uses a real browser"; pair with profile isolation per-session + network egress controls.

Network-level: every sandbox should have default-deny egress with an allow-list of destinations needed.

6. Supply chain — signing + verification

MCP servers are software you didn't write. Treat them like any other dependency:

• Pin by digest (acme/github-mcp@sha256:...), not by tag.
• Sign images with cosign (Sigstore). Verify signatures in CI + at deploy.
• Generate SBOMs (syft) and scan with grype / trivy. Reject high-severity unpatched CVEs.
• Provenance (SLSA): use GitHub OIDC + cosign keyless to attest the build.
• Approve new MCP server versions through a "third-party plugin review" process (security + product).

For multi-tenant agent products, maintain a centrally-curated MCP allow-list so end-users can only install vetted servers. Enterprises increasingly insist on this.

7. Rate limiting, quotas, and circuit breakers

A confused or hostile agent can DoS your MCP server or downstream APIs. Mandatory controls:

• Per-client rate limits (token-bucket).
• Per-tenant quotas (e.g., max 1000 tool calls/hour).
• Step budgets enforced by the agent runtime (lesson 6.2).
• Cost caps for downstream API spend per tenant.
• Circuit breakers on dependent services (Polly / Resilience4j / tenacity).

Expose these as observable metrics + alerts; throttled clients should get clear 429 responses with Retry-After.

8. Per-tenant isolation

MCP servers in B2B SaaS run for many tenants. Isolation tactics:

• One database row tag → one row in queries. Enforce with row-level security.
• Per-tenant API keys to downstream third parties; never share.
• Per-tenant filesystem / object storage prefixes; deny-by-default.
• Per-tenant tracing tags so logs can be quarantined / purged on request.

Run periodic cross-tenant attack drills: a test tenant whose calls should never see another tenant's data. Fail loudly.

9. Operational playbook for an MCP server

• Health: /healthz + readiness + liveness probes; failing JWKS fetch ⇒ unhealthy.
• Metrics: tool-call count, latency, deny rate, scope-elevation rate, p99 by tool.
• Logging: structured JSON; request_id, sub, act.sub, tenant, tool, args_hash, decision, outcome.
• Tracing: OpenTelemetry GenAI semantic conventions; spans for mcp.tool.call.
• Alerts: spike in deny, spike in scope-elevation requests, latency SLO breach.
• Disaster recovery: tokens compromised ⇒ rotate JWKS, force re-auth (drop refresh tokens).

Documented runbook for each alert. Test annually.

10. Hands-on lab (4 h)

1. Take the MCP server from lesson 4.1; add an OPA decision call before each tool runs. Write Rego that requires repo:write scope + business hours + same-tenant + (for delete_repo) HITL approval.
2. Add Pydantic schemas for every tool with extra="forbid" and tight regex constraints.
3. Wrap tool execution in gVisor (runsc) by deploying the server as a Pod with the gvisor runtime class on kind/k3d.
4. Sign the server image with cosign sign --yes <image>@<digest>; verify with cosign verify in a CI step.
5. Add a per-tenant rate limiter (fastapi-limiter + Redis) and a per-tool cost cap.
6. Simulate a malicious tool output containing <script> and an attempted prompt injection; verify your sanitiser + classifier reject it.

11. Common pitfalls

1. OAuth in place; per-tool authz missing — broad scopes = full agent compromise.
2. Schemas allowing extra fields → smuggling extra params past validation.
3. Sandbox without egress restrictions → exfil over allowed ports.
4. Cosign signatures created but never verified at deploy.
5. Treating supply chain as a one-time review; servers update silently.
6. Multi-tenant SQL without RLS → cross-tenant via a creative query string.
7. Health endpoint requires auth → no probes.

12. Self-check

1. Layers of an MCP server's defense in depth.
2. OPA vs Cedar vs OpenFGA: pick per scenario.
3. Why output sanitisation is identity-adjacent.
4. Sandbox options ranked by isolation.
5. Cosign verification flow.

13. References

• MCP Authorization spec (current revision).
• Sigstore + cosign docs.
• gVisor + Firecracker docs.
• OWASP "Top 10 for LLM Applications" (esp. LLM02, LLM03, LLM06).
• "Defense in Depth for AI Agents" — Anthropic engineering blog.
• AWS Cedar + OPA documentation.
• SLSA framework (slsa.dev).