Secrets management and credential brokering for agents

Why this matters

Agents accumulate credentials: OAuth refresh tokens for the user's Gmail, an API key for the team's Notion, a Stripe restricted key, a database connection string. If any of these live in env vars, code, or "the agent's memory", the inevitable LLM-output leak or compromise becomes a breach. Modern designs broker credentials just-in-time, per-call, scoped narrowly, with the agent never seeing the secret directly.

Learning objectives

1. Use HashiCorp Vault (or cloud equivalent) for centralised secret storage.
2. Generate dynamic, short-lived secrets (Vault DB secrets engine, AWS STS).
3. Build a credential broker between the agent and tools.
4. Use envelope encryption with cloud KMS for per-tenant secrets.
5. Prevent secret material from ever entering LLM context.

1. Secret types in an agent system

Never the same store for "secret material" and "application config". Mixing them is how Bash one-liners leak secrets into logs.

2. Vault basics for agent stacks

# Enable a KV v2 engine for static secrets
vault secrets enable -path=tools kv-v2
vault kv put tools/notion api_key=ntn_dev_xxxx
vault kv get -mount=tools notion

# Enable transit for envelope encryption
vault secrets enable transit
vault write -f transit/keys/tenants
echo -n '{"refresh":"rt_abc"}' | base64 | \
  vault write transit/encrypt/tenants plaintext=-

# Enable a DB secrets engine for dynamic Postgres creds
vault secrets enable database
vault write database/config/postgres \
  plugin_name=postgresql-database-plugin \
  allowed_roles="readonly-agent" \
  connection_url="postgresql://{{username}}:{{password}}@postgres:5432/app?sslmode=disable" \
  username="vault" password="..."

vault write database/roles/readonly-agent \
  db_name=postgres \
  default_ttl=15m max_ttl=1h \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";"

Now vault read database/creds/readonly-agent returns a fresh 15-minute Postgres user. The agent never knows long-lived DB creds.

For dev convenience, vault agent runs alongside your app and renders secrets to files / env vars + auto-renews.

In production you'd use Vault's Kubernetes auth with workload identity (lesson 2.2) so each pod authenticates to Vault using its K8s SA token — no static Vault token ever stored.

3. Cloud equivalents

External Secrets Operator is the canonical bridge for K8s: define ExternalSecret CRDs; the operator pulls from your store and projects into K8s Secrets automatically with rotation. Pair with Sealed Secrets or SOPS only for genuinely-static config you must commit to git.

4. The credential broker pattern

This is the key design idea for agent IAM. Rather than the agent holding secrets, a broker holds them and issues one-shot, scoped tokens per tool call.

Agent runtime
  ↓ (OAuth access token: sub=alice, act=agent.helpdesk-v1)
Credential broker (your code)
  ↓ verifies token, applies policy, fetches user's per-tool creds from Vault
  ↓ optionally mints a short-lived downstream token
  ↓ never returns raw creds to agent
Calls the tool itself on the agent's behalf and returns the *result*
  ↓
Agent receives `{ "tickets": [...] }` — no secret in the path.

The agent runtime never has the raw credential. If the LLM hallucinates "print my Stripe key", the broker simply has no value to leak.

Sketch (FastAPI)

from fastapi import FastAPI, Depends, HTTPException
import httpx, hvac, structlog

app = FastAPI()
vault = hvac.Client(url=VAULT_ADDR, token=VAULT_TOKEN)
log = structlog.get_logger()

def claims_from_token(authorization: str = Header(...)) -> dict:
    ...  # verify JWT, return claims (sub, act, scope, tenant)

@app.post("/tool/notion/search")
def notion_search(req: SearchReq, claims: dict = Depends(claims_from_token)):
    if "notion:read" not in claims["scope"].split():
        raise HTTPException(403)
    # Load the user's connected Notion token, decrypted from vault
    user_secret = vault.secrets.kv.v2.read_secret_version(
        mount_point="tools",
        path=f"users/{claims['sub']}/notion",
    )["data"]["data"]
    r = httpx.post("https://api.notion.com/v1/search",
                   headers={"Authorization": f"Bearer {user_secret['access_token']}"},
                   json={"query": req.query})
    log.info("tool_call", sub=claims['sub'], actor=claims['act']['sub'],
             tool="notion.search", outcome=r.status_code)
    return r.json()

The agent calls the broker; the broker calls Notion. The agent never sees user_secret.

Add: rate limits, structured audit, refresh-token rotation on expiry, OPA decision before action.

Why this is dramatically safer than "MCP server with embedded creds"

Even most MCP servers today have credentials baked at deploy time. A broker decouples agent identity from user credentials and lets you:

• Rotate creds without touching agent code.
• Revoke a user's Notion connection in one place.
• Audit all tool calls in one log.
• Apply HITL approvals per tool call.

For B2B agent products, the broker pattern + thin MCP servers is rapidly becoming the reference architecture.

5. Per-tenant secrets + crypto-shredding

Multi-tenant agents store secrets per tenant. Best practice:

• Envelope encryption: each tenant has its own data-encryption key (DEK). All tenant secrets are encrypted with that DEK. The DEK itself is encrypted (wrapped) by a key-encryption key (KEK) in cloud KMS / Vault Transit.
• Per-tenant KMS keys (or per-customer for high-tier customers): lets you crypto-shred.
• Crypto-shredding: delete the tenant's KEK → all encrypted data is now permanently undecryptable, including in backups and logs. Used to satisfy GDPR "right to erasure" for data that can't easily be physically deleted.

# Pseudocode: AWS KMS
kms.generate_data_key(KeyId=tenant_kek_arn, KeySpec="AES_256")
# → returns plaintext DEK + ciphertext DEK
encrypted = aesgcm.encrypt(nonce, plaintext_secret, aad=tenant_id)
store({"tenant": tenant_id, "ciphertext": encrypted,
       "dek_ciphertext": ciphertext_dek, "alg":"A256GCM", "nonce": nonce})

To delete: aws kms schedule-key-deletion --key-id tenant_kek_arn --pending-window-in-days 7. All ciphertext is now decoupled from any key — practically irrecoverable.

For cross-region replication and disaster recovery, keep separate KEK material per region; never replicate a master root key without strict legal review.

6. Preventing secrets from entering the LLM

Even when the agent doesn't store secrets, they can creep into LLM context:

• Tool errors leaking connection strings.
• Stack traces / debug logs surfaced to the agent.
• Headers (Authorization: Bearer ...) accidentally serialised into the assistant's memory.

Defenses:

• The broker returns only business-level results. Errors are translated to user-safe messages; full stack stays in your logs.
• Use a secret-scanning egress filter on every prompt/response edge (regex + entropy + provider patterns; tools: TruffleHog, Gitleaks, custom).
• Never log raw tokens. Log token hashes if needed for correlation.
• Audit your tracing tools (LangSmith, Langfuse) for secret leakage; redact at the SDK level.

A useful test: try to make your agent print its own headers. If anything sensitive comes back, your runtime is leaking.

7. Secret lifecycle

For each secret type:

• Issuance: who can request, when, via what flow.
• Rotation: cadence (90 days for static API keys; minutes for dynamic).
• Revocation: how to kill, who can trigger.
• Recovery: what to do after compromise (replay-window analysis, dependent system review).
• Audit: who accessed, when, for what.

Have this written per secret type, not for the org as a whole. Test rotation quarterly.

8. Pre-commit secret scanning

A surprisingly large fraction of secrets are leaked through commits. Mandatory:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.21.0
    hooks: [{ id: gitleaks }]
  - repo: https://github.com/trufflesecurity/trufflehog
    rev: v3.84.0
    hooks: [{ id: trufflehog, name: TruffleHog scan }]

Plus a server-side webhook on GitHub / GitLab that blocks pushes containing detected secrets.

9. Hands-on lab (3 h)

1. Spin up Vault (lesson 0.1). Configure:
   • KV v2 at tools/users/<sub>/notion for OAuth tokens.
   • DB secrets engine for dynamic Postgres creds (15-min TTL).
   • Transit engine for envelope encryption of tenant blobs.
2. Build a FastAPI credential broker with one tool route (/tool/notion/search).
3. Make a sample agent (with requests) that calls the broker using its OAuth token; verify the agent never sees the Notion secret.
4. Demonstrate dynamic DB creds: call /tool/db/query; verify Vault created a temporary Postgres user that expires in 15 min.
5. Implement crypto-shredding: encrypt a tenant's secret with a tenant-KEK; delete the KEK; show the ciphertext is unrecoverable.
6. Add gitleaks pre-commit; deliberately try to commit a fake AWS key; verify blocked.

10. Common pitfalls

1. Agent process reads secrets at boot and holds in memory — any debug dump leaks.
2. Single Vault token shared across many agents — no audit per agent.
3. Per-user OAuth tokens kept long after user offboarding.
4. KMS keys without rotation; one compromise = years of replay.
5. Dynamic secrets without enforced TTLs; agents keep them around indefinitely.
6. Pre-commit hooks not enforced server-side; bypass with --no-verify.

11. Self-check

1. Why dynamic DB creds beat static.
2. The credential broker pattern in 3 sentences.
3. Envelope encryption + crypto-shredding.
4. Two ways secrets leak into LLM context.
5. External Secrets Operator's role.

12. References

• HashiCorp Vault docs.
• External Secrets Operator docs (external-secrets.io).
• AWS Secrets Manager / GCP Secret Manager / Azure Key Vault docs.
• "Cryptographic Erasure" — Microsoft + NIST guidance.
• TruffleHog, Gitleaks docs.
• Sigstore "OpenSSF Best Practices for OSS Maintainers".
• "Zero Trust Architecture" — NIST SP 800-207.