Governance and compliance for agent systems

Cross-ref (Microsoft stack): Microsoft implements the identity-lifecycle half of these governance frameworks via Entra ID Governance for agents: sponsors + owners with auto-transfer on JML, access packages, lifecycle workflows, access reviews, and the M365 admin center agent registry (with the Unmanaged / Shadow / Agents-at-risk views). See Lesson 10.7. Combine with Lesson 10.5 (Conditional Access) and Lesson 10.6 (ID Protection) for the enforcement layer — that composition is your ISO 42001 A.6 + EU AI Act Article 10-14 evidence in the Microsoft world.

Why this matters

In 2026, agent products face a stack of governance regimes that didn't exist three years ago: EU AI Act (high-risk + GPAI obligations), NIST AI RMF, ISO/IEC 42001 (AI Management Systems), sectoral overlays (HIPAA, PCI, DORA, Colorado AI Act, NYC LL144). Engineering decisions you make now determine whether passing an audit is a Friday afternoon or a six-month rebuild.

Learning objectives

1. Map agent-system requirements across the four major frameworks.
2. Translate "the EU AI Act" into engineering controls.
3. Run an internal agent review board.
4. Build the documentation set (Model Card, Datasheet, Risk Register, Agent Card).
5. Demonstrate compliance with the audit + monitoring stack from earlier lessons.

1. The frameworks in one paragraph each

• EU AI Act (2024 → staged through 2027) — risk-tiered obligations. Prohibits some uses; heavy duties for "high-risk" systems (HR, credit, education, biometrics, critical infra, law enforcement, justice); separate duties for general-purpose AI / foundation models.
• NIST AI RMF 1.0 + Generative AI Profile (2023-24) — voluntary, framework-style. Four functions: Govern, Map, Measure, Manage. Pragmatic; the lingua franca for US enterprises.
• ISO/IEC 42001:2023 — AI Management System (analogue of ISO 27001 for AI). Auditable; certifiable. Increasingly demanded by enterprise customers.
• SOC 2 / ISO 27001 — security + privacy management. Not AI-specific but every AI vendor needs these; many controls reused.

Sectoral overlays: HIPAA (health), PCI DSS (payments), DORA (EU finance operational resilience), Colorado AI Act + NYC Local Law 144 (HR), CCPA / GDPR (privacy). When applicable, they layer on top.

You don't pick one — you accumulate them. Build a control matrix once; map to each framework.

2. The EU AI Act for agent engineers

Risk tiers — quick read

• Prohibited: social scoring, untargeted scraping of facial images, real-time biometric identification in public (narrow LE exceptions), emotion recognition in workplace + education, manipulative AI exploiting vulnerabilities.
• High-risk: HR / credit / education / law enforcement / justice / critical infra / biometrics / safety components.
• General-purpose AI (GPAI) / foundation models: transparency + copyright + training-data summary + evaluation; additional systemic-risk obligations for very large models.
• Limited risk: chatbots, deepfakes — transparency obligations only.
• Minimal risk: most agents that don't decide impactful things.

Many agentic products straddle tiers. A customer-support agent: limited risk. The same agent given authority to grant refunds + close accounts: arguably high-risk for the affected user. Document your tier choice + reasoning.

Engineering implications for high-risk

For high-risk agent systems, you must demonstrate:

In practice this is a long binder of evidence. The audit pipeline + decision logs from lessons 6-8 are most of the evidence; governance is connecting them.

GPAI obligations (if you train your own model)

• Technical documentation.
• Summary of training data.
• Copyright compliance policy.
• Evaluation results.
• For "systemic risk" models (very large): adversarial testing, incident reporting, cyber + biosecurity assessments.

Track which exemptions apply to open-weight models you deploy (different obligations from those who trained them).

3. NIST AI RMF — operationalising

Four functions, each broken into categories + subcategories. The agent-relevant slice:

• Govern: AI risk policy; roles + responsibilities; risk-aware culture; oversight body (your agent review board).
• Map: characterise the agent's purpose, users, context, stakeholders, risks.
• Measure: capabilities, performance, robustness, fairness, security, drift.
• Manage: risk treatment plans, residual risk acceptance, change management, incident response.

You don't need to memorise the taxonomy. Use it as a checklist when scoping a new agent and during quarterly reviews.

The Generative AI Profile layers on subcategories specific to LLMs: confabulation, value chain risks, harmful content, data privacy, environmental impacts.

4. ISO/IEC 42001 — the management system

ISO 42001 is structured like ISO 27001: an Annex A of controls + a Plan-Do-Check-Act loop.

Key control families for agents:

• A.2 Policies for AI.
• A.4 Internal organisation, roles.
• A.5 Resources (data, tooling, model lifecycle).
• A.6 AI system lifecycle.
• A.7 Data for AI.
• A.8 Information for interested parties (transparency).
• A.9 Use of AI systems (HITL, monitoring).
• A.10 Third-party (suppliers, customers).

If you already do SOC 2 well, ISO 42001 is incremental. The certification process takes 6-12 months; budget accordingly.

5. Translating compliance into engineering

Map every obligation to a control + an artefact. Example slice:

Maintain this matrix in a wiki, owned by the governance lead, referenced by engineering.

6. The Agent Card

Borrowed from the Model Card / Datasheet tradition. One page per registered agent (lesson 2.1). Living document. Suggested structure:

# Agent Card — Helpdesk Agent v1

## Identity
- agent_id: agent.helpdesk-v1
- owner: support-platform-team
- version: v1.4.2
- model: claude-opus-4.7 (via Anthropic API)
- system_prompt_version: v17

## Purpose
Helps tier-1 support reps draft replies, look up tickets, and (with approval) refund up to $500.

## Capabilities
| Tool | Scope | Risk tier | HITL? |
|------|-------|----------|-------|
| tickets.search | tickets:read | low | no |
| tickets.update | tickets:write | medium | session-consent |
| refunds.create (≤$500) | payments:write + RAR | high | CIBA + WebAuthn |

## Identity model
Delegation (Token Exchange). `act` = agent.helpdesk-v1, `sub` = end user.
Workload identity via SPIFFE; OAuth via Keycloak realm `agents-prod`.

## Data
Reads: tickets (per-tenant), CRM (per-tenant), KB.
Writes: tickets, refund requests.
Does NOT access: payroll, source code, customer health data.

## Risks + controls
Top 5 risks (cross-tenant query, indirect injection via attached PDFs, ...) →
see risk register entries THR-2026-014 .. THR-2026-018.

## Monitoring
SLOs: 99.5% decision-log delivery; <1% deny rate; <0.1% scope-elevation.
Dashboards: see grafana://agents/helpdesk-v1.
On-call: support-platform-eng.

## Approval + governance
Reviewed by Agent Review Board on 2026-04-12.
Next review: 2026-10-12 or on material change.

## Change log
v1.4.2 (2026-06-10) — added Spanish support, expanded scope `kb:read`.
v1.4.1 — bug fix.
v1.4.0 — added refund tool with CIBA gate.

Auditors love this document. Engineers should too — it forces clarity.

7. The Agent Review Board

A standing cross-functional body that approves new agents and material changes to existing ones.

• Members: engineering lead, security, product, legal, often a domain expert per use case.
• Cadence: weekly review of new agent proposals; quarterly review of all agents.
• Inputs: Agent Card, threat model, risk register, eval results, proposed scopes, monitoring plan.
• Outputs: approval (with conditions) or rejection with rationale; recorded in a register.

Without this body, governance becomes ad-hoc and reactive. With it, you have a defensible answer to "who decided this agent could do that?".

For low-risk agents, a lightweight async approval suffices; reserve the full board for high-risk / customer-facing.

8. Incident reporting

Define what counts as an incident:

• Confirmed unauthorized access via an agent.
• Cross-tenant data leakage.
• Hallucinated action that caused customer harm.
• HITL bypass.
• Supply-chain compromise affecting deployed agents.

For each: internal post-mortem within X days, customer notification per contract / law, regulator notification where required (e.g., EU AI Act serious incident reporting for high-risk; GDPR 72-hour breach notification).

Practice the response with annual tabletop drills.

9. Vendor / customer obligations

If you ship agents to customers:

• Sub-processor list updated for every model / IdP / MCP provider used.
• DPA / BAA signed where required (HIPAA, GDPR processor obligations).
• Customer-facing transparency notes: which model versions, which retention.
• No-train clauses with model providers.
• Audit rights: enterprise customers will ask for SOC 2 / ISO 42001 reports.

If you consume agent products:

• Vendor security questionnaire with agent-specific questions (identity model, log retention, MCP servers used, scope manifest).
• Pen-test results + remediation status.
• Right to revoke + data-deletion clause.
• AI Act conformity evidence where applicable.

Standardise these into a vendor-management process. Don't negotiate from scratch each time.

10. Hands-on lab (3 h)

1. Pick one agent (real or designed). Score it across:
   • EU AI Act tier (with reasoning).
   • NIST AI RMF Govern/Map/Measure/Manage gaps.
   • ISO 42001 Annex A applicability.
   • Sectoral overlays.
2. Draft its Agent Card in Markdown.
3. Build a control matrix (10-15 rows) tying obligations to lessons + artefacts.
4. Write a 1-page Risk Register entry for the top risk.
5. Draft an Agent Review Board charter (members, cadence, decision process, escalation, record-keeping).
6. Plan an incident drill: pick a scenario, write the runbook, schedule on the team calendar.

11. Common pitfalls

1. Compliance project run by legal alone — engineering controls become unrealistic.
2. Annual "binder" approach — no living artefacts; useless six months later.
3. Treating EU AI Act high-risk as "later" — preparation takes 12 months.
4. No incident response for AI-specific scenarios.
5. One Agent Card for the whole org; not per agent.
6. Confusing model card (model) with agent card (deployed agent + tools + policies).
7. No board sign-off → "shadow agents" proliferate.

12. Self-check

1. EU AI Act tiers in order.
2. NIST RMF four functions.
3. Three artefacts every high-risk agent must have.
4. The Agent Review Board's role.
5. Why ISO 42001 is being demanded by enterprise customers.

13. References

• EU AI Act final text (eur-lex.europa.eu) + Commission FAQs + guidance.
• NIST AI RMF 1.0 + Generative AI Profile (nist.gov/itl/ai-risk-management-framework).
• ISO/IEC 42001:2023 — AI Management System.
• ISO/IEC 27001 + SOC 2 Trust Services Criteria.
• Colorado AI Act; NYC Local Law 144; California AB 2013; UK ICO AI guidance.
• "AI Governance in Practice" — World Economic Forum + OECD AI Principles.
• ENISA "Multilayer framework for good cybersecurity practices for AI".
• Google "Secure AI Framework (SAIF)".
• HuggingFace "Model Cards" + Google "Model Cards for Model Reporting".