REFERENCE · §6 · LAST REVIEWED 2026-04-27

ACF §6 — Performance Monitoring

Agent performance monitoring is the continuous observation of accuracy, drift, hallucination rate, instruction adherence, and downstream business-outcome quality of an autonomous agent — calibrated to detect regressions that produce regulatory breaches before they accumulate.

Traditional model risk management (SR 11-7, PRA SS1/23) measures static-model performance against a hold-out set. Foundation models are not static. Provider updates, prompt changes, tool changes, and customer-input distribution shifts all move the agent’s performance daily. The framework section requires golden-set evaluation runs at defined cadence, drift detection on real-traffic samples, hallucination spot-checks for high-stakes actions, and breach-rate alerting before regulatory thresholds are crossed.

Regulatory anchors

• PRA SS1/23
• SR 11-7
• EU AI Act Art. 17
• ISO 42001 §9
• NIST AI RMF Measure 2

What this covers

• Golden-set evaluation runs and frequency
• Real-traffic sampling for drift detection
• Hallucination spot-check protocol for high-stakes actions
• Breach-rate monitoring per regulated obligation
• Trigger conditions for human escalation

Common gaps

• Eval set is from launch and never refreshed
• No drift detection — performance only checked when something visibly breaks
• Hallucination "rate" undefined; reported as "rare"
• Monitoring covers technical metrics but not regulatory-breach proxies

Related sections

• §2 — Action Perimeter

  What an agent must NOT do — technical and policy guardrails on action space.

• §3 — Audit Trail

  Every agent action logged with sufficient detail to reconstruct intent and outcome.

• §5 — Vendor Due Diligence

  Foundation model providers, MCP servers, tool authors — third-party risk for the agent stack.

• §7 — Operational Resilience

  What happens when the agent goes down mid-transaction.

Reference compiled by Sebastian Heine. Editorial perspective at The SHeine Brief.