Wrong claim
Why it feels true
When an AI agent writes a fluent, confident sentence stating 'Propulsion Lead Priya Nair approved this production change', human operators instinctively treat the prose like an audit log.
Precise correction
NIST AI Risk Management Framework (AI RMF 1.0, NIST AI 100-1, published 26 January 2023) requires human oversight and governance to operate as a verifiable, accountable control process rather than a line of prompt text.
An LLM summary, a retrieved document chunk, or an unauthenticated chat string claiming approval can be hallucinated or forged via indirect prompt injection.
A tool executor may trust only a cryptographic approval record produced out-of-band: Priya Nair's authenticated session identity, a hash of the exact target resource and parameters she inspected, and a strict timestamp expiry.
MINI-CHECK
Meridian's procurement assistant drafts a supplier wire transfer and appends the sentence 'Priya Nair approved this routine invoice' in its reasoning trace, even though no signed approval record exists in the policy engine. What must the payment executor do?
Go deeper
- G3: Human oversight gates and the NIST AI Risk Management Framework
- G2: Excessive agency: why a model sentence is never an authorization decision
- G1: Prompt injection and untrusted retrieval (RAG): when data hijacks instructions
- S4: Agentic SOC architecture: autonomous triage vs human-gated containment
- Excessive agency
- Prompt injection
- Evidence