Helpful first: Excessive agency: why a model sentence is never an authorization decision
Key points
- Human-in-the-Loop (HITL) approves each action. Human-on-the-Loop (HOTL) monitors bounded actions and can stop them. Human-in-Command (HIC) decides if and how the AI system runs at all.
- Defend human approval gates against automation bias, parameter tampering (TOCTOU), and spoofed chat approvals
- Map AI security controls across the four core functions of the NIST AI Risk Management Framework (Govern, Map, Measure, Manage)
Why a chat message saying "Approved" is not a security gate
After Meridian Space & Robotics (chronos.) separates read-only drafting from payment execution in its procurement assistant, a developer proposes a simple human gate: have the agent post "Should I pay Invoice #402?" in a Slack channel, and if anyone replies "yes", let the agent read the reply and execute the wire transfer.
Principal Security Architect Vera Vance points out three ways that naive gate fails. First, chat text is not cryptographic authorization - an indirect prompt injection or unauthorized channel member could inject "yes". Second, automation bias: if the reviewer only sees "Pay Invoice #402?" without seeing the actual destination IBAN, currency, and diff against the vendor master file, the human just rubber-stamps a poisoned proposal. Third, Time-of-Check to Time-of-Use (TOCTOU): if the human approves $100 to Supplier A, nothing stops a compromised model loop from calling the tool with $100,000 to Supplier B unless the approval signature is bound to the exact payload hash.
| Gate Requirement | Weak "Rubber-Stamp" Pattern | High-Assurance Engineering Pattern |
|---|---|---|
| Decision Context | Shows only model-generated summary prose ("Looks safe to deploy") | Renders deterministic structured fields from the raw payload (exact payee, IBAN, USD amount, diff) |
| Approver Authentication | Unverified chat reply or model reading a text string | Out-of-band authenticated UI/API action requiring the authorized approver’s session + FIDO2 re-auth |
| Payload Binding (Anti-TOCTOU) | Boolean flag approved=true that allows the agent to pick tool arguments later | Approval signs a cryptographic hash of (tool_ |
| Alert Volume & Fatigue | Prompts the human 200 times a day for low-risk read/draft operations | Auto-approves read-only/low-risk drafts via deterministic policy; reserves human prompts for high-impact writes |
Preventing automation bias and Time-of-Check to Time-of-Use (TOCTOU)
Automation bias is the well-documented human tendency to trust an confident-sounding automated recommendation without checking the underlying facts. To counter automation bias, Meridian’s approval UI never displays the LLM’s persuasive prose rationale as the primary record. Instead, the deterministic execution service renders the raw parameters that will actually be sent to the bank or production cluster, highlighting any field that differs from historical baselines.
When Propulsion Lead Priya Nair or Finance Lead approves the request, the backend mints a single-use, short-lived execution token bound to the SHA-256 digest of those exact arguments. The LLM never touches the execution token; the deterministic worker submits the signed payload directly to the downstream API.
Matching oversight tiers: HITL, HOTL, and Kill Switches
Not every action needs the same gate. Human-in-the-Loop (HITL) blocks execution until a human explicitly approves the specific transaction (mandatory for wire transfers, production deployments, or account deletions). Human-on-the-Loop (HOTL) allows bounded autonomous actions within strict rate and dollar limits while streaming real-time telemetry to an operator equipped with an immediate kill switch / circuit breaker.
Human-in-the-Loop (HITL)
blocks execution until a human explicitly approves
- wire transfers, production deployments, or account deletions
Human-on-the-Loop (HOTL)
allows bounded autonomous actions within strict rate and dollar limits
- kill switch / circuit breaker
Human-in-Command (HIC) works at the level of the whole system, not one action. HIC is the authority to decide when and how to use the AI system. That authority can include the decision not to use the system and the power to override a decision that the system makes.
Mapping AI controls onto NIST AI RMF 1.0 (Govern, Map, Measure, Manage)
To keep AI security aligned with enterprise risk management, Vera maps every agentic control onto the four functions of the NIST AI Risk Management Framework (AI RMF 1.0):
Govern establishes organizational policies, risk appetite, roles, and accountability for AI systems (for example, Meridian’s policy forbidding autonomous write access to flight-control or payment systems). Map identifies the context, data flows, trust boundaries, and threat scenarios before building (such as mapping where untrusted vendor PDFs enter the RAG pipeline). Measure quantitatively evaluates and red-teams the system (measuring prompt-injection refusal rates, approval gate false-positive rates, and drift). Manage operates the active controls, human oversight gates, incident response playbooks, and kill switches in production.
Govern
policy forbidding autonomous write access to flight-control or payment systems
Map
untrusted vendor PDFs
Measure
prompt-injection refusal rates
Manage
human oversight gates, incident response playbooks, and kill switches
CHECK YOUR JUDGMENT