Human oversight as a designed process

Oversight is roles, evidence, and a stop

NIST AI RMF 1.0 (NIST AI 100-1, January 2023) subcategory MAP 3.5 says processes for human oversight are defined, assessed, and documented in line with policies from the GOVERN function. GOVERN 2 expects accountability structures so named people are responsible. Appendix C of the same publication asks organizations to differentiate human roles: some systems may run without a person in the loop, and others require oversight. LanePay moves money. It is in the second group. A banner that says “a human is in the loop” is not yet a process.

Write the roles before writing the prompt. Sam may ask LanePay to draft a payment. Priya, the finance owner, may approve a release under a written limit. A separate executor identity may send the approved debit. Maya may review the log after the fact. The model is not in any of those roles. Approval fatigue is part of the design: if every harmless read stops for Priya, Priya will click through the one that pays.

When approval is mandatory

OWASP’s LLM03:2026 prevention guidance uses a graduated policy: low-consequence, reversible actions may proceed under policy, and high-consequence or irreversible ones route to a person. For LanePay, mandatory approval covers creating or changing a payee, releasing a payment, sending mail, and writing any instruction into memory or the retrieval store. Reading one carrier message can stay inside the scoped tool if the credential cannot do those other things.

The reviewer sees the exact action: payee name, routing number, amount, source message id, and the policy version. A model summary such as “routine Northline invoice” is not the action. Invisible or extra characters in the amount field are a reason to show the rendered arguments the executor will use. The approval record stores Priya’s authenticated identity, the arguments she saw, the time, and an expiry. A Slack sentence, a retrieved email, or the model saying “Priya approved” does not create that record. If the approval expires before execution, the debit does not send. Expiry does not silently substitute a different payee.

• Proposer: Sam, or the draft tool acting in Sam’s session.
• Approver: Priya, with a documented limit and a channel that is not the model’s chat.
• Executor: a service identity that can pay only what an unexpired approval names.
• Reviewer: Maya, using tool logs and the approval record, not the model’s recap.

Redesign: LanePay with a gate and scoped tools

The replacement is small enough to draw. read_carrier_mail accepts one mailbox and a message id, with a credential that cannot send, delete, or open payroll. draft_payment returns a structured request and cannot reach the bank. release_payment accepts only a request id that matches an unexpired approval, and its credential can debit only that payee and amount. Memory writes that would store “always pay this routing number” require the same approval path as a payee change.

Sam can still work. LanePay may summarize m-2044 and propose a draft that quotes both routing numbers. Priya approves or rejects the exact draft. The executor verifies the bank’s response, not the model’s claim that the transfer succeeded. That is the success check for this path: the mail-and-payments agent cannot move money on model text, and the tools it can call are narrower than a finance admin. Agentic SOC uses the same approval shape for incident actions. Use that lesson when the case is an investigation. Use this redesign when the case is an application that pays vendors.