When Oversight Becomes Forensics

The pilot failed quietly. The agent won’t.

The last two years had a logic to them. Open the door, watch what walks in, learn. Shadow AI proliferated outside procurement, outside IT, outside anyone’s line of sight. Governance lagged. It always does. But the outputs were containable. A bad summary stayed a bad summary. A hallucinated competitor analysis sat in a deck until someone caught it. Damage was bounded by the nature of the thing producing it: a system that generates text, not one that acts in the world.

When an agent acts on your behalf, it doesn’t produce something you review. It does something. Schedules. Procures. Escalates. Commits.

The difference between a bad output and a bad action is not speed, not scale. It is reversibility. You can delete a bad output. You cannot delete what an agent already did.

This is not a timing problem. Organisations have always governed after the fact, learned from incidents, tightened controls in arrears. That pattern works when the thing being governed operates within a known perimeter.

Agents don’t.

The first structural reason is the action space. A trader has position limits. A procurement officer has approval thresholds. The boundaries of consequential action are defined in advance, enumerated, enforced. An agent operating across systems with ambiguous instructions has no equivalent constraint. You cannot write a policy for what you cannot enumerate. And you cannot enumerate in advance what a high-agency system operating at the intersection of your tools, your data, and your workflows will decide to do.

The second reason runs beneath the first. The substrate itself is unstable. Yesterday, OpenAI retired GPT-5.1. Every workflow built on it changed, not because anyone chose to change it, but because the model underneath changed. This is not an edge case. Major labs are shipping significant model updates every two to three weeks. The system your agent runs on today is not the system it will run on in six weeks. In machine learning this is known as CACE: change anything, change everything. In agentic systems it operates at two levels simultaneously. Model updates shift behaviour beneath you without warning. And as agents embed deeper into workflows, the dependencies between them become invisible until they break. When passive systems break, they produce wrong outputs. When active systems break, they take wrong actions.

The third reason is arithmetic. An agentic pipeline executing twenty steps at 97% accuracy per step delivers a final output that is faithful to the original intent 54% of the time. A coin flip. And those steps are not text generation. They are actions. Procurement requests. Calendar commits. Customer communications. Escalations to systems that will themselves act. The error doesn’t sit in a document waiting to be caught. It propagates.

Here is the problem that has no comfortable resolution.

The governance cycle runs slower than the agent action cycle. Not because your governance is poorly designed. Because the cycle times are structurally different and always will be. Human review, escalation paths, audit trails, incident response: these were built for systems that wait. By the time a governance process catches a problem, the agent has acted on it seventeen more times.

Adding oversight doesn’t close this gap. It documents it.

Runtime oversight designed for passive systems is forensics when applied to active agency. It tells you what happened. It does not change what happened.

This is the thing Phase One did not teach you, because Phase One did not require you to learn it. The failure mode was recoverable.

This one isn’t.

The organisations that navigate agency well won’t necessarily be the ones that moved fastest. They will be the ones that knew exactly where they were, why they were there, and what they had decided not to do yet. That clarity is itself a competitive position. It compounds.

Here is the question that stops organisations cold when they first encounter it seriously. When the agent books the meeting, sends the proposal, escalates the ticket: who made that decision? Not who configured the agent. Not who approved the deployment. Who made that specific decision, at that moment, with those consequences attached to it.

Most organisations don’t have an answer. They have an assumption.

The gap between those two things is where the failure lives.

Staging the action space is how you close it. Start with suggest-only: the agent recommends, a human acts. You learn what the agent does with ambiguous instructions, where it interprets broadly, where its judgment diverges from yours. That is not a slow lane. It is the evidence base on which every subsequent decision rests. Then execute-with-approval, where the agent acts on a short enough leash that errors are caught before they propagate. Then guardrail-driven autonomy, where the boundaries are defined not by what you hope the agent will do but by what you have already observed it doing.

At each stage, the decision to advance is explicit. So is the decision to hold. An organisation that has concluded: not yet, for these specific reasons, across these specific workflows, is in a stronger position than one that has simply not gotten around to it. Informed restraint is not the same as delay. It is risk management with a clear view of what’s being managed.

The question about who made the decision also tells you what infrastructure has to exist before any of this begins. Decision rights: who owns the outcomes when the agent acts. Data contracts: what the agent can read, write, and touch, defined in advance and enforced at the boundary. Runtime oversight rebuilt for active systems, not retrofitted from passive ones. Clarity on roles: not what jobs disappear, but what judgment stays human and why.

Without these, you are not making a decision about agency. You are leaving one to be made for you.

The first major public agency failure is not a distant risk requiring imagination to take seriously. The conditions already exist: high-agency systems operating across organisational boundaries, governance infrastructure designed for a different failure mode, model updates arriving faster than any oversight cycle can track. The question is not whether it happens. It is whether your organisation is the one it happens to, or the one that had already drawn the boundary.

The staging structure sits on the Frameworks page.