Failure Modes and Intervention

An AI Native operating model will fail.

Not because the premise is wrong.

Because every operating model fails somewhere.

Work accumulates where capacity was misjudged. Authority proves too narrow for the work it governs. Controls create delay. Organizational boundaries interrupt execution. Management systems continue long after the conditions that justified them have disappeared.

The difference in an AI Native enterprise is the speed, distribution and opacity with which these failures can develop.

Machine actors can execute continuously. They can initiate work, call tools, coordinate with other systems and produce downstream actions at a rate that human operating structures were not designed to absorb. A poorly designed workflow can therefore become more efficient at producing the wrong thing. A badly placed approval can turn machine speed into a human queue. An agent with insufficient authority can escalate almost everything. An agent with excessive authority can propagate an error before a human understands that the operating condition has changed.

The problem is not simply whether the AI works.

The problem is whether the operating model works.

That distinction changes how failure must be understood.

A model error is a technical event.

An operating failure is structural.

It appears when the allocation of work, authority, management, organization, execution architecture, infrastructure or economics no longer produces the intended enterprise outcome.

The enterprise therefore needs more than AI monitoring.

It needs the ability to observe its own operating model.

The first generation of enterprise AI measurement has concentrated heavily on the machine.

These measures matter. None is sufficient.

An agent can perform its assigned task successfully while the workflow around it deteriorates.

Consider an agent that prepares a customer credit decision in thirty seconds instead of thirty minutes. If every recommendation still enters a human approval queue that requires four hours to clear, the agent has improved a task without materially improving the workflow.

The machine succeeded.

The operating model did not.

Or consider an agent that resolves most service requests autonomously but generates enough uncertain cases to double the workload of a specialist exception team.

Again, local performance can improve while system performance declines.

This is the central diagnostic problem of AI Native operations.

Local machine success does not establish operating success.

Production evidence already points toward this distinction. A large 2025 study of practitioners operating AI agents in production found that reliability remained the leading development challenge. Most production systems remained deliberately constrained. Sixty-eight percent executed no more than ten steps before human intervention, while 74 percent relied primarily on human evaluation. The evidence does not suggest that useful agents cannot be deployed. It suggests that production autonomy remains an operating design problem, not merely a model capability problem.

The same tension appears in enterprise evaluation. A 2026 survey of 157 enterprise respondents found that half of organizations running evaluations had experienced an AI feature that passed internal evaluation and subsequently produced a customer-facing failure. The sample is directional rather than representative of the entire enterprise market, but the finding exposes an important weakness: passing an evaluation is not equivalent to surviving the operating environment.

The environment contains conditions the test does not.

The operating model is where these conditions meet.

Failure must therefore be treated as evidence about the design of that system.

A failure observed at one layer may originate somewhere else.

The purpose of diagnosis is to locate that difference.

The location of the symptom is not necessarily the location of the intervention.

The most accessible form of enterprise AI transformation is task substitution.

A person performed an activity.

A machine can now perform some or all of it.

The activity is automated.

This can create genuine value. It can also preserve nearly every structural feature of the operating model that made the activity expensive in the first place.

Only one participant becomes faster.

This is task automation without workflow redesign.

It is one of the most consequential failure modes because early performance can make the transformation appear successful.

A machine may reduce the execution time of an individual task by 80 or 90 percent while producing little change in end-to-end cycle time.

The reason is straightforward.

The task was not the system.

If a workflow contains six stages and only one becomes dramatically faster, the remaining stages determine how much of that speed reaches the outcome. The bottleneck moves.

This is not unique to AI. It is a basic property of systems. What changes with AI is the magnitude of the mismatch. Machine execution can compress work so sharply that previously tolerable coordination structures become dominant constraints.

The enterprise then discovers that it did not automate a workflow.

It accelerated traffic into the next queue.

Gartner reached a related conclusion in its 2025 assessment of agentic AI adoption, warning that organizations were misapplying agents and that many implementations would struggle with cost, business value and risk controls. Its recommendation was notable: where agentic systems are appropriate, enterprises may need to rethink workflows rather than simply integrate agents into existing processes.

That distinction is fundamental.

AI adoption asks:

Where can AI perform an existing task?

AI Native transformation asks:

If this capability existed when the workflow was designed, would we design the workflow this way at all?

Those questions produce different enterprises.

Human oversight is necessary.

Human approval everywhere is not.

The distinction becomes critical as machine execution scales.

Human approval is often introduced as the default response to uncertainty. An agent can perform the work, but a person must approve the result before anything happens.

For low-volume systems, this can be sensible.

At scale, it can reproduce the very coordination architecture that machine execution was intended to change.

As machine throughput increases, approval demand increases with it. The human actor becomes a capacity-constrained resource positioned directly inside the execution path.

The result is an approval bottleneck.

This does not mean removing human control. Research from Microsoft and production evidence elsewhere support continued human involvement, particularly where agents operate imperfectly or actions carry meaningful consequences.

The design question is where that involvement belongs.

Not every action requires the same control.

The operating failure occurs when the enterprise uses human approval as a universal substitute for authority design.

That choice appears safe because every action remains supervised.

Operationally, it can create a system incapable of absorbing its own machine capacity.

The intervention is not automatically fewer controls.

It is better placement of control.

An agent cannot create operating leverage if it must repeatedly ask permission to perform the work it was designed to execute.

This is authority failure in its restrictive form.

The enterprise delegates execution but withholds the decision rights required to complete it.

The result is machine capacity trapped behind human authorization.

This produces a characteristic operating signature.

The enterprise appears to have introduced autonomy while preserving human execution through a different interface.

This is delegated labor without delegated authority.

It creates what looks like human oversight but often functions as machine underutilization.

The appropriate authority boundary depends on consequence, reversibility, uncertainty and the operating environment. A payment agent should not necessarily possess the same discretionary envelope as an internal research agent. An infrastructure remediation agent should not necessarily have unrestricted production access because it can diagnose an incident.

But if every meaningful action requires escalation, the machine is not operating autonomously.

It is generating requests for humans.

At sufficient scale, those requests become work.

The opposite failure is more dangerous.

Machine execution can propagate decisions much faster than traditional human structures.

An agent with excessive authority can therefore convert a local error into a system event.

The issue is not simply that an agent may make a bad decision.

Humans make bad decisions.

The operating distinction is the combination of speed, replication and access.

A machine actor may be able to execute the same decision hundreds or thousands of times. It may interact with multiple systems. It may initiate downstream workflows. Other agents may consume its output and continue execution.

A mistake can therefore acquire reach.

This makes delegated authority inseparable from containment.

Authority must have boundaries.

• Financial limits.
• System boundaries.
• Action classes.
• Data permissions.
• Escalation conditions.
• Reversibility requirements.
• Time constraints.
• Confidence or uncertainty conditions where appropriate.

The objective is not minimal autonomy.

It is bounded autonomy.

Enough authority to complete meaningful work.

Enough constraint to prevent a single actor from creating disproportionate enterprise consequences.

The operating model fails when either side dominates.

Too little authority and the enterprise creates queues.

Too much authority and it creates uncontrolled propagation.

The design problem is not whether machines should possess authority.

It is how much authority each machine actor requires to produce the outcome safely and efficiently.

Automation changes the composition of human work.

As routine execution moves to machines, the residual work left for humans becomes less representative of the original process.

It becomes concentrated in exceptions.

This is expected.

The failure begins when exception volume exceeds the human system designed to absorb it.

A workflow can therefore automate a high percentage of its transactions and still become operationally worse.

Suppose an agent handles 90 percent of cases autonomously.

That sounds successful.

But if the remaining 10 percent previously represented 2 percent of specialist workload and now arrives continuously, poorly classified and with insufficient context, the exception function may collapse under the new demand.

The percentage automated is not the relevant measure.

The relevant measure is whether the complete operating system can absorb the residual work.

These become operating measures.

Repeated exceptions also carry diagnostic information.

An exception that occurs once may be noise.

An exception that occurs repeatedly may indicate that the operating model has defined the normal path incorrectly.

At that point, the enterprise should not simply add people to the exception queue.

It should ask whether the workflow, authority boundary, context architecture or policy needs to change.

Exceptions are not merely work to be processed.

They are evidence about where the operating model has not yet learned.

The productivity narrative around AI often assumes a simple substitution.

Human work decreases because machine work increases.

Real operating systems are less tidy.

Automation can remove work in one location while creating it somewhere else.

The gross amount of automated work therefore says little about net operating leverage.

This is why economic measurement must follow the workflow rather than the feature.

The correct question is not:

How much work did the agent perform?

It is:

What work disappeared, what work moved and what new work was created because the agent exists?

Only then can the enterprise determine whether automation actually reduced the resources required to produce the outcome.

This becomes particularly important as agent systems grow more sophisticated.

Machine execution is not free simply because human labor is absent.

Every machine actor enters the enterprise with an operating footprint.

That footprint must be justified by the value it creates.

One of the most persistent transition failures is organizational rather than technical.

The enterprise introduces machine execution but preserves the management system designed for human execution.

The new system is added without removing the old one.

The enterprise now operates two management systems.

One directs execution.

The other describes, checks or recreates it.

This is the management double system.

It is easy to create because removing management mechanisms feels riskier than adding technology.

During transition, duplication may be necessary.

The enterprise may need parallel controls while confidence develops.

But temporary redundancy can harden into permanent structure.

When it does, the organization pays twice.

It pays for machine execution and for human coordination designed around the assumption that machines do not execute.

The operating model has changed underneath the management model.

Management has not followed.

This produces a deeper diagnostic question:

Which management activities still exist because they improve the outcome, and which exist because the previous operating model required them?

The answer cannot be determined from the organizational chart.

It must be determined from the workflow.

As agents become easier to create, the enterprise acquires a new form of capacity problem.

Too many machine actors.

Soon the architecture contains a population of machine actors whose existence is easier to explain individually than collectively.

More agents do not necessarily produce a better operating system.

Anthropic's experience building its multi-agent research system provides unusually concrete evidence. Multi-agent architectures improved performance for tasks that benefited from parallel exploration, but they carried substantial cost. Anthropic reported that agents used roughly four times the tokens of ordinary chat interactions and multi-agent systems roughly fifteen times as many. The company also found that some tasks were poor candidates for multi-agent architecture, particularly when agents required shared context or had tightly coupled dependencies.

Its early systems exposed another failure mode. Agents spawned excessive numbers of subagents, duplicated searches and continued working after sufficient evidence had already been collected. Anthropic introduced explicit scaling rules to align agent effort with task complexity.

The lesson extends beyond research systems.

Agent count is an architectural decision, not a maturity metric.

The enterprise should use multiple agents when decomposition creates a meaningful advantage.

If those conditions do not exist, additional agents may simply create coordination overhead.

Every additional machine actor introduces another context boundary, another execution path, another source of latency, another set of permissions, another evaluation surface and another possible failure relationship.

Agent sprawl is therefore not merely a technical cleanliness problem.

It is operating complexity.

And complexity must earn its place.

Every additional component must earn its place in the operating model.

Complexity is justified when it produces greater capability, resilience, control or economic value.

When it does not, simplification is itself an intervention.

Finding failure is not enough.

An enterprise can instrument every agent, trace every execution path and record every exception while remaining incapable of improving the operating model.

Observation produces evidence.

Intervention changes the system.

The distinction matters because many failures that appear technical are symptoms of operating design decisions made elsewhere.

An agent repeatedly requesting human approval may not have an agent problem.

It may have an authority problem.

A workflow generating thousands of exceptions may not have an exception handling problem.

Its allocation of work may be wrong.

A multi-agent system producing inconsistent outcomes may not require a better orchestration framework.

It may contain unnecessary machine actors.

An automation producing impressive task-level performance while increasing downstream workload may not have failed technically at all.

The workflow has failed economically.

The purpose of intervention is therefore not to repair every visible defect.

It is to identify the structural condition producing the defect and change the smallest part of the operating model capable of correcting it.

That requires a different operating discipline.

Every failure has a location where it becomes visible.

That location is not necessarily where the intervention belongs.

Consider a customer onboarding workflow.

An agent prepares an account configuration. A human must approve it. Approval queues begin to grow.

The visible failure occurs at the approval stage.

The enterprise could add reviewers.

That would increase capacity at the point where the failure appears.

But the underlying problem may be that every configuration requires approval regardless of risk.

The intervention point may therefore sit upstream, in the authority architecture.

The queue was real.

The queue was not the problem.

The operating rule that created the queue was the problem.

This distinction separates operational remediation from operating-model intervention.

Remediation restores execution.

Intervention changes the conditions under which execution occurs.

Both are necessary.

They solve different problems.

The intervention point may exist in:

Outcome

The system is optimizing the wrong result.

Workflow

The sequence of work is poorly designed.

Allocation

Work has been assigned to the wrong human or machine actor.

Authority

Decision rights are too broad, too narrow or positioned incorrectly.

Management

Humans are coordinating work that the execution system should manage, or machines are operating without adequate supervisory mechanisms.

Organization

Functional boundaries prevent end-to-end execution.

Execution Architecture

The technical system cannot reliably support the operating design.

Agent Architecture

Too many agents, inappropriate specialization or weak coordination creates unnecessary complexity.

Context

The machine actor does not possess the information required to act correctly.

Control

Governance mechanisms create too much friction or insufficient constraint.

Infrastructure

Capacity, latency or availability prevents reliable execution.

Economics

The cost of execution exceeds the value produced.

The operating model therefore becomes diagnosable.

Failure can be traced backwards through its architecture.

Traditional automation often failed locally.

Agentic systems can fail differently.

Machine actors increasingly interpret information, select tools, call other systems, delegate work and create outputs consumed by other machine actors.

An incorrect action can therefore become an input into subsequent execution.

Failure propagates.

By the time a human encounters the consequence, the original failure may be several execution steps removed.

Production research is already exposing the reliability problem.

A 2026 study of 306 practitioners across 26 domains found reliability to be the leading development challenge for production agents. The same research found that production systems tend to remain deliberately bounded. Sixty-eight percent of surveyed implementations executed ten steps or fewer before human intervention. Most also relied heavily on human evaluation rather than fully automated assessment.

The evidence does not suggest that useful agents cannot operate in production.

It suggests that successful production systems currently depend on control, bounded execution and systems-level reliability engineering.

As machine execution becomes more interconnected, the enterprise must therefore observe not only whether individual actors succeed.

It must understand how state and failure move through the execution system.

One of the most important operating failures will not appear in an agent trace.

It appears in the organization.

The enterprise redesigns execution but preserves the management system built for the previous form of work.

Machines begin preparing analysis, routing cases, resolving routine exceptions, producing recommendations and executing bounded decisions.

A second system forms.

The machine execution system performs the work.

The inherited management system continues coordinating the work as though machine execution did not exist.

This is the management double system.

Its cost can be substantial.

The enterprise pays for machine execution while retaining the human coordination architecture that automation was capable of reducing.

The result may appear organizationally stable because familiar management routines remain intact.

Economically, however, part of the operating leverage has disappeared.

The appropriate intervention is not necessarily another automation.

It may be the removal of obsolete coordination.

AI Native transformation therefore requires enterprises to inspect the management system whenever execution changes materially.

The question is not only:

What new work can machines perform?

It is also:

What human coordination is no longer necessary because machines now perform it?

AI systems are often deployed through local organizational ownership.

But enterprises are systems.

A local improvement can move cost, latency or work elsewhere.

The local metric improves.

The enterprise outcome deteriorates.

This is not an argument against local experimentation.

It is an argument for measuring the complete workflow.

Chapter 11 established that economic value must ultimately be measured at the level where enterprise economics change.

Chapter 12 adds the corresponding intervention principle:

When local performance improves but system performance deteriorates, the operating model must be examined across the workflow boundary.

Optimization must follow the outcome.

Not the organizational chart.

A system can also appear well governed while remaining poorly understood.

The enterprise develops confidence because the agent has passed its tests.

Production reality can still diverge.

Research published in 2026 illustrates the gap. In a survey of 157 enterprise respondents, half reported that an agent or large language model feature had passed internal evaluation and subsequently produced a customer-facing failure. Only 5 percent said they fully trusted automated evaluation. The sample was self-selected and should therefore be treated as directional rather than representative of all enterprises.

The operating implication is still important.

An evaluation is evidence about system behavior under evaluated conditions.

It is not proof that the operating model will work under every production condition.

Evaluation theatre begins when passing tests becomes a substitute for observing outcomes.

The enterprise therefore needs several layers of evidence.

These layers answer different questions.

Evaluation must therefore extend beyond machine correctness.

It must test operating performance.

Even a well-designed AI Native workflow does not remain economically correct forever.

The economics of the operating model drift.

Datadog's 2026 analysis of production large language model traces provides one example of how technical conditions can become economic and operational conditions. Its analysis found millions of rate-limit errors across observed workloads and identified capacity constraints as a material source of failed model calls.

The important point is broader than rate limits.

Machine capacity is now a productive resource.

Its consumption must be managed.

An AI Native enterprise therefore cannot calculate the economics of a workflow once and treat the result as permanent.

It needs continuous visibility into:

• cost per outcome
• cost per workflow execution
• cost by machine actor
• cost by model
• cost of retries
• cost of exceptions
• human intervention cost
• infrastructure consumption
• latency
• failure cost
• and the economic value produced.

A workflow whose economics drift beyond its value threshold requires intervention.

The response may be architectural.

Economic discipline is part of operating-model discipline.

The enterprise now needs a capability that connects all of these failure classes.

An operating capability.

Its purpose is to continuously determine whether the operating model is producing the intended outcome and to change the system when it is not.

This is the missing loop.

Without it, AI Native transformation remains a sequence of deployments.

With it, the operating model begins to become adaptive.

A diagnostic capability without the authority to change the system becomes another reporting function.

This creates an important organizational question.

Who can intervene?

The answer will vary by enterprise and by risk.

Some interventions can be automated.

Other interventions require human judgment.

These decisions alter the operating model itself.

They require accountable human authority.

The intervention system therefore follows the same principle developed throughout this publication.

Machine intelligence can expand observation, diagnosis and execution.

Human judgment remains essential where the enterprise must determine what trade-off it is willing to make.

There is a predictable response when autonomous systems fail.

Sometimes that is correct.

Sometimes it recreates the bottleneck that autonomy was intended to remove.

Research on production agents suggests why the balance matters. Current deployments frequently remain bounded and rely heavily on human evaluation. At the same time, research on human oversight architectures has identified the throughput cost of placing humans inside every consequential execution path.

The appropriate response to failure is therefore not automatically less autonomy.

It is better calibrated autonomy.

Authority may need to narrow.

It may also need to expand.

Intervention must diagnose before it constrains.

Accountable autonomy is not maximum autonomy.

It is the amount of autonomy the operating system can support for a given outcome, under defined conditions, with observable performance and recoverable failure.

This changes the role of failure in the AI Native enterprise.

Failure is not merely something to eliminate.

It is information about the operating design.

The enterprise becomes stronger when it can convert those signals into structural change.

The objective is not a failure-free operating model.

No operating model achieves that.

The objective is an operating model capable of detecting failure early, understanding what the failure means and changing itself before the consequences compound.

The argument of this publication has now moved through the operating system.

A final problem remains.

Most enterprises do not begin here.

The AI Native operating model cannot simply be placed on top of that enterprise.

The enterprise must move from one operating system to another while continuing to operate.

That transition is the subject of Chapter 13.