AI Agent Failure Recovery Playbook

An AI agent failure recovery plan starts after a live run stops matching its intended state. The operator’s first job is not to restart the model. It is to determine what actually happened: which decision completed, which tool call reached an external system, which side effect committed, and which state is still unknown.

That distinction matters because an agent can fail after a successful mutation but before receiving the receipt. Retrying may create a duplicate order, message, refund, account, or deployment. A fluent final answer is not proof that the underlying operation committed, and a timeout is not proof that it did not.

This playbook owns live incident recovery: failure classification, stop-or-continue decisions, retry budgets, checkpoints, compensation, escalation, rollback versus roll-forward, evidence preservation, and postmortems. Use the AI Agent Reliability Checklist for broader preventive controls, the AI agent testing framework for pre-production evaluation, and the AI agent observability tools comparison when selecting telemetry products.

The 60-second response

When an agent run fails in production:

1. Freeze new side effects. Pause the run and downstream consumers. Do not erase logs, re-run the whole workflow, or let a scheduler create a second incident.
2. Assign an incident key and owner. Record the workflow, tenant, run ID, business operation key, current checkpoint, and human incident commander.
3. Classify state, not symptoms. Separate a rejected operation from an unknown commit, a corrupted decision, a dependency outage, and an authorization or policy stop.
4. Reconcile external reality. Query the destination by a stable business key. Treat “request timed out” as unknown until verified.
5. Apply the decision gate. Continue only if the next action is permitted, bounded, idempotent or safely compensated, and based on a trustworthy checkpoint.
6. Choose one recovery path. Retry the failed step, resume from checkpoint, compensate, roll back, roll forward, or hand off. Never mix paths without recording the transition.
7. Close with evidence. Save the state before and after recovery, receipts, actor and approver identity, commands, timestamps, and follow-up actions.

Failure taxonomy for live agent runs

Classification determines which actions are safe. Use one primary class and add contributing factors rather than calling every event a “model error.”

Class	Typical signal	Default recovery stance
Decision failure	Unsupported conclusion, violated constraint, malformed plan	Stop before further tools; human review or regenerate from last trusted input
Tool contract failure	Schema validation, unsupported parameter, explicit 4xx rejection	Correct the request; do not spend blind retries
Transient dependency failure	429, 503, connection reset, bounded timeout with no evidence of commit	Reconcile, then retry under budget with backoff and jitter
Unknown commit	Timeout or worker crash after dispatch but before receipt	Stop; query by operation key before any mutation retry
Partial workflow commit	Some external effects succeeded and later steps failed	Resume from a checkpoint or execute ordered compensation
Stale or conflicting state	Version mismatch, duplicate work, changed record, concurrent approval	Re-read; use preconditions; replan or escalate
Authentication/authorization failure	Expired credential, revoked scope, MFA, policy denial	Stop; restore approved identity path—never bypass the control
Safety or governance stop	Required approval absent, sensitive-data boundary crossed, policy classifier blocks	Stop and escalate; retries cannot turn a forbidden action into an allowed one
Resource exhaustion	Context limit, token budget, disk, queue capacity, deadline exceeded	Shrink/split work or shed load; continue only from an explicit checkpoint
Orchestrator/worker loss	Process crash, lease expiry, duplicate delivery	Recover durable state; fence the old worker before resuming

A symptom can span classes. A 500 response may be transient, while a lost connection following a payment request is an unknown commit. Classify from evidence, not HTTP status alone.

Stop or continue: the operator gate

Continue only when every answer below is yes:

• Authority: Is this action still allowed for this identity, tenant, data class, and incident state?
• Known state: Is the last checkpoint trustworthy, and have external effects since it been reconciled?
• Bounded blast radius: Are affected records, destinations, spend, and time capped?
• Duplicate safety: Does the action have a stable operation key, conditional precondition, or proven compensation?
• Useful attempt: Is there evidence that conditions changed or another attempt can plausibly succeed?
• Remaining budget: Are attempt, elapsed-time, token/cost, and side-effect budgets all available?
• Verification: Can the operator prove the result through an independent read or receipt?

Stop when any answer is no. Also stop immediately for unknown commits, missing approvals, policy denials, inconsistent checkpoints, suspect credentials, irreversible high-impact actions, repeated non-retryable failures, or evidence that recovery is increasing load.

This gate is stricter than “the exception is retryable.” It decides whether the business action is safe to continue.

Build a retry budget, not a retry loop

The AWS Builders’ Library warns that retries are selfish and layered retries can multiply load. A five-deep stack making three attempts at every layer can create 243 database calls. Pick one layer to own retries and make all other layers report failure upward.

A production retry budget should cap four dimensions:

retry_policy:
  owner: workflow_orchestrator
  max_attempts: 3
  max_elapsed: 90s
  max_incremental_cost_usd: 0.25
  max_side_effect_attempts: 1
  initial_backoff: 2s
  backoff_multiplier: 2
  jitter: full
  retryable: [rate_limited, dependency_unavailable, connection_reset]
  non_retryable: [invalid_input, permission_denied, policy_stop, unknown_commit]

The numbers are examples, not universal defaults. Set them from the operation’s latency objective, dependency recovery time, cost, concurrency, and consequence. Reads may tolerate more attempts than mutations. A model call may be cheap in side effects but expensive in tokens. A single external mutation may exhaust the side-effect budget even when two transport attempts remain.

Before each retry, log: attempt number, classification, previous result, expected changed condition, delay, budgets remaining, and the idempotency or precondition mechanism. If you cannot explain why the next attempt is safer or more likely to work, stop.

Idempotency and the unknown-commit rule

Idempotency means repeating the same intended operation produces no additional business effect. It does not mean “send the request again and hope.”

Give every mutation a durable key derived from the business intent, such as tenant:invoice:2026-08:send-v1. Persist it before dispatch. Send it through the destination’s supported idempotency field when available, and store the request digest, destination, response, and external resource ID. Amazon EC2 documents this pattern through client tokens for supported requests; a repeated token with different parameters is rejected.

When no native key exists:

1. perform a read using a unique business key;
2. create with a conditional write or uniqueness constraint where possible;
3. record the resulting external identifier atomically;
4. verify through an independent read;
5. route ambiguous outcomes to reconciliation, not blind retry.

RFC 9110 defines idempotent HTTP methods, but application semantics still matter. A POST can be safely replayed with a service idempotency key; a nominally idempotent request can still trigger poorly designed side effects. Evaluate the destination contract, not the verb alone.

Unknown-commit rule: after a timeout, crash, or connection loss following dispatch, assume the effect may have committed. Query by operation key or destination receipt. Retry only after proving absence or using a contract that returns the original outcome for the same key.

Checkpoints that are safe to resume

A checkpoint is a durable statement of business state, not a memory dump or the last line printed. Save it at side-effect boundaries and human handoffs.

A useful checkpoint contains:

{
  "workflow_version": "claims-4.2",
  "run_id": "run_01K...",
  "tenant_id": "acme",
  "step": "issue_refund",
  "status": "dispatched_unknown",
  "input_digest": "sha256:...",
  "operation_key": "acme:claim-884:refund-v1",
  "external_receipt": null,
  "completed_effects": ["claim_locked", "approval_1729"],
  "pending_effects": ["refund", "customer_notice"],
  "policy_version": "refund-2026-08",
  "actor": "agent-service-prod",
  "trace_id": "00-...",
  "recorded_at": "2026-08-21T17:00:00Z"
}

Resume only with the same compatible workflow and policy version, or run an explicit state migration. Fence the prior worker with a lease or generation number so two executors cannot continue. Revalidate credentials, approvals, deadlines, and external state because they may have changed while the run was paused.

Temporal documents durable workflow state reconstructed from event history and requires deterministic workflow code. That is a product capability, not permission to replay arbitrary side effects; activities still need timeouts, retry policy, and idempotent boundaries.

Compensation: undo the business effect safely

A compensating action is a new, explicit operation that semantically neutralizes a committed effect. It is not database time travel. Microsoft’s Saga pattern notes that compensation may not run in exact reverse order and can fail itself.

For every compensable step, define:

• forward operation and commit evidence;
• compensation operation and eligibility window;
• ordering dependencies;
• idempotency key for the compensation;
• expected residual state;
• owner and approval threshold;
• verification query;
• what to do if compensation fails.

Example: if an agent created a support entitlement and then failed to issue a refund, the safe response may be to complete the refund (roll forward), not revoke the entitlement (roll back). If it sent an incorrect customer message, deletion may be impossible; compensation is a correction with preserved audit history.

Make compensation resumable. Record not_started, dispatched_unknown, committed, verified, or failed for each item. Use a separate operation key such as original-key:compensate-v1 and never hide the original effect.

Roll back or roll forward?

Roll back when the previous state remains valid, the reversal is supported and complete, downstream consumers have not made the change irreversible, and recovery time is shorter than repairing forward. Configuration and versioned deployments often fit this pattern; Kubernetes Deployments document revision history and rollback behavior.

Roll forward when data has migrated, external parties have observed the effect, compensation would cause more harm, the previous version cannot read current state, or a small patch can restore the intended invariant faster.

Use this decision record:

Question	Rollback signal	Roll-forward signal
Can every effect be reversed?	Yes, verified compensations	No or uncertain
Is old code compatible with current data?	Yes	No
Have customers/partners acted on it?	No	Yes
Is the root cause understood?	Reversion removes it	Patch is known and bounded
Which path restores invariants fastest?	Previous state	Corrected current state
Can the result be independently verified?	Full reconciliation	Full reconciliation

Do not treat rollback as inherently safer. The correct target is a verified business invariant, not an earlier timestamp.

Escalation packet and incident roles

Escalate before budgets expire when the blast radius grows, two recovery attempts fail for the same cause, a sensitive or irreversible effect is possible, authorization is unclear, multiple tenants are affected, or reconciliation cannot establish state.

Google’s SRE incident-response guidance separates command, operations, communications, and planning. For agent incidents, the handoff packet should include:

• incident ID, severity, commander, operators, and decision authority;
• start time, detection source, affected workflow/version/tenants;
• current taxonomy and confidence;
• last trusted checkpoint and all known external effects;
• operation keys, trace IDs, receipts, screenshots or response bodies;
• budgets spent and remaining;
• actions attempted, exact results, and commands;
• rollback and roll-forward options with risks;
• next decision deadline and stakeholder communication state.

The human should not have to reconstruct this from chat. A handoff is successful when the recipient can state the current truth, the next permitted action, and how success will be verified.

Worked scenario: duplicate-risk refund

A claims agent receives approval, calls a payment provider to issue a $240 refund, and loses the connection before the response. The workflow marks the step failed and prepares to retry.

1. Freeze: pause this claim and any automatic customer notification.
2. Classify: unknown commit—not a confirmed provider failure.
3. Reconcile: search the provider using acme:claim-884:refund-v1 and the charge ID.
4. Branch:
   • If a refund exists, persist its provider ID, verify amount/status, mark the effect committed, and resume at notification.
   • If no refund exists and the provider guarantees idempotency for that key, retry once under budget.
   • If the query is unavailable or ambiguous, escalate; do not send another refund.
5. Verify: independently retrieve refund and ledger states; confirm the claim total invariant.
6. Evidence: save approval, request digest, operation key, provider lookup, final receipt, actor, and timestamps.
7. Close: communicate the outcome and file a postmortem if the unknown window exceeded the response objective or exposed missing reconciliation.

The crucial recovery action is reconciliation. More model reasoning does not resolve an external commit ambiguity.

Evidence capture and postmortem

Preserve evidence before changing state. At minimum capture the original input (with sensitive data handled appropriately), model and workflow versions, prompts or policy references, tool request/response digests, operation keys, checkpoints, trace identity, approvals, external receipts, retry decisions, compensation records, and the final independent verification.

W3C Trace Context standardizes distributed trace identity through traceparent and tracestate. Propagation helps correlate boundaries; it does not replace business operation keys or prove a mutation’s outcome.

A useful postmortem records:

1. impact and duration;
2. timeline in UTC;
3. intended and actual state transitions;
4. trigger, contributing conditions, and why controls did not contain it;
5. classification and evidence quality;
6. recovery decisions, budgets, and verification;
7. what went well, what slowed response, and where state was ambiguous;
8. corrective actions with owners and deadlines.

Google SRE’s postmortem guidance emphasizes learning without blame. Avoid “the agent hallucinated” as a root cause. Ask why that decision could reach a tool, why a duplicate was possible, why the checkpoint was insufficient, and why escalation or reconciliation did not happen sooner.

Copyable incident runbook

INCIDENT: [id / severity / commander / UTC start]
SCOPE: [workflow version / tenants / records / destinations]
FREEZE: [scheduler, queue, worker lease, downstream effects]
LAST TRUSTED CHECKPOINT: [id / state / digest / timestamp]
FAILURE CLASS: [primary / contributing / confidence]
EXTERNAL REALITY: [committed / absent / unknown + evidence]
AUTHORITY: [policy, identity, approvals still valid?]
BUDGETS: [attempt / elapsed / cost / side effects remaining]
PATH: [retry | resume | compensate | rollback | roll forward | escalate]
NEXT ACTION: [owner / command or operation / deadline]
SUCCESS INVARIANT: [what must be true]
VERIFICATION: [independent query / receipt / reconciled totals]
EVIDENCE: [trace, operation keys, requests, responses, screenshots]
COMMUNICATIONS: [audience / last update / next update]
POSTMORTEM: [required? owner / due date]

Recovery readiness questions

Before an agent is allowed to mutate production, operators should be able to answer:

• What durable key identifies each business operation?
• Which errors are non-retryable, and which layer owns attempts?
• What are the time, cost, attempt, and side-effect budgets?
• Where are checkpoints written relative to external commits?
• How is the old worker fenced during resume?
• How do we query whether an ambiguous operation committed?
• Which effects can be compensated, in what order, and with whose approval?
• What invariants decide rollback versus roll-forward success?
• What evidence survives a worker crash?
• Who can stop the system, command the incident, and approve recovery?

If those answers are missing, the immediate mitigation is a human-controlled stop—not an unlimited fallback model.

Build the recovery path before the incident

A strong recovery system makes uncertainty explicit. It does not promise that agents never fail. It ensures that operators can freeze side effects, identify the last trusted state, reconcile external reality, and take one bounded, authorized path to a verified invariant.

If you are designing high-stakes agent operations and need recovery controls mapped to your systems, talk to Midpoint about enterprise AI workflow orchestration.