SECURITYATLAS✳

✳ The reference desk / In practice

Retry budget

A limit on how often automation may repeat an action, especially when the previous outcome is uncertain.

What it means

A retry budget limits how much additional work a system may perform after an unsuccessful or uncertain attempt. Limits may apply per operation, across a time window, or to the fraction of total requests that are retries. The purpose is to avoid endless loops, duplicated side effects, and extra load on an already struggling service. A timeout is especially important: the server may have completed the action even though the caller never received its response. Before repeating a consequential action, automation needs a way to determine whether the intended effect already occurred.

See the idea / Automation

A lost response gets bounded reconciliation.

Use counted readbacks to resolve one uncertain write. Stop when the configured check budget ends.

Fictional Q92 targets U92 only. The service applies one authorized set-disabled write, but its response is lost. The operation allows one write total and two readbacks total, including the first read, before t=2.0 seconds. Compare a delivered first read with two timed-out reads. The scene’s playback pace is not this exercise clock.

Retry budget: Use counted readbacks to resolve one uncertain write. Stop when the configured check budget ends. The chapter text below describes every state.AccountDesk · operation Q92 · U92One caller · deadline t < 2.0 secondsTotal operation budgetWrites 0/1 · reads 0/2Read cap includes the first readService state · teaching viewEnabled · unchangedOriginal enabled snapshot retainedCaller’s observationNo observation yetTimeout does not undo a writeNext step for Q92Not started · deadline t<2sNo other account or session changesBUDGET SET BEFORE REQUESTS

01 / 04

Set limits

Q92 has one write and at most two readbacks. U92 is enabled; no request has been attempted.

More about this step

The read limit counts total attempts, so two reads permit only one repeated read. A deadline check must also pass before sending. The writer will not be repeated in this exercise. Separate per-client load limits, multiple layers and jitter are omitted rather than silently assumed.

Keep in mindThe limits and timings are synthetic policy choices, not universal retry advice. One caller has no hidden library or worker retries. No existing-session closure, analyst completion or full containment is established. A timeout leaves the caller uncertain even when the teaching view shows a completed write.

Read the visual, step by step

First read delivered

  1. Set limits. Q92 has one write and at most two readbacks. U92 is enabled; no request has been attempted. The read limit counts total attempts, so two reads permit only one repeated read. A deadline check must also pass before sending. The writer will not be repeated in this exercise. Separate per-client load limits, multiple layers and jitter are omitted rather than silently assumed.
  2. Lose response. The write reaches the service and disables U92, but Q92 loses the response. The caller’s result becomes unknown. The modeled write is applied at t=0.08s; no conclusive response arrives by t=0.10s. A completed write is not undone by that loss. Q92 counts one write attempt and schedules a read of the same account. No second write or related-account action is authorized by the timeout.
  3. Read once. The first read returns enabled=false. Q92 knows the scoped state and needs no repeated read. Q92 sends GET /accounts/U92/enabled at t=0.50s. The delivered path receives HTTP 200 {"enabled":false} at t=0.70s. The timeout path receives no usable response by t=0.80s. Either way, this consumed one read attempt and never changes U92. These are fixed synthetic events.
  4. Stop early. Q92 stops after the first delivered read. The unused read allowance does not require another request. The record contains one write attempt and one read attempt. No further operation runs merely to spend the allowance. The observed disabled flag closes this scoped check only; application sessions and other containment work have not been tested.

Both reads time out

  1. Set limits. Q92 has one write and at most two readbacks. U92 is enabled; no request has been attempted. The read limit counts total attempts, so two reads permit only one repeated read. A deadline check must also pass before sending. The writer will not be repeated in this exercise. Separate per-client load limits, multiple layers and jitter are omitted rather than silently assumed.
  2. Lose response. The write reaches the service and disables U92, but Q92 loses the response. The caller’s result becomes unknown. The modeled write is applied at t=0.08s; no conclusive response arrives by t=0.10s. A completed write is not undone by that loss. Q92 counts one write attempt and schedules a read of the same account. No second write or related-account action is authorized by the timeout.
  3. Read once. The first read times out; Q92 still has no delivered state result. Q92 sends GET /accounts/U92/enabled at t=0.50s. The delivered path receives HTTP 200 {"enabled":false} at t=0.70s. The timeout path receives no usable response by t=0.80s. Either way, this consumed one read attempt and never changes U92. These are fixed synthetic events.
  4. Stop at cap. The repeated read also times out. Q92 has used both reads, stops automatic work, and leaves a case for Nora. The second GET starts at t=1.40s and times out at t=1.70s, inside the 2.0s deadline. The two-read count is exhausted, so no third read or second write occurs even before that deadline. Nora receives Q92’s attempt record for reconciliation; her investigation remains unfinished.

The limits and timings are synthetic policy choices, not universal retry advice. One caller has no hidden library or worker retries. No existing-session closure, analyst completion or full containment is established. A timeout leaves the caller uncertain even when the teaching view shows a completed write.

Flow references: Google SRE: Handling Overload · AWS Builders Library: Timeouts, retries, and backoff with jitter · IETF RFC 9110: Idempotent Methods

AN ILLUSTRATIVE SCENARIO

One uncertain write and counted readbacks

Fictional Q92 applies one U92 set-disabled write but loses the response. Its policy permits one write total, two readbacks total and a deadline before t=2.0 seconds. A delivered first read ends the scoped check. Two timed-out reads exhaust the check count and leave Nora’s reconciliation case open, even though the teaching view shows the completed write. No second write or related-account action occurs.

Put it to work

  1. Classify operations by side effect and retry safety. Identify which failures are transient and whether the service supports an idempotency key, a status lookup, or another reliable reconciliation mechanism.
  2. Set a maximum elapsed time and attempt count appropriate to the action. Use increasing delays with random variation where suitable, and respect service limits so many clients do not retry simultaneously.
  3. Record attempts under one stable operation identifier and verify the resulting state. Stop on exhausted budget, a non-retryable error, or unresolved ambiguity, and route the case to a responsible person.

How to check your work

Simulate a response lost after the server applies a test action. Confirm automation recognizes the existing result, avoids duplicate side effects, respects its total budget, and reports an unresolved state clearly when independent verification is unavailable.

Connect the ideas

  • Idempotency

    The property that repeating the same operation has the same intended effect on state as performing it once; separate audit records may still be created.

  • Postcondition

    The independently checked state of the world that should be true after a response action.

  • Uncertainty

    What you still do not know, written explicitly so it is not filled with false precision.

Explore a field lesson

Find your next idea.

Tip: press / to open search. Escape closes this window.