A Timeout Is Not a Retry Signal

A Timeout Is Not a Retry Signal An agent asks a deployment provider to start a release. The provider accepts the request. The network connection dies before the response reaches the agent. What happened? The release may be running. It may have failed before it started. The provider may have accepted the request but delayed execution. From the caller's point of view, all three outcomes look the same: no trustworthy result arrived. This is where a surprising number of "reliable" automation flows become duplicate-action machines. They translate missing evidence into failure, then retry the write. The missing response is not a failure verdict. It is an uncertain external effect. Idempotency is necessary, but it does not answer the question An idempotency key prevents the same logical request from creating two effects, provided the whole path respects that key. It should be minted before dispatch and bound to the exact actor, target and payload. That handles one problem: If I submit this logical operation again, will the system create a second release? It does not automatically answer another: Did the first operation commit, and what state did it leave behind? Those questions are often collapsed into one boolean. They should not be. A local service can remember that it dispatched key deploy-42. The provider may still expose no lookup by that key. Or the provider may return an operation ID, but the caller may time out before persisting it. Or a process may restart while the local record still says deploying. In all three cases, replaying the local operation record is not reconciliation. It only proves what the caller remembers. Model the attempt and the outcome separately I find it useful to keep two related records. The attempt record describes the requested external action: stable operation ID; idempotency key; actor and authorization reference; target account, project or resource; normalized payload hash; provider route; dispatch timestamp; provider request ID, if one was received. The outcome record describes what can actually be proved: provider acknowledgement; provider object ID; current provider state; independently observed result; verification timestamp and source; final classification; recovery owner and deadline when unresolved. This split stops an accepted HTTP request, a started command or an in-memory success flag from masquerading as the business outcome. For a deployment, "the command started" is not the same as "the requested revision is serving healthy traffic." For a publication, "the submit button was clicked" is not the same as "the canonical post exists at this public URL." For a payment, "the client timed out" says nothing reliable about whether money moved. Give uncertainty a durable state The state machine needs a place where automation stops making new writes. One practical sequence is: prepared -> authorized -> dispatching -> acknowledged -> verifying -> succeeded | failed dispatching | acknowledged | verifying -> unknown -> reconciling -> succeeded | failed | confirmed_absent | needs_human unknown should be persisted, not thrown as a transient exception. It should carry enough context for another process to continue the investigation without reconstructing intent from logs. At minimum, store: the exact request fingerprint; the last confirmed transition; which evidence is missing; whether another write is forbidden; who owns reconciliation; when the next read should happen; what would count as proof of commit or proof of absence. This turns "something went wrong" into a bounded operational task. Reconcile with reads before considering another write A safe recovery path is deliberately boring. Freeze redispatch. Keep the original idempotency key and mark the operation as unresolved. Query the provider. Use a provider request ID, client reference, deterministic name or bounded time window. Match the intended effect. Compare target, payload hash, revision, account and timestamps. A vaguely similar object is not enough. Verify the customer-visible result. Check the declared postcondition through an independently meaningful read. Provider acknowledgement alone may be insufficient. Record the conclusion. Persist the provider object ID, evidence source and classification. Permit a new write only after confirmed absence. Use a new attempt identity while retaining the link to the unresolved attempt. The difficult branch is confirmed_absent. Many provider APIs cannot prove a global negative. Pagination may be incomplete, draft inventory may be hidden, or the caller may lack permission to inspect all relevant objects. When coverage is incomplete, the honest result is still unknown. Manual review is safer than manufacturing certainty. A bug we found in our own deployment flow In a bounded deployment dogfood project, we wrote two rules early: repeating a deploy request with the same idempotency key must return the original operation rather than dispatch a second deployment; missing or uncertain evidence must never become running or ready. Those rules were useful, but a later assurance review exposed a gap. A crash after dispatch could strand an operation in deploying. Replaying the request returned the stale local record. The system still did not prove whether the external effect occurred. That finding changed the question we ask during review. We no longer stop at "is the write idempotent?" We also ask "what authoritative read can settle an interrupted write, and who owns that reconciliation?" The review did not prove that every provider needs the same recovery mechanism. It did show that idempotency without an evidence path can freeze ambiguity rather than resolve it. Tests that reveal the real contract Happy-path tests rarely expose this class of failure. Inject uncertainty at the boundaries: The provider commits the effect, then the client connection times out. The provider returns an operation ID, then the local process crashes before persisting it. The local record is persisted, but the provider rejects the request before commit. The provider reports success, but the customer-visible postcondition fails. Reconciliation finds two similar objects and cannot identify a unique match. Provider inventory is paginated or incomplete, so absence cannot be proved. A worker tries to redispatch while the original attempt is still unknown. For each case, assert more than "no exception": at most one external effect exists; no unresolved operation triggers a blind retry; the evidence record names what is known and what is missing; success requires the declared postcondition; manual escalation preserves the original operation identity. The small design change with the largest effect If an agent can mutate an external system, add unknown to its domain model before adding another retry policy. Then define one authoritative read path for that state. It can be a provider operation lookup, a public-object search, a database reconciliation query or a manual inspection queue. The exact mechanism depends on the provider. The invariant stays the same: uncertainty blocks another write until evidence resolves it. That makes the system less eager. It also makes it much easier to trust. Practical exercise: pick one external write in your system and answer four questions: What is its stable operation identity? What proves the effect happened? What proves it did not happen? Who owns the operation when neither answer is available?