A timeout is not permission to repeat a write
Reliable retries require an explicit operation identity, an atomic local boundary and a plan for external commits you cannot observe.
Start reading
Name the ambiguity before adding retries
In a hypothetical refund workflow, a service commits a refund but its response never reaches the caller. The caller sees a timeout. Retrying with a fresh operation identity can create another refund; abandoning the request can leave the user uncertain about an action that already happened. Neither choice follows safely from the timeout itself.
Distinguish the business operation from each transport attempt. The operation is the user's authorized intent. Attempts are deliveries of that intent across an unreliable boundary. A retry policy controls timing and load; an idempotency contract controls whether repeated delivery creates another effect. Backoff and jitter do not provide deduplication.
AWS's discussion of idempotent APIs favors an explicit caller-provided request identifier over inferring intent from identical parameters [1]. Two legitimate operations can have identical bodies. An agent must therefore preserve an operation identifier through retries instead of improvising a new one whenever a tool call fails.
Bind the key to a stable request meaning
Scope an idempotency key to the tenant and operation type, and authenticate every attempt. A key is not a bearer credential. Generate it when the business operation is created and persist it before sending the first attempt. Concurrent workers handling that operation must retrieve the same identity.
Store a cryptographic request hash with the key. Canonicalize the validated semantic request before hashing: normalize field ordering, amount representation, defaults and relevant API version. Include fields that change intent, such as payment, amount and currency. Exclude transport metadata that legitimately changes between attempts. Define this canonicalization as part of the contract; arbitrary JSON serialization is not enough.
The hash detects reuse of one key with different parameters; it does not decide whether two requests represent the same intent. Reject a mismatch explicitly. Stripe documents parameter comparison for reused keys and distinguishes API-version behavior [2]. Treat that as an example of a published contract, not a universal promise about every downstream API.
Make local uniqueness and mutation atomic
A cache lookup followed by a write has a race: two workers can both observe no prior request. Enforce uniqueness in durable storage, scoped by tenant, operation type and key. For effects wholly inside one transactional database, commit the deduplication record, mutation and result together. AWS explicitly describes the need for an atomic operation encompassing the token and associated mutations [1].
Illustrative local-only transaction:
BEGIN
INSERT operation(scope, key, request_hash)
-- durable UNIQUE(scope, key)
validate authorization and business preconditions
apply local mutation
store stable result and resource identifier
COMMIT
On unique conflict, after the winning transaction resolves:
load committed operation
reject a different request_hash
otherwise return its stored result
This outline omits database-specific conflict and rollback handling. Serialize competing changes to the payment through a lock or checked version; uniqueness on request keys alone cannot enforce a refund limit across distinct operations. A worker must not return an uncommitted result. Recheck current access before exposing a stored response, and document which preconditions are evaluated only on first execution. Returning the prior operation's result is different from rerunning today's business logic against changed state.
Do not extend a local transaction across the network
The preceding transaction cannot atomically commit a refund at an independent provider. Holding a database lock while calling that provider does not close the crash window: the provider may commit before the local process records success. A local uniqueness constraint alone therefore cannot guarantee exactly-once external effects.
Persist an intent and an outbox entry together, then let a worker deliver it using a stable downstream idempotency key. The outbox prevents losing the intent after a local commit, but normally permits repeated delivery. Safety still depends on the provider's documented deduplication behavior and retention window.
Represent an ambiguous response as unknown or pending reconciliation. Query status by a stable provider reference where supported; otherwise route to a controlled review process. Do not announce success or automatically mint a new key. If the external API has neither deduplication nor reliable lookup, that is a real limit: restrict automation or accept an explicitly reviewed risk. A compensating action may itself fail and is not an atomic rollback.
Treat expiry and retry budgets as correctness decisions
Keys cannot necessarily be retained forever, but deleting them changes semantics. A delayed retry arriving after expiry may be treated as a new operation. Stripe's API v1 documentation, for example, permits pruning keys after at least 24 hours and describes a new request when a pruned key is reused [2]. That window is provider-specific, not a recommended universal setting.
Choose retention using the maximum supported retry and reconciliation horizons, queue delays and business lifetime. Consider durable business uniqueness constraints or longer-lived tombstones for effects that must never repeat. Retaining less response data may reduce privacy exposure while preserving an operation identity, but resource lookup must still be possible.
Bound retries independently with deadlines and backoff. Classify validation failures, throttling, transport failures and ambiguous server errors according to the actual endpoint contract. A repeated error may be a stored result rather than a fresh attempt. Exhausting the retry budget should produce a visible unresolved operation, not silently declare the effect absent.
Test the contract at its crash windows
Use a fault harness to interrupt the workflow at causal boundaries, not only to return random errors. The following matrix is illustrative guidance, not a report of measured product behavior.
| Scenario | Required observation |
|---|---|
| Concurrent identical keys | One local mutation; consistent operation result |
| Same key, changed amount | Explicit mismatch; no second mutation |
| Crash after external commit | Reconciliation; no unguarded new operation |
| Retry after key expiry | Documented rejection or reviewed duplicate risk |
Also test two distinct keys with identical payloads: whether both should execute depends on the business contract. Inspect durable state and provider references, not just response counts. Idempotency trades storage, coordination and reconciliation complexity for safer repeated delivery. It becomes useful when those limits are explicit enough that both the agent and its operators know when to retry, when to inspect and when to stop.
References
Primary sources and further technical context. Guidance should be checked against your own system contracts.
By Nuhaverse editorial. Engineering guidance for system design and review. Deployment requirements and capabilities are agreed for each engagement; test results are not safety guarantees.
Back to the journal
