Troubleshooting Durable Agent Runs: Retries, Checkpoints, and Side Effects

The proposed cut-point test establishes an important behavioral property, but the safe-re-execution claim needs an evidentiary acceptance criterion as well: after suppressing completion, the retry must be able to identify the prior external operation from durable external evidence, rather than infer success from a reused request identifier or a later successful call.

For each test run, retain a stable business-operation identity, the exact attempt timeline, the externally acknowledged operation identifier, and a provider-side lookup result keyed to that identity. The recovery result should classify the state explicitly:

• confirmed committed: an external record identifies the operation and its final effect;
• confirmed absent: the authoritative external lookup establishes that no operation exists, so a new submission is permitted;
• unknown: neither condition is established, so automatic re-execution is not evidence of safety and should not create another effect.

This distinguishes the competing explanations that a retry either recovered an already-committed operation or created a second one after an inconclusive first attempt. Temporal’s guidance supports designing Activities for repeated execution, but it does not itself supply the external record needed to resolve that ambiguity. Temporal Activity Definition

The recovery design should be stated as an adapter contract, not inferred from Temporal’s idempotency guidance. The supplied source establishes only that an Activity may be safely executed more than once and that repeated execution should have the same result; it does not define the external operation identifier, lookup endpoint, result representation, or error taxonomy needed to make that true for a particular provider. Temporal’s Activity Definition documentation

Accordingly, an integration should explicitly declare, per provider version: operation_identity construction and scope; the request fields carrying it; which response proves a prior operation was recovered rather than newly performed; how an indeterminate transport failure is represented; and whether a provider-status lookup is available and authoritative. If the provider cannot return a stable prior result for the same identity, the adapter cannot honestly promise that a retry preserves the business-effect contract; it must expose the ambiguity for a separate reconciliation or compensation decision.

This also keeps checkpoints from becoming an undocumented substitute for provider truth. A local checkpoint can record an attempted request or an observed response, but it should not be treated as proof of an external outcome unless the provider contract makes that observation authoritative. The source supports the idempotency objective; the provider-facing adapter must supply the missing request, response, error, and version semantics.

The diagnostic boundary should be observable rather than inferred from a single “retry” counter. Record, for each logical operation, the Temporal Activity identity and attempt number alongside a Runner invocation identifier, turn count, and terminal reason. Then ask: did one Runner invocation exhaust its turn cap, or did Temporal schedule a new Activity Task Execution after an Activity failure? Those explanations imply different recovery actions.

This distinction is concrete in Temporal’s semantics: an Activity retry places a new Activity Task on the Activity Task Queue, while successful completion resumes Workflow code as though the failure had not occurred. That signal does not establish that a fresh Runner invocation retained prior iteration state or avoided replay of a completed external effect. The Runner documentation establishes its in-process entry points, but the supplied material does not define a mapping from Runner turn termination to Activity failure or retry.

• Detection: alert on an Activity reaching its attempt or timeout boundary, separately from a Runner ending at its configured turn limit.
• Diagnosis: correlate a side-effect operation identifier with Activity attempt and Runner invocation; an unchanged operation identifier across attempts identifies the case that requires idempotent handling or reconciliation.
• Bounded action: do not automatically convert a turn-limit termination into an Activity retry classification until an integration test shows the emitted failure, retry behavior, and resume boundary.

Temporal retry policies documents the Activity retry boundary and defaults; Runner documentation documents the Runner entry points. The missing adapter contract remains the decisive evidence for any claim that the two counters describe one recovery policy.

The word must should be scoped as an integration reliability requirement, not presented as a behavior supplied by either component. Temporal’s retry policy retries a failed Activity Task Execution, but it does not decide whether a model refusal, malformed tool arguments, tool timeout, or an already-applied tool operation whose result was lost is reported as an Activity failure. The supplied Runner documentation establishes its in-process entry points, not a durable error-classification contract.

A useful recovery test is to make a tool complete its external write, then terminate the Activity before its result is reported. On the retried Activity Task, verify all of the following: the integration classifies the missing result consistently; the tool receives a stable operation identity; the external system shows one effect rather than two; and the workflow reaches either a recorded success or an observable terminal/reconciliation state. Without that test, mapping every runner exception to a retryable Activity failure can turn a transport ambiguity into repeated side effects, while mapping every error to non-retryable can strand recoverable work.