When a connected system times out: deciding whether to retry

Begin with the business action

In the example, an approved CRM request should create one installation job in a separate job board. The operator needs the job identifier to continue scheduling. A repeat can cause two teams to work from different records, so the workflow must preserve one logical request even when communication is unreliable. Before discussing retry counts, write down the intended action, the system responsible for recording it and the evidence that would show it happened. Separate a read of job status from a request to create a job. They have different consequences. An automated assistant should use the same defined recovery path as other callers; it should not invent a new business action because the first response was slow.

Classify the observation before choosing an action

Use three questions in order. First, did the service return a documented final response? Second, does that response establish whether the requested change occurred? Third, if the outcome is unknown, does the API offer a supported way to reconcile or safely repeat the same request? Keep the actual response code, request reference and time beside the classification. A generic error message from an intermediate proxy may not answer the second question. The branches below are examples to adapt to the specific API contract. Do not treat every timeout, server error or rate-limit response as interchangeable.

Case one: a read can be retried within limits

Suppose the operator is checking the status of an existing job and the documented status endpoint only reads data. A transient connection failure may justify a limited retry under that endpoint's rules. Microsoft's Retry pattern describes delaying further attempts and choosing a bounded policy suited to the operation. Account for provider guidance, an overall deadline and retry behaviour already present in the client library. In the exercise, the business owner could choose to stop waiting after a short operational deadline and show that status is temporarily unavailable. That is an agreed design choice, not a universal number or a reason to change the job itself.

Case two: a clear validation rejection needs correction

Imagine the job-board API returns a documented validation rejection explaining that a required location reference is missing and no job was created. Sending the same invalid payload repeatedly does not fix it. Return the request to the appropriate operator, show the field that needs attention and keep the rejection evidence. The statement that nothing was created must come from the service's documented response semantics, not from the presence of an error word. If the location or other business details change, follow the provider's rules for a new logical request and obtain any approval the changed action requires.

Case three: a create request times out

Now imagine the request left the CRM, but the connection ended before the response arrived. The job board might have rejected the request, still be processing it or already have created the job. Record an outcome-unknown state and preserve the original request reference. Look for a supported status lookup or the provider's documented idempotent retry mechanism. AWS's discussion of idempotent APIs explains the value of a caller-provided request identifier for recognising repeated intent. Such a mechanism only protects this workflow if the particular service implements the required contract; adding an arbitrary header on the client is not enough.

Keep one logical request intact

For the example, give the approved job request a durable local identifier before sending it. If the provider supports an idempotency key, store the key and the exact intended payload together, scoped to the correct account. A permitted retry of that same action should follow the documented key, payload and retention rules. Do not generate a fresh key merely because the first attempt timed out. Conversely, do not reuse an old key for a different job. Check how long the provider remembers keys and what it does when the same key arrives with different parameters. These are integration requirements to verify, not capabilities to assume from the word idempotent.

When there is no reliable lookup, keep the hold visible

Suppose the provider offers neither a dependable request-status query nor a documented safe retry. An empty search result may reflect delayed visibility, a wrong filter or the wrong account. It is not enough evidence to promise that the first request failed. Leave the action unresolved, assign an operator and describe the next reconciliation step. The operator may need the provider's support process or an authorised check in the destination system. Keep other independent work moving, but prevent another path from quietly submitting the same job. The workflow should make uncertainty visible to the person responsible for resolving it.

Test the branches with controlled examples

Prepare a small set of cases in an approved test environment: a normal response, a documented validation rejection, a temporary failure before a read completes, and a created job whose response is deliberately lost in the test harness. For the last case, the expected business result is one job and a reconciled local record, not merely a successful second HTTP response. Also check what happens when the overall deadline expires or an operator restarts the worker. Retain the original logical request across restarts. Use fictional data and provider test facilities; do not create real customer jobs to discover how retries behave.

Make the recovery record useful to the next operator

Record the request identifier, intended action, destination account, attempt times, available provider references, observed outcome and next owner. Keep secrets and unnecessary personal data out of routine logs. Distinguish a request accepted for processing from a completed job, and distinguish a completed job from a verified local reconciliation. A concise handover could read: 'Creation outcome remains unknown; original request retained; assigned operator will reconcile through the provider's supported lookup.' That is illustrative wording. Use a completed status only when the evidence supports it, and record how the conclusion was reached.

Agree the recovery contract before launch

For a bounded integration, ask the implementation team to demonstrate these branches and explain the provider-specific limits. Decide who can retry, who can correct the payload and who resolves an unknown outcome. Include the recovery work in the project scope rather than judging the integration only by its successful demonstration. Agenticures' CRM integration work can help define that contract for a particular workflow. The final behaviour still depends on the actual systems, access, testing and approved operating rules.