API integration done right: idempotency, retries and webhooks

An API call can complete on the provider while the response is lost on the network. A webhook can arrive twice, late or out of order. A timeout can mean the remote system did nothing, or that it finished successfully and the caller stopped waiting. Reliable integration design starts by treating these events as normal operating conditions.

The local application and an external provider do not share one database transaction. That means there is always a period when their views of the world may differ. The goal is not to pretend this gap disappears. The goal is to record enough state to retry safely, reconcile later and explain what happened.

Give each business operation a stable local identity. Store the provider, operation type, current state, attempt history and remote identifier when one exists. This record becomes the place operators inspect and the basis for deciding whether work should run again.

An idempotent operation produces the same business result when the same intent is submitted more than once. Payment creation is the familiar example: retrying after a timeout must not charge the customer twice. The same concern applies to reservations, shipments, account creation and message delivery.

Create an idempotency key from the local operation, not from a random value generated for every HTTP attempt. Send that key when the provider supports it and persist it before the first request. If the process crashes, the next worker can reuse the same key and ask for the same outcome.

Your own inbound endpoints may also need idempotency. Store a caller's key with a fingerprint of the request and the completed response. A repeat with the same payload can receive the stored result. Reusing the key with different input should be rejected because it represents ambiguous intent.

Database constraints provide a final line of defence. A unique provider event identifier prevents two workers from applying the same webhook. A unique local-operation key prevents competing requests from creating duplicate work. Application checks alone can race under concurrency.

Retries help with temporary failures: connection resets, timeouts, rate limits and many server errors. They do not fix invalid credentials, malformed requests or a resource that the provider says does not exist. Classify errors so permanent failures stop quickly and surface a useful reason.

Use exponential backoff with random jitter. Increasing delays reduce pressure on a struggling provider, while jitter keeps many workers from retrying at the same instant. Respect a provider's retry guidance when it supplies one. Put a ceiling on both delay and attempts, then move exhausted work into a visible failed state.

The retry policy belongs to the operation, not only the HTTP client. A low-level library cannot always know whether a POST is safe to repeat. The application must pair retries with an idempotency strategy and understand the provider's semantics.

Set explicit connection and response timeouts. A request without a practical deadline can occupy workers and connections long after it stopped serving the user. For slow operations, prefer an asynchronous workflow: accept the local request, queue provider work, and show a pending state that can later resolve.

Verify webhook signatures against the raw request body before parsing or transforming it. Enforce a reasonable timestamp tolerance if the provider includes signed timestamps. Keep secrets outside source control and support rotation where the provider allows multiple active secrets.

Acknowledge valid webhooks quickly. Persist the event and queue its processing rather than performing slow business logic in the request. Providers commonly retry when acknowledgements are late, which creates more traffic precisely when the system is already busy.

Store the provider event identifier under a unique constraint. If the same event arrives again, return success without applying it twice. Keep the original payload long enough for investigation, while following the data-minimisation and retention rules appropriate to the information it contains.

Do not assume delivery order. A cancellation may arrive before a creation event because separate network paths and retry schedules are involved. Compare provider timestamps or resource versions where available. When an event cannot be applied, fetch the current remote resource or defer it rather than forcing an impossible state transition.

Integration status is clearer when represented as a small state machine: pending, processing, succeeded, retrying, failed or needs review. Transitions should be deliberate and recorded. A generic boolean such as synced cannot explain whether work has never run, is currently delayed or failed permanently.

Separate the business record from the integration attempt. An order can remain valid while its fulfilment export is retrying. A customer profile can exist while a marketing-provider update has failed. This keeps an external outage from corrupting the application's core state.

Record remote identifiers as soon as they are known. Before creating a new remote resource, check whether a previous attempt already stored one. If the provider offers a lookup by idempotency key or external reference, use it during recovery.

Webhooks improve freshness but should not be the only path to correctness. Events can be disabled, dropped after prolonged failure or mishandled by a bug. A scheduled reconciliation process compares local records with the provider and repairs or flags differences.

Choose the reconciliation window based on business risk and provider limits. Recent pending operations may need frequent checks, while older settled records can be sampled or checked less often. Use cursors and bounded batches so the job can resume after interruption.

Make manual replay safe. An operator should be able to retry a failed operation or reprocess an event without bypassing idempotency. The interface should show attempts, response categories and correlation identifiers, while avoiding secret values and unnecessary personal data.

Monitor outcomes rather than raw request volume. Useful signals include pending age, retry exhaustion, webhook verification failures, rate-limit responses and reconciliation mismatches. Alert on conditions that require action, and attach enough context to find the affected operation.

Use provider sandbox environments for routine development, but also test behaviours that sandboxes often miss: duplicated events, timeouts after success, out-of-order delivery, expired credentials and production-like rate limits. Contract tests can detect response-shape changes without making the full suite depend on a live service.

A dependable integration accepts that uncertainty exists between systems. Stable identities, idempotent operations, bounded retries, verified webhooks, explicit states and reconciliation turn that uncertainty into something the team can observe and manage.