Commerce webhooks: handle duplicates, late events and safe replay

An order webhook arrives three times. A worker crashes after submitting a refund but before saving the provider response. Later, an old order update arrives after a cancellation. A consumer that only checks whether it has seen the latest message can still duplicate money or reverse a valid state.

Design three boundaries: authenticated intake, durable processing and reconciled business effects. Deduplicating deliveries helps, but it does not by itself make external payments or stock updates execute exactly once.

Keep three identities

For each provider account or tenant, record delivery ID, event identity where available, entity ID, event type, provider version or effective time, received time and a restricted payload reference. The business operation needs a separate identity tied to the approved action, line and amount or quantity.

Shopify documents X-Shopify-Webhook-Id for delivery deduplication and X-Shopify-Event-Id for correlating deliveries from one merchant action. Separate subscriptions can produce distinct delivery IDs for that action. Scope keys by the correct merchant and provider contract; an unqualified ID string is not a safe global identity design.

Do not deduplicate all future events merely by order ID. One order can legitimately receive several fulfilments, cancellations and refunds. The identity must distinguish those operations while recognising retries of the same one.

Acknowledge durable intake, not completed business work

Verify the provider's signature against the required original bytes and apply its authentication contract. For Shopify HTTPS deliveries, the documented HMAC uses the raw request body. Parsing and reserialising it first can change those bytes.

After verification and basic validation, atomically insert an inbox record under a uniqueness constraint and record the work to be processed. A repeated delivery can be acknowledged once its existing durable record is established. If persistence fails, do not send a success response that implies the message was accepted.

Keep acknowledgement fast enough for the provider's delivery contract. Process slower business work asynchronously. Retain explicit states such as received, processing, completed, retryable failure, quarantined and uncertain external result. A crash must leave recoverable work rather than an in-memory queue that disappears.

Follow the duplicate and crash sequence

Suppose D101 for E77 is delivered three times. The intake audit preserves three attempts, while the unique inbox holds one logical D101 record. A second subscription sends D102 for the same E77. It has its own delivery record, but the processor must not create the same approved business effect again.

For a ₹500 refund against a ₹2,000 capture, the approved effect A9 is 50,000 paise, within the 200,000-paise payment ceiling. Repeating A9 would still be wrong even though two such refunds total only ₹1,000. Amount limits and operation uniqueness solve different problems.

The worker persists A9 and its stable provider request identity before submission. The provider accepts the refund, but the worker crashes before saving the response. A restarted worker sees an uncertain A9, not an unattempted new refund. It queries the outcome where supported or retries the same request under the provider's idempotency contract.

Razorpay's normal-refund idempotency documentation requires the same key and body for a retry of the same request. A new key belongs to a genuinely different authorised operation, not an attempt to escape uncertainty about the first one.

1. Verify and persistAuthenticate the delivery and atomically store the inbox/work record.
2. Claim recoverable workOne worker owns the attempt; expired claims can be recovered safely.
3. Apply the approved effectUse a unique local operation and a stable external request identity.
4. Confirm or reconcileStore the provider result; query uncertain outcomes and compare missing events.

Exception: A crash after external acceptance cannot be fixed by a new operation ID. Retain uncertainty until provider evidence resolves it.

Make local concurrency explicit

Two workers can both read “not processed” before either writes “processed”. A check followed by a later save is not an atomic guard. Use a unique effect record and transaction or equivalent conditional update so only one local operation is created.

Where a local state change creates outgoing work, commit the state and durable outgoing record together, commonly through an outbox pattern. This closes a local crash gap; it does not make the remote system part of the same transaction. The sender still needs retries, stable identity and reconciliation.

Define how workers claim work, renew or expire claims, and recover abandoned attempts. A lease alone does not prevent a slow original worker from completing after another takes over. Guard writes with the current attempt/version and make the external effect idempotent where the provider supports it. If the external service has no safe duplicate protection or outcome lookup, require an owned manual reconciliation before another consequential submission.

Do not confuse arrival order with business order

A late “order accepted” message should not reopen a cancelled order. Use the provider's entity version or authoritative state where available, together with an explicit transition policy. Event timestamps can help, but may not be sufficiently precise or comparable to establish a total ordering.

A refund event arriving after an order update may describe a different state dimension rather than an older event to discard. Keep payment, fulfilment and commercial state separate. Preserve the late message as evidence even if it does not change the current projection.

Replaying an old payload is especially risky after catalogue mappings or policies change. Retain the original mapping and decision version where they determine the approved effect. If current authoritative state conflicts with the replay, quarantine it for review instead of rerunning historical code as a new business action.

Reconciliation covers events that never arrived

A queue can report zero failures while an entire event is absent. Periodically compare provider records with local entity and effect records over an explicit period, including pagination and late-arrival overlap. Deduplicate the overlap using stable identities. Retain unmatched records on both sides.

Shopify's delivery guidance also recommends reconciliation for missed data. Your job's cadence, lookback and retention must fit provider limits and the operational risk. Do not promise that a fixed retry count guarantees completeness.

Engineering owns transport, persistence and recovery. Operations owns quarantined business decisions; finance owns uncertain money effects. Give each queue an age, reason and accountable owner. Logs should retain useful identities without unnecessarily exposing customer payloads.

Prove recovery at each crash boundary

Test invalid signatures, repeated D101, separate D102 for E77, two workers racing, a crash before local commit, a crash after external acceptance, a late state update and a missing event recovered by reconciliation. Inspect final stock and money effects, not merely HTTP response codes.

Acceptance means each approved business action has one established result or an explicit unresolved state. It does not mean every message ran once. Start with a bounded consumer and controlled replay permissions; widen its scope after these failure cases pass.

Replay work, not uncertainty

A reliable webhook consumer preserves what happened and what remains unknown. Its recovery path prevents technical retries from becoming new commercial decisions.

1. Document provider authentication, identity, ordering and retry contracts.
2. Implement durable intake and atomic local effect creation.
3. Define external idempotency, lookup and uncertain-result recovery.
4. Run crash and missing-event exercises before authorising production replay.