Webhooks that survive production: verification, idempotency and retries

A webhook integration usually works the first time it's tested. A payment is made in a sandbox, a request arrives, a record is updated. The problems arrive later, under conditions a test doesn't create: duplicate deliveries, events out of order, a slow database, a deploy in the middle of a burst of traffic.

Most of these failures are avoidable with a few decisions made up front.

Verify before you trust

A webhook endpoint is a public URL that changes data. Anyone who finds it can send it a request. Most providers sign each delivery, typically with an HMAC of the raw request body using a shared secret, and the receiver must check that signature before doing anything else.

Two details catch people out:

• Verify against the raw body, byte for byte. Parsing the JSON and re-serialising it will change whitespace or key order and break the signature.
• Check the timestamp if the provider includes one, and reject old deliveries. A valid signature on a request captured last week is still a replay.

Acknowledge fast, process later

Providers wait only a few seconds for a response before treating a delivery as failed and retrying. If the handler does real work, such as calling other APIs, sending emails or generating documents, it will eventually be too slow, and the provider will send the same event again while the first attempt is still running.

The reliable pattern is:

1. Verify the signature
2. Store the event (or put it on a queue)
3. Return a 2xx immediately
4. Process the stored event separately

This also means a bug in processing doesn't lose events. They're recorded, and can be reprocessed once the bug is fixed.

Assume every event arrives more than once

Webhook delivery is almost always at least once. The provider promises you'll get each event, but it may arrive more than once. So processing must be idempotent: handling the same event twice must leave the system in the same state as handling it once.

In practice:

// Record the provider's event ID with a unique constraint.
// A second insert with the same ID fails, and we skip processing.
const inserted = await db.webhookEvents.insertIfAbsent({ id: event.id, type: event.type });
if (!inserted) return; // already handled

Where the provider doesn't supply a stable event ID, derive one from the fields that identify the change.

Don't rely on order

Events can arrive out of order: an "updated" before a "created", or two updates swapped. If the event carries a timestamp or version, compare it with what you've stored and ignore anything older. If it doesn't, consider treating the webhook as a notification only, and fetch the current state of the object from the provider's API before acting on it.

Retry with limits, and make failures visible

When processing fails because a downstream service is unavailable, retry with increasing delays rather than immediately. After a fixed number of attempts, stop and move the event to a list that someone will actually look at. An integration that fails silently is worse than one that fails loudly, because the first anyone hears of it is a customer.

Useful signals to monitor:

• Signature failures (a sudden rise often means a rotated secret)
• Processing failures by event type
• Age of the oldest unprocessed event
• Events that have exhausted their retries

Reconcile on a schedule

Even with all of the above, some events will be missed, perhaps because of a provider outage or an endpoint misconfigured during a deploy. For anything where correctness matters, such as stock levels or payment status, a scheduled job that compares both systems and corrects drift is the safety net that makes the rest trustworthy.

None of this is exotic, but it's rarely in the first version. If you have an integration that "mostly works", these are the places to look first. We cover this in more depth on API integration.