Skip to content

Webhooks Are Simple Until You Have to Deliver Them Reliably

4 min read1 views[Backend]
Cover image for "Webhooks Are Simple Until You Have to Deliver Them Reliably"

A webhook looks like the simplest integration in the world: someone sends a POST request to your application, you update a record, and you return 200.

Then the same event arrives twice. Another event arrives late. Your process restarts after accepting a delivery but before doing the work. A customer asks why their subscription is wrong, and the only log says “webhook received.”

The HTTP handler was easy. The delivery contract is where the engineering starts.

Acknowledgment transfers responsibility

Imagine a provider sends subscription.updated. Your server responds successfully, places a callback in memory, and plans to process it after the request finishes.

The process exits. The callback disappears. The provider saw your success response, so it has no reason to retry.

A better boundary is durable acceptance:

Receive bytes
  -> verify authenticity
  -> validate the supported envelope
  -> commit event to an inbox
  -> acknowledge delivery
  -> process asynchronously

The inbox can be a database table. The critical detail is that you acknowledge after the transaction commits. If durable storage fails, the response should reflect that failure so the sender can follow its retry policy.

This does not require a separate broker. It requires a place where accepted work survives a process restart.

Verify the message you actually received

Signature verification belongs before processing. Providers may sign the raw request bytes, so decoding JSON and serializing it again can change the signed representation. Use the provider's supported verifier and its documented timestamp/replay checks.

Stripe explicitly documents raw-body verification, duplicate deliveries, non-guaranteed ordering, and returning a successful response promptly. These are useful concrete constraints, but check your own provider's contract rather than assuming every webhook system behaves identically. Stripe webhook documentation

An event's signature establishes its source under that scheme. It does not tell you that its contents are current, that your account mapping is correct, or that repeating the business operation is harmless.

Deduplicate with a database constraint

For this hypothetical receiver, an inbox might contain:

CREATE TABLE webhook_inbox (
    provider text NOT NULL,
    account_id text NOT NULL,
    event_id text NOT NULL,
    payload jsonb NOT NULL,
    state text NOT NULL DEFAULT 'pending',
    received_at timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (provider, account_id, event_id)
);

Scope the event identifier according to the provider's actual uniqueness guarantee. Don't trust an arbitrary account identifier in the payload as authorization; resolve it through your configured integration.

Inserting with conflict handling makes two simultaneous deliveries converge on one inbox record. A “select first, insert if missing” sequence without a constraint leaves a race.

The inbox state still needs a worker protocol: claiming, retries, ownership, and recovery after a crash. A pending field alone is not that protocol.

Event deduplication and business deduplication differ

One event can be delivered repeatedly. Two different events can also describe a change that should produce only one business effect.

Suppose two notifications both lead to fulfilling order 812. Deduplicating event IDs won't prevent two shipments if the fulfillment operation itself has no uniqueness rule.

Put the durable business invariant where the effect is decided: one fulfillment record for the relevant order and fulfillment type, for example. For an external shipping API, also use a stable operation identifier and the provider's idempotency support where available.

Don't mark an event processed, then perform the effect later without another durable work record. Don't perform an external effect, then assume a local transaction can roll it back.

Arrival order is not business order

Consider these deliveries:

Delivery 1: subscription canceled
Delivery 2: subscription active — generated earlier, delivered late

Blindly overwriting a local status with the latest arrival resurrects the subscription.

The repair depends on the domain. If the provider exposes authoritative resource versions, compare those versions. If the event is a signal that a resource changed, fetching the current resource can be appropriate. If the event represents an immutable business occurrence, such as a specific invoice payment, process that occurrence rather than replacing it with a current-state lookup.

Those approaches solve different problems. A timestamp comparison is not automatically a reliable version protocol, especially when events can share timestamps or describe different aspects of a resource.

Make recovery a normal operation

I would want an admin view showing the event identifier, account, receipt time, attempts, current state, and the last classified error. Payload retention should be deliberate; webhook bodies can contain customer information.

An operator should be able to retry a failed event through the same idempotent path. “Run this SQL and hope” is not a recovery interface.

The receiver's acceptance tests should include:

  • The same event delivered concurrently.
  • A crash immediately after inbox commit.
  • A crash after an external effect but before local acknowledgment.
  • An old event arriving after a newer event.
  • A valid signature with an unsupported event type.

A webhook is a small distributed workflow. Once you design it that way, duplicate deliveries stop being mysterious exceptions and become inputs the system knows how to handle.

# Comments (0)

No comments yet. Be the first.

0/1000