Idempotency in Webhook Receivers: Patterns We Recommend to Partners

Webhooks are at-least-once. Plan for it.

Every reliable webhook system — CRMBridge included — sits somewhere on the spectrum between "fire and forget" and "exactly-once delivery." In practice, nobody is exactly-once. We retry on transient failures (5xx, timeouts, network resets) on a 1m / 2m / 4m / 8m schedule because dropping a real-time event is worse than re-delivering one. Other webhook providers do the same.

The consequence: your receiver will see the same event more than once. Sometimes within seconds (your 200 was lost in transit). Sometimes minutes apart (the connection dropped after your handler committed but before the response made it back). Occasionally hours apart (a downstream system was being restored from a snapshot). If the second delivery does anything different than the first — sends a second SMS, charges a second invoice, opens a second support ticket — you have a bug. The fix is idempotency: design your handler so the second, third, and Nth delivery of the same event are visibly safe no-ops.

The patterns below are what we recommend to every CRMBridge partner. They are not specific to our platform — they apply equally to Stripe, GitHub, Twilio, or any webhook source you integrate with.

Pattern 1: Pick an idempotency key, put it in a unique index

The simplest, most durable dedupe is a unique index on a column that identifies "this exact event." When the second delivery hits, the insert fails on the constraint, and you treat it as a no-op success.

For CRMBridge webhooks, a good key is the SHA-256 of the raw request body. The body is identical across re-deliveries of the same event (we sign byte-for-byte equivalent payloads), so the hash is stable. Some partners prefer a logical key like {Entity}:{Action}:{PrimaryKey}:{CommandTime} — that works too, with the caveat that you need to be sure those fields uniquely identify an event in your domain.

CREATE TABLE WebhookDedupe (
    IdempotencyKey CHAR(64) PRIMARY KEY,    -- sha256 hex
    ReceivedAt     DATETIME2 NOT NULL,
    Entity         NVARCHAR(64),
    EventType      NVARCHAR(32)
);

-- Optional: TTL job to drop rows older than your retry window + a buffer
DELETE FROM WebhookDedupe WHERE ReceivedAt < DATEADD(day, -7, SYSUTCDATETIME());

Seven days is comfortably wider than the longest retry window any reliable webhook provider uses (CRMBridge tops out at 15 minutes), but small enough that the table stays cheap. Adjust to whatever your storage budget tolerates.

Pattern 2: Apply the effect and record the key in a single transaction

The most common subtle bug: the handler does its work, returns 200, then crashes before recording the dedupe key. The retry comes in, sees no key, does the work again. To avoid this, both the side-effect and the dedupe insert must commit together — or both roll back.

If your effect lives in the same database as your dedupe table, this is one transaction:

BEGIN TRANSACTION;

INSERT INTO WebhookDedupe (IdempotencyKey, ReceivedAt, Entity, EventType)
VALUES (@key, SYSUTCDATETIME(), @entity, @event);
-- If this insert fails on the unique constraint, the whole tx aborts and we
-- return 200 without re-applying the effect.

-- Then the actual side effect:
INSERT INTO PatientAuditLog (...) VALUES (...);

COMMIT;

If the effect is in a different system — sending an email, calling a payment processor, posting to Slack — you can't transact across both. The trick is to insert the dedupe row first, in its own short transaction, and use it as the gate:

// Pseudocode:
try {
    db.Insert(new WebhookDedupe { Key = key, Status = "pending" });
} catch (UniqueConstraintViolation) {
    // We've seen this event before. Look up its status:
    var existing = db.GetDedupe(key);
    if (existing.Status == "completed") return Ok();   // safe replay
    if (existing.Status == "pending")   return Conflict(); // another worker is mid-flight
}

// We are the first to claim this event. Do the side effect.
SendEmail(...);

db.Update(key, status: "completed");
return Ok();

The trade-off is that a crash between "pending" and "completed" leaves a dangling row and the event is never retried. Mitigate with a sweep job that flips long-stale "pending" rows back to "expired" so the next webhook delivery (if any) can re-claim them.

Pattern 3: Verify the signature before you do anything else

CRMBridge webhooks are signed with HMAC-SHA256 in the X-Hub-Signature-256 header, computed over the raw request body using the per-subscription secret you received at signup. Verify the signature before you parse the body, before you write to your dedupe table, before you do anything else.

// C# example:
public async Task<IActionResult> Receive() {
    using var ms = new MemoryStream();
    await Request.Body.CopyToAsync(ms);
    var raw = ms.ToArray();

    var signature = Request.Headers["X-Hub-Signature-256"].ToString();
    if (!VerifyHmac(raw, signature, _secret))
        return Unauthorized();

    // Only NOW parse and process
    var payload = JsonSerializer.Deserialize<WebhookPayload>(raw);
    ...
}

private static bool VerifyHmac(byte[] body, string headerValue, string secret) {
    if (!headerValue.StartsWith("sha256=")) return false;
    var expected = headerValue["sha256=".Length..];
    using var hmac = new HMACSHA256(Encoding.UTF8.GetBytes(secret));
    var computed = Convert.ToHexString(hmac.ComputeHash(body)).ToLowerInvariant();
    return CryptographicOperations.FixedTimeEquals(
        Encoding.UTF8.GetBytes(expected),
        Encoding.UTF8.GetBytes(computed));
}

Three details that catch people: (1) read the raw bytes, not a re-serialized JSON object — even one byte of whitespace difference breaks the hash; (2) use a constant-time comparison to defend against timing oracles; (3) reject when the header is missing rather than treating it as "no signature, allow." A misconfigured proxy stripping the header is a much more likely failure mode than a CRMBridge bug.

Pattern 4: Respond fast, do real work asynchronously

CRMBridge gives your endpoint a few seconds to respond before treating the delivery as failed and queuing a retry. If your handler does heavy work synchronously — calling third-party APIs, generating reports, running ML inference — you will time out, get retried, time out again, and end up doing the work three or four times in parallel. Worse, your dedupe table will only catch the duplicates after the fact.

The fix is the standard "ingest then process" pattern:

1. HTTP handler verifies signature, computes idempotency key, inserts into a durable queue (SQS, RabbitMQ, Kafka, Postgres+SKIP LOCKED, even a SQL Server table) with the key as a unique constraint, returns 200. This should take well under 100 ms.
2. A worker process picks up the queue entry, does the actual work, and marks it complete.
3. If the worker crashes mid-work, the queue's visibility-timeout / leasing semantics re-deliver the entry — and your downstream effect should itself be idempotent on the same key, so the second worker run is safe.

This pattern also gives you back-pressure: if a downstream API is slow, the queue drains slower, but you never miss a webhook from CRMBridge.

Pattern 5: Make the side effect itself idempotent where possible

Dedupe tables are a safety net. The first line of defense is writing effects that are naturally safe to repeat:

• UPSERT instead of INSERT when syncing PMS data into your own tables — "create or update by PatientId" is naturally idempotent, "always insert" is not.
• Use idempotency keys when calling third-party APIs. Stripe, Twilio, SendGrid, and most modern APIs accept an Idempotency-Key header. Pass the webhook hash through; if the call repeats, the third party will recognize it and return the original response instead of doing the work twice.
• Conditional state transitions. Instead of "set status = paid," write "set status = paid WHERE status = unpaid." The second run is a no-op rather than a regression.
• Avoid append-only side effects. "Add a comment to the chart for every webhook" guarantees duplicates the first time delivery hiccups. Either UPSERT a single comment keyed by event id, or accept duplicates as a known cost.

When the effect is naturally idempotent, the dedupe table is just a latency optimization — you skip work you don't need to do. When the effect isn't naturally idempotent (sending an SMS, posting to a social channel), the dedupe table is the only thing standing between you and an angry customer.

Pattern 6: Tolerate out-of-order delivery

Idempotency handles the same event twice. Out-of-order handles a different problem: Update arriving before the Insert, or Delete arriving before the Update. CRMBridge preserves intra-table order per (Business, Location), but if your processing pipeline parallelizes work across workers, you can re-order events on your end.

Two strategies that work well:

• Use the event's logical timestamp, not your wall clock. CRMBridge events carry a CommandTime field. Store it on the row you write; on subsequent updates, only apply if the incoming CommandTime is newer than what's stored. Older events become idempotent no-ops.
• Partition workers by primary key. Hash PatientId (or whatever the row's PK is) and route all events for that key to the same worker. Within a single worker, processing is serial, so order is preserved without distributed coordination.

For high-volume systems we usually recommend the second strategy. It scales linearly and avoids the distributed-lock complexity of "wait for the prior event to commit before applying this one."

Anti-patterns we see (and how they bite)

"We dedupe in memory"

A LRU cache in your web process catches duplicates within a few seconds, then forgets. Re-deliveries from the 1m, 2m, 4m, 8m retry windows blow right past it — and any horizontal scaling means workers don't share the cache anyway. Dedupe must be persistent and shared.

"We rely on the database to detect duplicates downstream"

"Inserting the same patient twice will just fail on the unique key, no big deal" — until that second insert fails halfway through a multi-row transaction and rolls back the rest of the work. The dedupe check belongs at the entry point, not buried inside business logic.

"We respond 200 immediately and process later, no queue"

Fire-and-forget Task.Run, in-memory channels, "I'll just spin off a thread" — all of these lose work the instant the process restarts. If you're going to ack the webhook, the work has to be on durable storage before the ack goes out.

"We use the wall-clock timestamp from the request"

Dedupe windows keyed on "received within N minutes" miss re-deliveries that span the window boundary, and they double-count legitimately distinct events that happen to arrive in close succession. Use the deterministic hash or logical key. Time-based dedupe is a leaky abstraction.

The short version

1. Verify the HMAC signature on the raw bytes, constant-time, before anything else.
2. Compute an idempotency key — SHA-256 of the body works in 95% of cases.
3. Insert that key into a durable store with a unique index, transactionally with your side effect (or as a gate before the side effect, with a sweep for stranded "pending" rows).
4. Respond 200 fast; do heavy work in a queue-backed worker that is itself idempotent on the same key.
5. Make the side effect naturally idempotent where you can (UPSERT, conditional updates, third-party Idempotency-Key headers).
6. Use logical timestamps, not wall-clock, when deciding whether an event is "newer" than what you already have.

Get those six right and your webhook receiver will absorb our retries (and any future failure mode we add) without you ever shipping a "we double-billed the customer" hotfix.