Designing Webhook Delivery You Can Defend in an Audit

Most webhook designs answer the question "did it get there?" An auditor asks a harder one: prove it. Prove that the critical lab value that crossed a threshold at 03:14 produced an alert, that the alert was attempted, that it was attempted again when the receiving system 502'd, that the duplicate which arrived at 03:16 did not page the on-call nurse twice, and that when all of it failed, a human was told inside the window your clinical policy promised.

That is a different engineering problem from "POST some JSON and retry on failure." It changes what you store, how long you store it, and which component owns the truth.

The delivery record is the artifact, not the log line

The most common mistake is treating delivery as a side effect of a background job. The job runs, Rails.logger.info fires, and the evidence of delivery lives in a log aggregator with 30 day retention and no schema. Six months later you cannot reconstruct an attempt sequence, and your log retention is shorter than your documentation retention obligation, the HIPAA Security Rule requires that policies and procedures, and the records of any action, activity, or assessment the rule requires you to document, be retained six years from creation or last effective date, whichever is later (45 CFR §164.316(b)(1) and (b)(2)(i)).

Make delivery a first class database row:

create_table :webhook_deliveries do |t|
  t.references :endpoint,     null: false
  t.string   :event_id,       null: false  # producer-assigned, stable forever
  t.string   :event_type,     null: false
  t.string   :payload_digest, null: false  # SHA-256 of the exact signed bytes
  t.string   :state,          null: false, default: "pending"
  t.integer  :attempt_count,  null: false, default: 0
  t.datetime :deliver_by                   # the clinical deadline, not a guess
  t.datetime :succeeded_at, :dead_lettered_at
  t.timestamps
end
add_index :webhook_deliveries, %i[endpoint_id event_id], unique: true

with a child webhook_delivery_attempts row per HTTP call recording attempted_at, duration_ms, response_status, a truncated redacted response body, and the failure classification. The parent row is the fact, this event was owed to this endpoint. The children are the narrative.

The payload_digest column matters more than it looks. The Security Rule's integrity standard (§164.312(c)(1)) asks you to protect ePHI from improper alteration, and its addressable specification (§164.312(c)(2)) asks for "electronic mechanisms to corroborate" that data has not been altered. A digest of the exact bytes you signed lets you prove, years later, that the payload in your archive is the payload that went over the wire, without retaining the payload itself.

Retries: a budget, not a default

Sidekiq's default backoff is (count ** 4) + 15 seconds plus a jitter of rand(10 * (count + 1)), 25 retries over roughly 20 days. That is an excellent default for a thumbnail-resizing job and an actively dangerous one for clinical alerting. A critical value alert delivered on day 19 is not a late delivery; it is a false record of delivery. Worse, it is a false record your system will cheerfully mark succeeded.

Replace the default with an explicit retry budget derived from the clinical deadline:

class WebhookDeliveryJob
  include Sidekiq::Job
  sidekiq_options retry: 8, dead: false   # dead-lettering is ours, not Sidekiq's

  sidekiq_retry_in do |count, _exception, msg|
    deadline = Time.iso8601(msg["args"].first["deliver_by"])
    next :kill if Time.current > deadline # :kill/:discard need Sidekiq >= 6.5.2
    rand(0..(2**count))                   # full jitter
  end
end

Note the next, not break: Sidekiq stores this block and calls it later, so break raises LocalJumpError. The block's third argument is the job hash, which is why the deadline is read out of the serialized job arguments rather than pulled from thin air.

Two things are load bearing. First, jitter. Deterministic backoff means a receiver that just came back from a 90 second outage gets your entire backlog in one synchronized burst and falls over again. Full jitter, pick uniformly across the whole interval rather than adding a random fraction on top of a fixed delay, spreads the recovery out.

Second, error classification decides whether you retry at all. Retrying a 422 twelve times is not resilience; it is a self-inflicted denial of service against a partner plus twelve worthless rows in your audit trail. Treat timeouts, connection resets, 429, and 5xx as retryable. Treat 400, 401, 403, 404, and 422 as terminal, they signal a contract or credential problem that time will not fix. Record the classification on the attempt so the distinction is visible as evidence rather than inferred from a retry count.

The deadline check is what converts a technical policy into a clinical one. When the budget expires, the event doesn't quietly keep retrying, it dead letters and escalates, which is the only honest outcome.

Idempotency: assign the key at the source of truth

Idempotency fails when the key is generated in the wrong place. If your worker mints a UUID at send time, a retry after a network timeout, where the receiver actually processed the request but the response was lost, produces a new key and a duplicate alert. The nurse gets paged twice, and your reconciliation report insists both were legitimate.

The key must be assigned at the moment of the state change that caused the event, in the same transaction as the record itself. The transactional outbox pattern makes that structural:

ActiveRecord::Base.transaction do
  result = LabResult.create!(attrs)
  WebhookEvent.create!(
    id: SecureRandom.uuid,          # the idempotency key, born here
    event_type: "lab_result.critical",
    subject: result
  )
end

Either the lab result and its event both exist, or neither does. No dual write gap, no "we sent an alert for a result that rolled back."

Ship that key in the payload as event.id, echo it in an Idempotency-Key header, and sign the request with an HMAC over a timestamp plus the raw body. The Stripe style scheme is widely copied and worth copying rather than reinventing: the signed payload is timestamp + "." + raw_body, HMAC-SHA256 keyed with the endpoint's signing secret, compared in constant time, with the five minute timestamp tolerance Stripe's libraries default to so a captured request can't be replayed next week. Verify against raw bytes before any JSON round trip, re serialization changes whitespace and breaks the MAC.

Then document the receiver's half of the contract explicitly, because you are asking a partner to uphold a guarantee: store event.id, reject duplicates for at least 30 days, and return 2xx only after the event is durably persisted, not after it's accepted into memory. An idempotency scheme only works when both sides implement it, and "we told them in a PDF" is a weaker audit answer than a conformance test you run against their staging endpoint on a schedule.

Dead letters need an owner and a clock

A dead letter queue nobody reads is worse than none, it converts a loud failure into a quiet one while creating the appearance of diligence. In a regulated context a dead letter is an unfulfilled obligation, and it needs three things Sidekiq's dead set doesn't give you:

A reason code. endpoint_unreachable, signature_rejected, deadline_exceeded, permanently_rejected. Free text exception messages don't aggregate, and aggregation is how you notice one endpoint has been failing for six days.
An escalation path. Dead lettering triggers the fallback channel, SMS to the on-call, a page, a ward dashboard row, inside the same deadline the webhook had. The webhook is a transport, not the promise.
An operator visible replay that preserves the original key. Replay re sends the same event_id. If the receiver honors idempotency, replay is safe; if replay creates duplicates, you've discovered their implementation is broken, during a drill, rather than during an incident.

Track dead letters as an SLO, not a backlog. "Zero events dead lettered without escalation within 5 minutes" is a measurable claim. "We have a DLQ" is not.

What the auditor actually asks

Five questions. A well built system answers each with a query rather than Slack archaeology:

For event X, show every attempt with timestamps and responses → webhook_delivery_attempts.
Show that duplicates could not cause duplicate clinical action → idempotency key provenance plus receiver conformance tests.
Show what happened when delivery failed → dead letter reason codes and escalation records.
Show that payloads were not altered in transit → HMAC signing plus payload_digest, the integrity controls §164.312(e)(2)(i) asks for.
Show that the controls you documented were actually operating → delivery records retained alongside the six year documentation, which is affordable precisely because payloads carry resource references ({type, id}) rather than PHI, with receivers fetching detail over an authenticated API.

That last choice is the quiet one that makes the rest possible. Keeping PHI out of the webhook body shrinks your delivery archive toward an ordinary, cheap, indefinitely retainable ledger of who was owed what, and when they got it.

Build for the second question, not the first. "It delivers" is table stakes. "Here is the evidence" is the product.

Sources: Sidekiq error handling & retry backoff, Sidekiq job_retry.rb source, Stripe webhook signature verification, Webhook replay prevention, 45 CFR §164.312, 45 CFR §164.316

Navigation