How to build webhook fan-out

What matters here

• Verify the signature before parsing the body. An unverified webhook is an unauthenticated write into your systems.
• Acknowledge first, fan out second. Holding the acknowledgement open makes the slowest consumer everybody problem.
• Retry per consumer, not per event. One failing consumer must not redeliver to the four that succeeded.
• Give each consumer its own dead letter, and alert on the first entry.
• Store the raw event before processing. Replay is only possible if the original survived.
• Deduplicate on the sender event id. Every provider redelivers, and it is not a fault.

Who this is for

You run platform or integration engineering. Several teams want the same provider events, the provider allows one endpoint, and today somebody forwards them with a script.

How it works in practice

Everything that sits between a provider firing an event and every consumer having handled it.

1. 1

   The signature is verified

   Before the body is parsed, because an unverified webhook is an unauthenticated write.

2. 2

   The raw event is stored

   Exactly as received, which is the only thing that makes replay possible later.

3. 3

   The delivery is acknowledged

   Immediately, so the sender stops waiting and stops retrying.

4. 4

   The event is fanned out per consumer

   Each independently, none of them able to affect another.

5. 5

   Each consumer retries on its own schedule

   With backoff, and its own dead letter when it gives up.

6. 6

   Anything undelivered is visible

   Because a dead letter store nobody watches is data loss with extra steps.

Paste these into Claude Code or Codex with the Tray Headless plugin installed. Each stage runs on its own. The systems named in them are the worked example rather than a requirement, and every prompt says so.

Once per project, run /tray-workflows:set-workspace to pick the workspace these build in. Point it at a sandbox first.

1. 1

   Start here: verify the signature, then store what arrived

   An unverified webhook is an unauthenticated write.

   Headless skills build-workflow

   Copy

   Use build-workflow. The systems in play are Stripe and the other
   providers we receive webhooks from, or whatever we run in those seats.
   
   Verify the signature before parsing the body, on the raw bytes exactly as
   received. Parsing first and verifying afterwards means the parser has
   already run on untrusted input.
   
   Reject anything that fails verification, and anything whose timestamp is
   outside a tolerance window, which is what stops a captured request being
   replayed at you later.
   
   Then store the raw event: the body as bytes, every header, the received
   time and the sender event id. Store it before any processing, because
   this record is the only thing that makes a replay possible.

2. 2

   Acknowledge immediately, then fan out

   The sender is waiting, and it will retry.

   Headless skills build-workflow

   Copy

   Use build-workflow. Return the acknowledgement as soon as the event is
   verified and stored. Do not wait on any consumer.
   
   Then fan out: one delivery per consumer, each independent, each with its
   own status.
   
   The synchronous version is the trap. Five consumers delivered inline,
   one of them slow, and the acknowledgement misses the sender timeout. The
   sender redelivers, and the four fast consumers process the event twice
   while the slow one is still working on the first copy.
   
   Deduplicate on the sender event id at intake. Every provider redelivers
   sometimes and it is not a fault, so a repeat id is acknowledged and not
   fanned out again.

   The sender timeout is usually a few seconds and is not negotiable. Anything you do before acknowledging is spent from that budget.

3. 3

   Retry per consumer, with its own dead letter

   One failing consumer must not affect the four that worked.

   Headless skills tray-patterns

   Copy

   Give each consumer its own delivery state:
   
     Its own retry schedule with exponential backoff and jitter, so a
     consumer coming back up is not hit by every pending event at once
     Its own maximum attempts
     Its own dead letter store, holding the event and the last error
     Its own circuit breaker, so a consumer that is down stops being
     hammered and starts being queued
   
   Never retry the whole fan-out. Redelivering to consumers that already
   succeeded is how a retry mechanism becomes a duplicate generator.
   
   Alert on the first dead letter for a consumer, not on the hundredth. The
   first one is a fault; the hundredth is an outage somebody should already
   know about.

4. 4

   Make replay a normal operation

   A consumer was broken for six hours and needs those events.

   Headless skills tray-gotchas

   Copy

   Use tray-gotchas, then build replay properly:
   
     Replay a time range to one named consumer, never to all of them
     Replay from the stored raw event, so what the consumer receives is what
     the provider sent
     Mark replayed deliveries as replays, so a consumer can tell
     Rate limit the replay, because six hours of events delivered in one
     minute is an outage of its own
   
   Replay is the reason to store raw events, and it is the operation people
   need at exactly the moment they are least able to write it. Build it
   before you need it.
   
   Handle a consumer being added. A new consumer usually wants the recent
   past, and that is a replay with a start time, not a special case.

5. 5

   Report what is stuck, not what is flowing

   Throughput is not the interesting number.

   Headless skills tray-patterns

   Copy

   Report per consumer: delivery success rate, current lag, retry counts,
   dead letter volume, and time since the last successful delivery.
   
   Time since last success is the one that catches a silent failure. A
   consumer receiving nothing looks identical to a consumer with nothing to
   receive until you compare it against the others.
   
   Also report events received but not fanned out to anybody, which means a
   consumer was removed and nothing took its place, and signature failures,
   which are either a rotated secret or somebody probing the endpoint.
   
   Do not lead with total events delivered. It goes up and to the right
   whatever is broken.

6. 6

   Prove it works, then hand the consumer list to platform engineering

   Because consumers get added and retry policies need tuning.

   Headless skills tray-patterns

   Copy

   Run the per-step schema checks and the whole-workflow audit before this
   touches production. Point one consumer at an endpoint that returns errors
   and confirm the other consumers are unaffected, the dead letter fills and
   the first entry alerts.
   
   Then open the same workflow in Tray Build so platform engineering can add
   consumers, change retry policies and run a replay from the visual
   canvas.

What it connects to

Events come from providers and go to consumers, with a stored copy in between so a replay is possible.

Named systems are the ones most teams run, not the only ones that work. Each is an authentication in your Tray workspace, referenced by name, so the workflow never holds a credential. Where we have a connector page, the name links to it.

Questions people ask

Related guides