HookRelay

reliable webhook delivery

HookRelay — Multi-tenant Webhook Delivery Service

Webhook delivery that survives crashes, duplicates and receivers that stay down for days.

role
designed and directed it
status
runs locally on Docker Compose · not deployed
scale
2 instances · 1,200 events/s ingest on a laptop
stack
Java 21 · Spring Boot · MariaDB · RabbitMQ
console
Next.js 16 · github.com/mohammed-usmani/hookrelay-console
type
personal project · open source

FIG.01 // HOOKRELAY · ONE WEBHOOK

Accepted once, delivered at least once

The event and its outbox rows commit together; the broker only triggers work, and the delivery row decides what is done.

Accepted once, delivered at least once01 · INGESTAPI × 2idempotency key02 · ONE TRANSACTIONMariaDBevent + outbox rows03 · OUTBOX RELAYSKIP LOCKEDpublisher confirms04 · BROKERRabbitMQquorum queues05 · WORKERSdispatchHMAC-signedFAILED → BACK OFFTTL delay queues5s · 5m · 30m … 3 daysRECEIVERendpoint2xx = doneGUARD LANE · BEFORE EVERY SENDCIRCUIT BREAKERper endpointopens at 10 · one half-open probeRATE LIMITslot on the rowexact across instances, no RedisGIVES UPafter 3 daysdead letter · replay from consoleaccepted = committed:no event without its outbox rowa worker claims the row first,so a duplicate message is a no-op
01

Accepted means it will be sent

An event, one delivery row per subscribed endpoint, and the matching outbox rows commit in a single MariaDB transaction. If the API answers 202, the work exists on disk; there is no window where the event is saved but never queued.

A relay claims outbox rows with FOR UPDATE SKIP LOCKED and marks them published only after RabbitMQ's publisher confirm, so two app instances can run the relay side by side without double-sending or dropping rows.

Delivery is at-least-once, and the database is the source of truth: each worker claims the delivery row with a dispatch token before sending, so a duplicate broker message does nothing.

1 transaction per event2 app instancesat-least-once

02

A receiver that is down cannot drag the rest down

Failed attempts back off on RabbitMQ TTL delay queues: 5s, 5m, 30m, 2h, 5h, 10h, 10h, inside a 3-day window enforced by a deadline in the database. After that the delivery is dead-lettered, an alert event goes to any endpoint subscribed to it, and it can be replayed from the console.

Each endpoint has a circuit breaker that opens after 10 consecutive failures, cools down for 5m, 30m, then 1h, and lets one half-open probe through. An endpoint that keeps failing for 3 days is disabled automatically.

Outbound rate limits are exact across instances without Redis: a worker reserves the next send slot on the endpoint's own row under its row lock.

7 retries over 3 daysbreaker opens at 10no Redis

03

Tested against the real thing, then measured

179 tests: 85 unit tests and 94 integration tests that run against real MariaDB and RabbitMQ in Testcontainers, with WireMock standing in for receivers. Eight ADRs record why each design choice was made.

A final review found 7 concurrency and data bugs, each fixed with a test that failed first. One: the relay held its SKIP LOCKED claim while waiting on broker confirms, and under REPEATABLE READ the gap locks blocked new outbox inserts for up to 5 seconds. The relay now runs at READ COMMITTED.

A k6 run on a laptop with 2 instances accepted 1,200 events/s at p95 121 ms with 0 errors. End-to-end delivery tops out near 640/s; above that, events queue safely and drain once load drops.

1,200 events/s ingestp95 121 ms~640 deliveries/s

screens