reliable webhook delivery
HookRelay — Multi-tenant Webhook Delivery Service
Webhook delivery that survives crashes, duplicates and receivers that stay down for days.
- role
- designed and directed it
- status
- runs locally on Docker Compose · not deployed
- scale
- 2 instances · 1,200 events/s ingest on a laptop
- stack
- Java 21 · Spring Boot · MariaDB · RabbitMQ
- console
- Next.js 16 · github.com/mohammed-usmani/hookrelay-console
- type
- personal project · open source
FIG.01 // HOOKRELAY · ONE WEBHOOK
Accepted once, delivered at least once
The event and its outbox rows commit together; the broker only triggers work, and the delivery row decides what is done.
Accepted means it will be sent
An event, one delivery row per subscribed endpoint, and the matching outbox rows commit in a single MariaDB transaction. If the API answers 202, the work exists on disk; there is no window where the event is saved but never queued.
A relay claims outbox rows with FOR UPDATE SKIP LOCKED and marks them published only after RabbitMQ's publisher confirm, so two app instances can run the relay side by side without double-sending or dropping rows.
Delivery is at-least-once, and the database is the source of truth: each worker claims the delivery row with a dispatch token before sending, so a duplicate broker message does nothing.
1 transaction per event2 app instancesat-least-once
A receiver that is down cannot drag the rest down
Failed attempts back off on RabbitMQ TTL delay queues: 5s, 5m, 30m, 2h, 5h, 10h, 10h, inside a 3-day window enforced by a deadline in the database. After that the delivery is dead-lettered, an alert event goes to any endpoint subscribed to it, and it can be replayed from the console.
Each endpoint has a circuit breaker that opens after 10 consecutive failures, cools down for 5m, 30m, then 1h, and lets one half-open probe through. An endpoint that keeps failing for 3 days is disabled automatically.
Outbound rate limits are exact across instances without Redis: a worker reserves the next send slot on the endpoint's own row under its row lock.
7 retries over 3 daysbreaker opens at 10no Redis
Tested against the real thing, then measured
179 tests: 85 unit tests and 94 integration tests that run against real MariaDB and RabbitMQ in Testcontainers, with WireMock standing in for receivers. Eight ADRs record why each design choice was made.
A final review found 7 concurrency and data bugs, each fixed with a test that failed first. One: the relay held its SKIP LOCKED claim while waiting on broker confirms, and under REPEATABLE READ the gap locks blocked new outbox inserts for up to 5 seconds. The relay now runs at READ COMMITTED.
A k6 run on a laptop with 2 instances accepted 1,200 events/s at p95 121 ms with 0 errors. End-to-end delivery tops out near 640/s; above that, events queue safely and drain once load drops.
1,200 events/s ingestp95 121 ms~640 deliveries/s