Your App Is Getting Ghosted and Doesn't Know It: The Webhook Survival Guide
Fire and Forget (and Pray): The unfiltered truth about webhooks
It’s Friday. Your on-call phone buzzes. A customer is furious — they paid for an order an hour ago, but their account still shows pending.
You check Stripe. The payment went through. You check your database. Crickets. Somewhere between Stripe’s servers and yours, a message was born, traveled across the internet… and vanished into the void.
Welcome to your first real lesson in webhooks. Pull up a chair.
The Mental Model Shift Nobody Warns You About
Most of us spend years thinking in polling: “I’ll check every 5 minutes if something changed.” It’s comfortable. You’re in control. You ask, you receive, you move on.
Webhooks flip this completely. Instead of you asking, someone else knocks on your door — once, immediately, when something happens. Think of it like switching from checking your mailbox every morning to having the postal service ring your doorbell the moment a package arrives.
The catch?
If you’re not home when they knock, the package doesn’t wait. Most webhook providers will retry a few times, then give up. Your door stays unanswered, and their logs show delivered. Yours show nothing.
This is the mindset shift: you are no longer the caller. You are the receiver. And receivers have responsibilities.
Why Things Go Sideways
The Ghost Event Problem
Picture a GitHub Actions workflow. A developer pushes code, GitHub fires a webhook to trigger your CI pipeline — but your server was mid-deployment and returned a 503. GitHub retried twice. Both failed. Your pipeline never ran. Your dashboard says “no recent builds.” No alert. No error. Just silence.
This is the ghost event: the sender thinks it delivered, the receiver never knew it happened. It’s the most dangerous failure mode because nothing looks broken.
The Duplicate Delivery Surprise
Now flip it. Your server received the Stripe payment.succeeded event, processed it, but took 24 seconds to respond. Stripe’s timeout is 20 seconds.
It retried. Your server processed it again. Your customer just got two “Thank you for your order!” emails and — if you’re not careful — two shipments.
Stripe’s own documentation is explicit: webhook delivery is at-least-once, not exactly-once. Duplicates are expected. Planning for them is not optional.
Out-of-Order Delivery
IoT sensors are a great example here. A smart thermostat sends temperature.critical at 9:00:01, then temperature.normal at 9:00:02. Due to network jitter, your server receives them in reverse order. You log “all clear” and ignore the critical alert that arrived late. Your office bakes.
Events arrive in the order the network decides, not the order they were sent.
The Three Laws of Webhook Survival
After enough production scars, patterns emerge. Here are the three you can’t skip:
1. Always Verify the Knock
When someone rings your doorbell at 2 AM claiming to be a delivery person, you check through the peephole first. For webhooks, that’s HMAC signature validation (Stripe uses stripe-signature, GitHub X-Hub-Signature, Shopify similar).
Most providers (Stripe, GitHub, Shopify) include a signature header computed from the payload + a shared secret. Verify it before processing anything:
import hmac, hashlib
def is_valid_signature(payload: bytes, secret: str, header_sig: str) -> bool:
expected = hmac.new(secret.encode(), payload, hashlib.sha256).hexdigest()
return hmac.compare_digest(expected, header_sig)Skip this and you’re leaving your front door open. Anyone can POST to your endpoint and trigger actions.
2. Make Your Handler Idempotent
Idempotency means processing the same event twice produces the same result as processing it once. The simplest approach: store the event ID, check before processing.
if db.exists(event_id=”evt_abc123”):
return # already processed, skip
db.process_and_mark(event_id=”evt_abc123”)No idempotency check = duplicate Stripe charges, duplicate emails, duplicate database rows. It will happen.
3. Respond Fast, Process Slow
Webhook providers have short timeouts — typically 3–30 seconds. If your handler does anything slow (email sending, database writes, calling other APIs), you will hit that timeout and trigger retries.
The fix is boring but bulletproof: acknowledge immediately with a 200, enqueue the actual work. Your webhook handler is a mailroom, not a factory floor.
Receive, stamp, pass along.
Webhooks vs WebSockets: The Doorbell vs The Open Phone Line
Imagine you’re waiting for important news.
Polling is you nervously checking your mailbox every 5 minutes — wasteful, exhausting, and you still miss things.
WebSockets are keeping a constant video call open with the sender. You can talk back and forth in real time (“Hey, did you see that?”). Perfect for live chat, collaborative editing, or multiplayer games where both sides need to chatter continuously. But that open line burns resources, requires constant connection management, and scales poorly when you have thousands of users.
Webhooks, on the other hand, are the classic doorbell. The sender knocks only when something actually happens, delivers the package (one-way), and leaves. No open connection. No wasted resources. Your server can be asleep until the knock comes.
Use webhooks when you just need to know “something changed” — like Stripe telling you a payment succeeded, or GitHub announcing a new push. It’s fire-and-forget (with the survival rules we just covered). Simple, scalable, and perfect for server-to-server integrations.
Choose the right tool for the job:
constant conversation → WebSocket.
One important knock → Webhook.
