webhook-receiver-hardener

Webhook Receiver Hardener

Build secure, reliable webhook endpoints that handle failures gracefully.

Core Security

Signature Verification: HMAC validation before processing Deduplication: Track processed webhook IDs Idempotency: Safe to process same webhook multiple times Retries: Handle provider retry attempts Rate Limiting: Prevent abuse

Signature Verification

import crypto from "crypto";

export const verifyWebhookSignature = (
  payload: string,
  signature: string,
  secret: string
): boolean => {
  const hmac = crypto
    .createHmac("sha256", secret)
    .update(payload)
    .digest("hex");

  return crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(hmac));
};

// Stripe example
router.post(
  "/webhooks/stripe",
  express.raw({ type: "application/json" }),
  async (req, res) => {
    const sig = req.headers["stripe-signature"];

    try {
      const event = stripe.webhooks.constructEvent(
        req.body,
        sig,
        process.env.STRIPE_WEBHOOK_SECRET
      );
      await processStripeEvent(event);
      res.json({ received: true });
    } catch (err) {
      return res.status(400).send(`Webhook Error: ${err.message}`);
    }
  }
);

Deduplication

// Redis-based dedupe
const WEBHOOK_TTL = 60 * 60 * 24; // 24 hours

export const isDuplicate = async (webhookId: string): Promise<boolean> => {
  const key = `webhook:${webhookId}`;
  const exists = await redis.exists(key);

  if (exists) return true;

  await redis.setex(key, WEBHOOK_TTL, "1");
  return false;
};

// Usage
if (await isDuplicate(webhook.id)) {
  return res.status(200).json({ received: true }); // Already processed
}

Idempotent Processing

export const processWebhook = async (webhook: Webhook) => {
  // Use database transaction with unique constraint
  try {
    await db.transaction(async (trx) => {
      // Insert webhook record (unique constraint on webhook_id)
      await trx("processed_webhooks").insert({
        webhook_id: webhook.id,
        processed_at: new Date(),
      });

      // Do actual processing
      await performWebhookAction(webhook, trx);
    });
  } catch (err) {
    if (err.code === "23505") {
      // Unique violation
      console.log("Webhook already processed");
      return; // Idempotent - already processed
    }
    throw err;
  }
};

Retry Handling

// Acknowledge immediately, process async
router.post("/webhooks/provider", async (req, res) => {
  // Verify signature
  if (!verifySignature(req.body, req.headers["signature"])) {
    return res.status(401).send("Invalid signature");
  }

  // Return 200 immediately
  res.status(200).json({ received: true });

  // Process async
  processWebhookAsync(req.body).catch((err) => {
    console.error("Webhook processing failed:", err);
    // Will be retried by provider
  });
});

// Exponential backoff for provider retries
// Attempt 1: immediate
// Attempt 2: +5 minutes
// Attempt 3: +15 minutes
// Attempt 4: +1 hour
// Attempt 5: +6 hours

Error Responses

// Return appropriate status codes
const webhookHandler = async (req, res) => {
  // 400: Malformed payload (won't retry)
  if (!isValidPayload(req.body)) {
    return res.status(400).json({ error: "Invalid payload" });
  }

  // 401: Invalid signature (won't retry)
  if (!verifySignature(req.body, req.headers["signature"])) {
    return res.status(401).json({ error: "Invalid signature" });
  }

  // 200: Already processed (idempotent)
  if (await isDuplicate(req.body.id)) {
    return res.status(200).json({ received: true });
  }

  // 500: Processing error (will retry)
  try {
    await processWebhook(req.body);
    return res.status(200).json({ received: true });
  } catch (err) {
    console.error("Processing error:", err);
    return res.status(500).json({ error: "Processing failed" });
  }
};

Monitoring & Runbook

## Webhook Incidents

### High Error Rate

1. Check provider status page
2. Review recent code deploys
3. Check signature secret rotation
4. Verify database connectivity

### Missing Webhooks

1. Check provider sending (their dashboard)
2. Verify endpoint is accessible
3. Check rate limiting rules
4. Review dedupe cache TTL

### Duplicate Processing

1. Check dedupe cache connectivity
2. Verify unique constraints
3. Review idempotency logic

Best Practices

• Verify signature BEFORE any processing
• Return 200 quickly, process async
• Dedupe with Redis + database constraints
• Log all webhook attempts
• Monitor processing latency
• Set up alerts for failures
• Document expected payload schemas

Output Checklist

• Signature verification
• Deduplication mechanism
• Idempotent processing
• Async processing pattern
• Proper status codes
• Error logging
• Monitoring/alerts
• Incident runbook