Building a Webhook System in Spring Boot — Delivery, Retries, and Signature Verification

The reliability requirements

A webhook is an HTTP POST to a customer-configured URL, notifying them that something happened in your system. The basic implementation — webClient.post().uri(endpoint.url()).bodyValue(payload).retrieve().block() — works for the happy path and fails in production in several ways:

• The customer's endpoint is temporarily unavailable — the delivery fails and is lost
• The delivery call blocks a thread for the duration — at scale, this exhausts thread pools
• The customer can't verify that the request actually came from your system
• There's no record of what was sent, when, and whether it was delivered

A production webhook system needs: asynchronous delivery, retry with backoff, signature verification on outgoing webhooks, idempotency on incoming webhooks, and a delivery log for debugging.

The data model

Three tables underpin the webhook system:

-- Webhook endpoint registrations (customer-configured)
CREATE TABLE webhook_endpoints (
    id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id   VARCHAR(255) NOT NULL,
    url         VARCHAR(2048) NOT NULL,
    secret      VARCHAR(255) NOT NULL,       -- HMAC signing secret
    event_types TEXT[] NOT NULL,             -- which events to receive
    active      BOOLEAN NOT NULL DEFAULT true,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- Webhook deliveries (one per endpoint per event)
CREATE TABLE webhook_deliveries (
    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    endpoint_id     UUID NOT NULL REFERENCES webhook_endpoints(id),
    event_type      VARCHAR(255) NOT NULL,
    payload         JSONB NOT NULL,
    status          VARCHAR(50) NOT NULL DEFAULT 'PENDING',
    attempts        INTEGER NOT NULL DEFAULT 0,
    next_attempt_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    last_response   INTEGER,                 -- HTTP status of last attempt
    last_error      TEXT,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    delivered_at    TIMESTAMPTZ
);

CREATE INDEX idx_webhook_deliveries_pending
    ON webhook_deliveries(next_attempt_at)
    WHERE status = 'PENDING';

The index on next_attempt_at WHERE status = 'PENDING' makes the delivery processor's "find work to do" query efficient — only pending deliveries are in the index.

Entities and repositories

@Entity
@Table(name = "webhook_endpoints")
public class WebhookEndpoint {
    @Id private UUID id;
    private String tenantId;
    private String url;
    private String secret;  // stored encrypted or hashed

    @Type(PostgreSQLArrayType.class)
    @Column(columnDefinition = "text[]")
    private String[] eventTypes;

    private boolean active;
    private Instant createdAt;
}

@Entity
@Table(name = "webhook_deliveries")
public class WebhookDelivery {
    @Id private UUID id;
    @ManyToOne(fetch = FetchType.LAZY)
    private WebhookEndpoint endpoint;
    private String eventType;

    @JdbcTypeCode(SqlTypes.JSON)
    private Map<String, Object> payload;

    @Enumerated(EnumType.STRING)
    private DeliveryStatus status;  // PENDING, DELIVERED, FAILED, ABANDONED

    private int attempts;
    private Instant nextAttemptAt;
    private Integer lastResponse;
    private String lastError;
    private Instant createdAt;
    private Instant deliveredAt;
}

Creating deliveries when events occur

When an event happens, find all active endpoints that subscribe to that event type and create a delivery record for each:

@Service
@Transactional
public class WebhookService {

    private final WebhookEndpointRepository endpointRepository;
    private final WebhookDeliveryRepository deliveryRepository;

    public void dispatchEvent(String tenantId, String eventType,
            Map<String, Object> payload) {

        List<WebhookEndpoint> endpoints = endpointRepository
            .findActiveByTenantIdAndEventType(tenantId, eventType);

        List<WebhookDelivery> deliveries = endpoints.stream()
            .map(endpoint -> WebhookDelivery.builder()
                .id(UUID.randomUUID())
                .endpoint(endpoint)
                .eventType(eventType)
                .payload(payload)
                .status(DeliveryStatus.PENDING)
                .nextAttemptAt(Instant.now())
                .createdAt(Instant.now())
                .build())
            .toList();

        deliveryRepository.saveAll(deliveries);
    }
}

This creates delivery records in the same transaction as the triggering event. The delivery records persist even if the application restarts before delivery — no events are lost.

Signing outgoing webhooks

Customers need to verify that webhook requests come from your system. HMAC-SHA256 signatures over the payload are the standard:

@Component
public class WebhookSigner {

    public String sign(String payload, String secret) {
        try {
            Mac mac = Mac.getInstance("HmacSHA256");
            mac.init(new SecretKeySpec(secret.getBytes(StandardCharsets.UTF_8), "HmacSHA256"));
            byte[] signature = mac.doFinal(payload.getBytes(StandardCharsets.UTF_8));
            return "sha256=" + HexFormat.of().formatHex(signature);
        } catch (NoSuchAlgorithmException | InvalidKeyException e) {
            throw new IllegalStateException("HMAC-SHA256 unavailable", e);
        }
    }

    public boolean verify(String payload, String secret, String expectedSignature) {
        String actualSignature = sign(payload, secret);
        // Constant-time comparison — prevents timing attacks
        return MessageDigest.isEqual(
            actualSignature.getBytes(StandardCharsets.UTF_8),
            expectedSignature.getBytes(StandardCharsets.UTF_8)
        );
    }
}

MessageDigest.isEqual is a constant-time comparison — it takes the same time regardless of how many characters match. A naive String.equals() comparison is vulnerable to timing attacks: an attacker can measure response time differences to guess the correct signature character by character.

Include the signature in the X-Webhook-Signature header. Include a timestamp to prevent replay attacks:

private WebhookDelivery.RequestDetails buildRequest(WebhookDelivery delivery) {
    String payload = objectMapper.writeValueAsString(delivery.getPayload());
    String timestamp = String.valueOf(Instant.now().getEpochSecond());
    String signedPayload = timestamp + "." + payload;
    String signature = signer.sign(signedPayload, delivery.getEndpoint().getSecret());

    return new RequestDetails(
        payload,
        Map.of(
            "X-Webhook-Signature", signature,
            "X-Webhook-Timestamp", timestamp,
            "X-Webhook-Event", delivery.getEventType(),
            "X-Webhook-Delivery-ID", delivery.getId().toString(),
            "Content-Type", "application/json"
        )
    );
}

Including the timestamp in the signed payload means a captured request can't be replayed — the signature only validates for that specific timestamp. Customers should reject deliveries where the timestamp is more than five minutes old.

The delivery processor

A scheduled job picks up pending deliveries and attempts them:

@Component
public class WebhookDeliveryProcessor {

    private static final Duration[] BACKOFF_SCHEDULE = {
        Duration.ofSeconds(30),
        Duration.ofMinutes(2),
        Duration.ofMinutes(10),
        Duration.ofHours(1),
        Duration.ofHours(6),
        Duration.ofHours(24)
    };

    private final WebhookDeliveryRepository deliveryRepository;
    private final WebClient webClient;
    private final WebhookSigner signer;
    private final ObjectMapper objectMapper;

    @Scheduled(fixedDelay = 5_000)  // run every 5 seconds
    @Transactional
    public void processDeliveries() {
        List<WebhookDelivery> pending = deliveryRepository
            .findPendingDue(Instant.now(), Limit.of(50));

        pending.forEach(this::attemptDelivery);
    }

    private void attemptDelivery(WebhookDelivery delivery) {
        String payload = serializePayload(delivery.getPayload());
        String timestamp = String.valueOf(Instant.now().getEpochSecond());
        String signature = signer.sign(timestamp + "." + payload,
            delivery.getEndpoint().getSecret());

        try {
            ResponseSpec response = webClient.post()
                .uri(delivery.getEndpoint().getUrl())
                .header("X-Webhook-Signature", signature)
                .header("X-Webhook-Timestamp", timestamp)
                .header("X-Webhook-Event", delivery.getEventType())
                .header("X-Webhook-Delivery-ID", delivery.getId().toString())
                .contentType(MediaType.APPLICATION_JSON)
                .bodyValue(payload)
                .retrieve()
                .onStatus(HttpStatusCode::isError, resp ->
                    resp.bodyToMono(String.class)
                        .flatMap(body -> Mono.error(
                            new DeliveryFailedException(resp.statusCode().value(), body))));

            response.toBodilessEntity()
                .timeout(Duration.ofSeconds(30))
                .block();

            // Success
            delivery.setStatus(DeliveryStatus.DELIVERED);
            delivery.setDeliveredAt(Instant.now());
            delivery.setLastResponse(200);

        } catch (DeliveryFailedException e) {
            handleFailure(delivery, e.statusCode(), e.getMessage());
        } catch (Exception e) {
            handleFailure(delivery, null, e.getMessage());
        }

        deliveryRepository.save(delivery);
    }

    private void handleFailure(WebhookDelivery delivery, Integer statusCode, String error) {
        delivery.setAttempts(delivery.getAttempts() + 1);
        delivery.setLastResponse(statusCode);
        delivery.setLastError(error);

        int attemptIndex = Math.min(delivery.getAttempts() - 1, BACKOFF_SCHEDULE.length - 1);

        if (delivery.getAttempts() >= BACKOFF_SCHEDULE.length) {
            // Exhausted all retry attempts
            delivery.setStatus(DeliveryStatus.ABANDONED);
            notifyEndpointOwner(delivery);
        } else {
            delivery.setStatus(DeliveryStatus.PENDING);
            delivery.setNextAttemptAt(Instant.now().plus(BACKOFF_SCHEDULE[attemptIndex]));
        }
    }
}

The backoff schedule: 30s, 2m, 10m, 1h, 6h, 24h — total of 6 attempts before abandoning. This matches Stripe's webhook retry pattern. Adjust the schedule and attempt count based on your SLA requirements.

Limit.of(50) caps how many deliveries the processor handles per cycle — prevents a backlog of millions of deliveries from overwhelming the system in one run.

Verifying incoming webhooks

When your application receives webhooks from external systems (Stripe, GitHub, Twilio), verify their signatures before processing:

@RestController
@RequestMapping("/webhooks")
public class IncomingWebhookController {

    private final WebhookSigner signer;
    private final OrderService orderService;

    @Value("${stripe.webhook-secret}")
    private String stripeWebhookSecret;

    @PostMapping("/stripe")
    public ResponseEntity<Void> handleStripeWebhook(
            @RequestBody String rawBody,  // raw String — not deserialized yet
            @RequestHeader("Stripe-Signature") String signature,
            @RequestHeader("X-Stripe-Timestamp") String timestamp) {

        // Verify signature before processing
        if (!verifyStripeSignature(rawBody, signature, timestamp)) {
            return ResponseEntity.status(HttpStatus.UNAUTHORIZED).build();
        }

        // Check timestamp to prevent replay attacks
        long webhookAge = Instant.now().getEpochSecond() - Long.parseLong(timestamp);
        if (webhookAge > 300) {  // reject webhooks older than 5 minutes
            return ResponseEntity.status(HttpStatus.BAD_REQUEST).build();
        }

        // Safe to process now
        StripeEvent event = objectMapper.readValue(rawBody, StripeEvent.class);
        processStripeEvent(event);

        return ResponseEntity.ok().build();
    }

    private boolean verifyStripeSignature(String payload, String signature, String timestamp) {
        String signedPayload = timestamp + "." + payload;
        // Stripe sends: "t=timestamp,v1=signature"
        String expectedSig = "sha256=" + hmacSha256(signedPayload, stripeWebhookSecret);
        String receivedSig = extractSignature(signature);
        return MessageDigest.isEqual(expectedSig.getBytes(), receivedSig.getBytes());
    }
}

Critical: use @RequestBody String rawBody, not @RequestBody StripeEvent. Signature verification must happen against the raw bytes received — after Jackson deserializes and re-serializes the body, field ordering and whitespace may change, invalidating the signature. Verify first on the raw string, then deserialize.

Idempotent incoming webhook processing

External webhook senders retry on timeout and failures. The same webhook may be delivered multiple times. Process each event exactly once:

@Service
@Transactional
public class StripeEventProcessor {

    private final ProcessedEventRepository processedEventRepository;
    private final OrderService orderService;

    public void process(StripeEvent event) {
        // Check if already processed
        if (processedEventRepository.existsByEventId(event.getId())) {
            log.debug("Duplicate Stripe event {}, skipping", event.getId());
            return;
        }

        // Process based on event type
        switch (event.getType()) {
            case "payment_intent.succeeded" -> handlePaymentSucceeded(event);
            case "payment_intent.payment_failed" -> handlePaymentFailed(event);
            default -> log.debug("Unhandled Stripe event type: {}", event.getType());
        }

        // Record as processed — in the same transaction as the business logic
        processedEventRepository.save(new ProcessedEvent(event.getId(), Instant.now()));
    }
}

The processedEventRepository.save() in the same transaction as the business logic ensures that if the business logic succeeds, the event is recorded as processed atomically. A crash between business logic and recording would cause re-delivery and re-processing — the same transaction prevents this.

The processed_events table needs a unique index on event_id and a cleanup job to remove old entries (events older than 30 days will never be redelivered):

CREATE TABLE processed_events (
    event_id   VARCHAR(255) PRIMARY KEY,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- Cleanup job
DELETE FROM processed_events WHERE processed_at < NOW() - INTERVAL '30 days';

Monitoring and alerting

Webhook delivery health metrics:

@Component
public class WebhookMetrics {

    private final Counter deliverySuccess;
    private final Counter deliveryFailure;
    private final Counter deliveryAbandoned;
    private final Timer deliveryDuration;

    public WebhookMetrics(MeterRegistry registry) {
        this.deliverySuccess = Counter.builder("webhook.delivery")
            .tag("status", "success").register(registry);
        this.deliveryFailure = Counter.builder("webhook.delivery")
            .tag("status", "failure").register(registry);
        this.deliveryAbandoned = Counter.builder("webhook.delivery")
            .tag("status", "abandoned").register(registry);
        this.deliveryDuration = Timer.builder("webhook.delivery.duration")
            .register(registry);
    }
}

Alert on:

• webhook.delivery{status="abandoned"} rate above zero — customer endpoints are consistently failing, customers aren't receiving events
• Pending delivery count growing monotonically — the processor isn't keeping up with event volume
• p99 delivery latency above acceptable threshold

Expose a delivery status API so customers can inspect their webhook history:

@GetMapping("/api/v1/webhook-deliveries")
public Page<DeliveryResponse> getDeliveries(
        @AuthenticationPrincipal Jwt jwt,
        @RequestParam(defaultValue = "0") int page) {
    String tenantId = jwt.getClaimAsString("tenant_id");
    return deliveryRepository.findByTenantId(tenantId, PageRequest.of(page, 25))
        .map(DeliveryResponse::from);
}

@PostMapping("/api/v1/webhook-deliveries/{id}/retry")
public ResponseEntity<Void> retryDelivery(
        @PathVariable UUID id,
        @AuthenticationPrincipal Jwt jwt) {
    String tenantId = jwt.getClaimAsString("tenant_id");
    deliveryService.retryDelivery(id, tenantId);
    return ResponseEntity.accepted().build();
}

A retry endpoint lets customers manually trigger redelivery of abandoned webhooks — essential for debugging and for when their endpoint was down during the retry window.