Handling Out-of-Order and Duplicate Payroll Webhooks

A webhook consumer that applies updates in the order they arrive over the wire, rather than the order the source system assigned to them, will eventually let a stale payload overwrite a correct one and no exception will ever fire. This pattern belongs to the REST API Payroll Sync topic within the Multi-Format Payroll Data Ingestion & Normalization framework, and it isolates the receipt discipline a payroll webhook endpoint needs: verify the sender before trusting the payload, treat every delivery as possibly duplicated, and let the entity’s own version number — never arrival time — decide whether an update is newer.

Problem Framing

Most HCM and payroll vendors deliver webhooks under an at-least-once contract: a delivery can be retried after a timeout even though the consumer already received and processed it, and network paths can reorder concurrent deliveries for the same entity. A consumer that is not built for this receives three real failure shapes:

• Arrival order is not entity-version order. TCP guarantees byte order within one connection, not delivery order across retried connections, worker pools, or queue partitions. An update carrying version=1 can physically arrive after one carrying version=3 if the first attempt was slow or timed out and retried. A handler that applies “whatever showed up most recently” will happily let the late version=1 payload stomp the already-applied version=3 state.
• Redelivery looks identical to a new event. At-least-once delivery means the same logical event can be POSTed twice — the vendor’s retry policy does not know the first attempt actually succeeded. Without a dedup key, a duplicate delivery re-runs the same mutation, which is harmless for an idempotent write but re-triggers side effects (a second disbursement notice, a duplicated deduction adjustment) for anything that is not.
• An unverified payload is not a payroll fact. A webhook endpoint is a public HTTP surface. Without signature verification, anything that can reach the URL and guess the shape of the payload can inject a fabricated pay-rate or deduction change, and the endpoint has no way to tell it apart from the vendor.

The fix is to stop trusting the network’s delivery order and instead trust two things the sender can prove: a signature over the payload, and a monotonically increasing version per entity. An update is only ever accepted when its version is strictly greater than the version already stored for that entity_id — last-writer-by-version, not last-writer-by-arrival. Everything else is either a no-op (a duplicate) or a stale drop (an update superseded by one already applied).

Prerequisites & Data Requirements

The endpoint needs five fields on every inbound envelope before it can make a receipt decision. payload carries the mutation itself and is opaque to the guard logic below; the five listed fields are what the guard actually reads.

Two preconditions sit outside the field list. First, signature verification must run against the raw bytes of the request body, before any JSON deserialization — verifying a re-serialized copy can pass even when the original bytes were tampered with, because re-encoding can normalize away the difference. Second, any monetary field in the payload (net_pay, gross_pay, a deduction delta) is parsed through Decimal(str(value)); a float in the payload is a boundary violation and the record is quarantined the same way Enforcing Decimal Precision Across Payroll Fields requires everywhere else in the pipeline.

Step-by-Step Implementation

The routine below verifies the signature first, deduplicates by event_id inside a bounded window, applies an update only when its version exceeds the entity’s stored version, and persists the result as a versioned upsert. Every branch logs a structured key=value line.

Step 1 — Model the envelope and the durable stores. The store below stands in for a keyed table (Postgres row, DynamoDB item) that already carries a version column; the seen-event ledger stands in for a TTL-indexed table or a Redis key with an expiry equal to the replay window.

"""Idempotent, version-guarded ingestion of at-least-once payroll webhooks."""
import hashlib
import hmac
import logging
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from decimal import Decimal, InvalidOperation
from typing import Any

logger = logging.getLogger("payroll.webhooks.hcm")

REPLAY_WINDOW = timedelta(hours=24)


@dataclass(frozen=True)
class WebhookEnvelope:
    event_id: str
    entity_id: str
    version: int
    signature: str
    ts: datetime
    payload: dict[str, Any]


class SignatureError(Exception):
    """Raised when the HMAC signature does not match the raw body."""


class EntityStore:
    """Stand-in for a durable keyed store with a version column."""

    def __init__(self) -> None:
        self._versions: dict[str, int] = {}
        self._data: dict[str, dict[str, Any]] = {}

    def get_version(self, entity_id: str) -> int:
        return self._versions.get(entity_id, 0)

    def upsert(self, entity_id: str, version: int, data: dict[str, Any]) -> None:
        self._versions[entity_id] = version
        self._data[entity_id] = data

Step 2 — Verify the HMAC signature before parsing anything. Expected output: a tampered or misrouted delivery raises SignatureError and never reaches the dedup ledger or the store.

def verify_signature(raw_body: bytes, signature: str, secret: bytes) -> None:
    """Verify an HMAC-SHA256 signature over the raw request body."""
    expected = hmac.new(secret, raw_body, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected, signature):
        raise SignatureError("hmac_mismatch")

Step 3 — Deduplicate by event_id within a bounded window. A durable ledger with a TTL index is the production equivalent; the window bounds both memory growth and how far back a legitimate late redelivery can still be recognized as a duplicate. Expected output: replaying the same event_id inside the window returns True on the second call.

class SeenEvents:
    """Bounded event-id ledger; production uses a TTL-indexed durable table."""

    def __init__(self, window: timedelta = REPLAY_WINDOW) -> None:
        self._window = window
        self._seen: dict[str, datetime] = {}

    def _evict(self, now: datetime) -> None:
        cutoff = now - self._window
        for event_id in [e for e, t in self._seen.items() if t < cutoff]:
            del self._seen[event_id]

    def is_duplicate(self, event_id: str, now: datetime) -> bool:
        self._evict(now)
        if event_id in self._seen:
            return True
        self._seen[event_id] = now
        return False

Step 4 — Apply only if the incoming version exceeds the stored version. Money is cast through Decimal(str(value)); a float is rejected outright rather than coerced. Expected output for a payload with "net_pay": 0.1 as a JSON float: ValueError: float_money_rejected value=0.1.

def _extract_net_pay(payload: dict[str, Any]) -> Decimal:
    raw = payload.get("net_pay")
    if isinstance(raw, float):
        raise ValueError(f"float_money_rejected value={raw!r}")
    try:
        return Decimal(str(raw))
    except InvalidOperation as exc:
        raise ValueError(f"non_decimal_money value={raw!r}") from exc


def apply_webhook(
    envelope: WebhookEnvelope,
    raw_body: bytes,
    secret: bytes,
    seen: SeenEvents,
    store: EntityStore,
) -> str:
    """Verify, dedup, and version-guard one webhook delivery.

    Returns "applied", "duplicate", or "stale". Raises SignatureError or
    ValueError for payloads that must be quarantined, never applied.
    """
    verify_signature(raw_body, envelope.signature, secret)

    if seen.is_duplicate(envelope.event_id, envelope.ts):
        logger.info(
            "wh_duplicate entity=%s event=%s version=%s",
            envelope.entity_id, envelope.event_id, envelope.version,
        )
        return "duplicate"

    stored_version = store.get_version(envelope.entity_id)
    if envelope.version <= stored_version:
        logger.info(
            "wh_stale_drop entity=%s event=%s incoming=%s stored=%s",
            envelope.entity_id, envelope.event_id, envelope.version, stored_version,
        )
        return "stale"

    net_pay = _extract_net_pay(envelope.payload)
    store.upsert(
        envelope.entity_id,
        envelope.version,
        {**envelope.payload, "net_pay": net_pay},
    )
    logger.info(
        "wh_applied entity=%s event=%s version=%s net_pay=%s",
        envelope.entity_id, envelope.event_id, envelope.version, net_pay,
    )
    return "applied"

Step 5 — Feed the three deliveries out of order. This reproduces the diagram above: version=3 arrives first, then the two older versions arrive later and are dropped. Expected output: results == ["applied", "stale", "stale"] and store.get_version("emp-4471") == 3.

secret = b"shared-hmac-secret"
seen = SeenEvents()
store = EntityStore()

deliveries = [
    ("e-101", 3, {"net_pay": "2150.00"}),  # arrives first, highest version
    ("e-099", 1, {"net_pay": "2000.00"}),  # arrives second, stale
    ("e-100", 2, {"net_pay": "2075.00"}),  # arrives third, stale
]

results = []
for event_id, version, payload in deliveries:
    body = b"body"  # illustrative; production signs the exact serialized bytes
    envelope = WebhookEnvelope(
        event_id=event_id,
        entity_id="emp-4471",
        version=version,
        signature=hmac.new(secret, body, hashlib.sha256).hexdigest(),
        ts=datetime.now(timezone.utc),
        payload=payload,
    )
    results.append(apply_webhook(envelope, body, secret, seen, store))

# results == ["applied", "stale", "stale"]
# store.get_version("emp-4471") == 3

Verification

Confirm each receipt path independently rather than trusting one happy-path run:

1. Duplicate is a no-op. Feed the same event_id twice with the same version inside the replay window and assert the second call returns "stale" — wait, it should never even reach the version compare. Assert it returns "duplicate" and that store.get_version is unchanged after the second call.
2. Out-of-order stale drop. Apply version=3, then apply version=1 for the same entity_id. Assert the second call returns "stale" and store.get_version(entity_id) == 3, not 1.
3. Newer version still applies. After version=3 is stored, apply version=4 and assert it returns "applied" and store.get_version(entity_id) == 4. The guard only blocks non-increasing versions, never legitimate progress.
4. Bad signature is rejected before dedup or version logic runs. Call verify_signature with a signature computed over a different body and assert SignatureError is raised; assert seen.is_duplicate was never called for that delivery, since the signature check must short-circuit first.
5. Float money is quarantined, not coerced. Call _extract_net_pay({"net_pay": 0.1}) and assert ValueError; call it with {"net_pay": "0.10"} and assert Decimal("0.10") is returned exactly.

Failure Modes

• Ordering by arrival time instead of by version. Root cause: the handler compares ts or simply applies whatever payload arrived most recently, so a slow retry carrying an old version overwrites a newer state that already posted. Remediation: gate every write on incoming.version > stored.version for that entity_id; treat ts as metadata for the replay window only, never as the ordering key.
• Missing dedup lets a retried delivery double-apply. Root cause: the endpoint has no event_id ledger, so a vendor retry after a slow response (or a client-side timeout that masked a successful 200) re-runs the mutation and, for anything with a side effect beyond the upsert itself, fires that side effect twice. Remediation: check event_id against a bounded, durable ledger before any state change, and return the original success response on a detected duplicate so the sender’s retry terminates cleanly.
• No signature check accepts a forged payload. Root cause: the endpoint trusts any POST that matches the expected JSON shape, so anything that can reach the URL — a misconfigured internal service, a replayed capture, a malicious actor — can inject a fabricated pay-rate change indistinguishable from a real one. Remediation: verify an HMAC-SHA256 signature over the raw body with hmac.compare_digest before parsing, per the construction in RFC 2104 and the keyed-hash guidance in NIST SP 800-107; reject on mismatch and never fall back to processing the payload “just this once.”

Frequently Asked Questions

Related

• Syncing Payroll APIs with Rate Limiting — backoff and throttling for the polling side of the same REST integration.
• REST API Payroll Sync — the parent topic covering token lifecycle, pagination, and canonical schema enforcement this webhook path shares.
• Retry Semantics, Dead-Letter Queues & Idempotency Keys — the broader idempotency-key pattern this page’s event_id ledger specializes.
• Async Batch Processing — bounded-concurrency ingestion for high-volume payroll feeds beyond single webhook deliveries.