Syncing payroll APIs with rate limiting

A payroll sync that treats a 429 as an error instead of a contract will either drop a page of pay records during the cutoff window or hammer the provider into an IP block — both of which surface as a short payroll run that reconciles against nothing. This guide is the throttling specialization of the REST API Payroll Sync pattern: it keeps that pattern’s Decimal precision, idempotent ingestion, and audit-trail guarantees, but adds the header parsing, backoff math, request accounting, and circuit-breaking a rate-limited endpoint demands so that exactly-once delivery holds under load.

Problem Framing

Payroll providers enforce asymmetric, endpoint-specific rate limits that rarely align with naive HTTP assumptions. A pay-run extract endpoint might allow 120 requests per minute while the same tenant’s adjustments endpoint allows 20; a burst limit overlays a sustained limit; and the published SLA is often the ceiling, not the threshold at which the provider starts shedding load. A client that paginates at full speed until it sees a 429 is already too late — the truncated page is gone, and the cursor that would have fetched it advanced.

Three behaviours specific to a throttled payroll endpoint break naive implementations:

• Silent page truncation. Under burst throttling some providers return 200 with a short or empty data array rather than a 429. A loop that trusts page size as an end-of-data signal stops early and posts a partial pay run. The cursor, not the page count, is the only authoritative end marker.
• Retry storms. Exponential backoff without jitter makes every concurrent worker retry on the same schedule, so a fleet recovering from one 429 synchronizes into a thundering herd that re-trips the limit. Backoff must be randomized, capped, and bounded by a retry budget.
• The clock is a compliance boundary. Polling cadence that exceeds the provider’s rate window stretches a sync past the jurisdictional cutoff. California overtime under FLSA Threshold Mapping and 8 CFR record timing both assume the run completed inside the pay-period close; a backoff that drifts the finish past midnight UTC turns a network event into a wage-and-hour exposure.

Rate limiting here is not a transport detail; it is the control surface that decides which API responses are allowed to reach the calculator, and a wrong decision costs a paycheck.

Prerequisites & Data Requirements

Before applying this pattern, the engineer must have:

• The provider’s documented limits, per endpoint. Sustained and burst request counts, the window length, and the exact header names. Most payroll APIs emit X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset (an epoch second or a delta), plus a Retry-After on 429. Never assume the standard X- spelling — GitHub-style, Stripe-style, and RFC 9239 RateLimit headers all differ.
• A monotonic clock for backoff and a UTC wall clock for cutoffs. Sleep intervals are computed against time.monotonic() so a system clock adjustment cannot make a backoff negative; cutoff checks use datetime.now(timezone.utc) against pay_period_end.
• A durable cursor checkpoint. A small record holding {source_run_id, last_cursor, requests_this_window, window_reset_epoch} so a sync paused by a long backoff or a tripped breaker resumes mid-pagination instead of restarting.
• The canonical record contract. The same PayrollRecord schema enforced by Data Boundary Definitions — employee_id, ISO-8601 period dates, and gross_pay/tax_withheld/net_pay parsed as Decimal from their string form, never float. Throttling changes when a record arrives, never what shape it must take.
• A fallback sink. A dead-letter queue reachable through Fallback Routing Strategies for records that survive the channel but fail validation, plus a degraded path to CSV Ingestion Pipelines when the endpoint stays throttled past the retry budget.

Step-by-Step Implementation

Step 1 — Parse and cache the rate-limit headers

Read the quota state from every response and hold it between requests. Apply a safety buffer so the client pauses before the provider starts shedding load rather than after. For a published remaining count $r$ and a buffer fraction $b$, the usable budget is:

$r_{\text{eff}} = \left\lfloor r \cdot (1 - b) \right\rfloor$

A 15% buffer ($b = 0.15$) absorbs provider-side burst throttling and clock skew between your accounting and theirs.

import time
from dataclasses import dataclass

@dataclass
class RateLimitState:
    limit: int = 120
    remaining: int = 120
    reset_epoch: float = 0.0          # provider window reset, UTC epoch seconds
    safety_buffer: float = 0.15        # pause at 85% of published remaining

    @property
    def effective_remaining(self) -> int:
        return int(self.remaining * (1.0 - self.safety_buffer))

    def update_from_headers(self, headers) -> None:
        def _int(name, default):
            try:
                return int(headers.get(name, default))
            except (TypeError, ValueError):
                return default
        self.limit = _int("X-RateLimit-Limit", self.limit)
        self.remaining = _int("X-RateLimit-Remaining", self.remaining)
        reset = headers.get("X-RateLimit-Reset")
        if reset is not None:
            try:
                # Treat values <= now as a delta, otherwise an absolute epoch.
                val = float(reset)
                self.reset_epoch = val if val > time.time() else time.time() + val
            except ValueError:
                pass

Expected output: after a response carrying X-RateLimit-Remaining: 100, state.effective_remaining is 85, and reset_epoch is a future UTC epoch second regardless of whether the provider sent an absolute timestamp or a delta.

Step 2 — Compute a deterministic backoff with Retry-After priority

The provider’s Retry-After is authoritative; honour it over any locally computed curve. Only when it is absent do you fall back to exponential backoff with jitter, capped, and clamped to the reset window. For attempt $n$, base $\beta$, and cap $c$:

$\text{delay} = \min\!\left(\beta^{\,n} + U(0, j),\; c\right)$

where $U(0,j)$ is uniform jitter that desynchronizes concurrent workers.

import random

def compute_backoff(state: RateLimitState, attempt: int,
                    retry_after: str | None,
                    base: float = 2.0, cap: float = 30.0,
                    jitter: float = 0.5) -> float:
    # Priority 1: the provider told us exactly how long to wait.
    if retry_after is not None:
        try:
            return max(0.0, float(retry_after))
        except ValueError:
            pass  # HTTP-date form is rare for payroll APIs; fall through
    # Priority 2: exponential backoff + jitter, capped.
    delay = min(base ** attempt + random.uniform(0.0, jitter), cap)
    # Priority 3: never sleep past the window reset; resume as soon as it clears.
    if state.reset_epoch > 0:
        window_left = state.reset_epoch - time.time()
        if window_left > 0:
            delay = min(delay, window_left + random.uniform(0.0, jitter))
    return delay

Expected output: a 429 carrying Retry-After: 12 yields 12.0 exactly; with no header, attempt 3 yields a value in [8.0, 8.5], and a value capped at 30.0 no matter how high attempt climbs.

Step 3 — Account for the budget with a sliding window

Header state can lag by one response, so keep a local sliding-window counter as a second guard. It throttles the client even before the first header arrives and covers providers that omit headers entirely.

from collections import deque

class SlidingWindow:
    """Local request accounting: at most `limit` requests per `window` seconds."""
    def __init__(self, limit: int, window: float = 60.0):
        self.limit = limit
        self.window = window
        self._stamps: deque[float] = deque()

    def time_until_slot(self) -> float:
        now = time.monotonic()
        while self._stamps and now - self._stamps[0] >= self.window:
            self._stamps.popleft()
        if len(self._stamps) < self.limit:
            return 0.0
        return self.window - (now - self._stamps[0])

    def record(self) -> None:
        self._stamps.append(time.monotonic())

Expected output: after limit calls to record() inside one window, time_until_slot() returns a positive number of seconds until the oldest stamp ages out; once it does, it returns 0.0 and a new request is allowed.

Step 4 — Trip a circuit breaker on sustained failure

A retry budget protects a single page; a circuit breaker protects the provider from a fleet. After a threshold of consecutive failures the breaker opens, all workers stop calling for a cool-down, then a single HALF_OPEN probe decides whether to close.

class CircuitBreaker:
    def __init__(self, threshold: int = 5, cooldown: float = 120.0):
        self.threshold = threshold
        self.cooldown = cooldown
        self._failures = 0
        self._opened_at = 0.0

    @property
    def state(self) -> str:
        if self._failures < self.threshold:
            return "closed"
        if time.monotonic() - self._opened_at >= self.cooldown:
            return "half_open"
        return "open"

    def record_success(self) -> None:
        self._failures = 0

    def record_failure(self) -> None:
        self._failures += 1
        if self._failures == self.threshold:
            self._opened_at = time.monotonic()

Expected output: five consecutive record_failure() calls move state to open; after cooldown seconds it reports half_open, and a single record_success() resets it to closed.

Step 5 — Wire the throttled, resumable fetch loop

The loop checks the breaker, waits for a local slot, respects the header budget, retries 429/5xx within a budget, and advances the durable cursor only after a page commits — so a long backoff or a tripped breaker resumes mid-pagination.

import logging

logger = logging.getLogger("payroll.rate_limit")

def fetch_all(client, window: SlidingWindow, breaker: CircuitBreaker,
              state: RateLimitState, checkpoint, max_retries: int = 5):
    cursor = checkpoint.load_cursor()
    while True:
        if breaker.state == "open":
            logger.warning("event=circuit_open cool_down_s=%.1f", breaker.cooldown)
            time.sleep(breaker.cooldown)
            continue
        wait = window.time_until_slot()
        if wait > 0 or state.effective_remaining <= 0:
            sleep_for = max(wait, compute_backoff(state, 0, None))
            logger.info("event=throttle_pause wait_s=%.2f remaining=%s",
                        sleep_for, state.effective_remaining)
            time.sleep(sleep_for)
            continue

        for attempt in range(max_retries):
            window.record()
            resp = client.get_page(cursor)
            state.update_from_headers(resp.headers)
            if resp.status_code in (429, 500, 502, 503):
                delay = compute_backoff(state, attempt,
                                        resp.headers.get("Retry-After"))
                breaker.record_failure()
                logger.warning("event=backoff status=%s attempt=%s wait_s=%.2f",
                               resp.status_code, attempt + 1, delay)
                time.sleep(delay)
                continue
            resp.raise_for_status()
            breaker.record_success()
            page = resp.json()
            yield from page.get("data", [])
            cursor = page.get("next_cursor")
            checkpoint.save_cursor(cursor)        # advance only after commit
            logger.info("event=page_committed next_cursor=%s remaining=%s",
                        (cursor or "")[:12], state.remaining)
            break
        else:
            raise RuntimeError("event=retry_budget_exhausted cursor=%s" % cursor)

        if not cursor:                            # cursor, not page size, ends the run
            logger.info("event=sync_complete")
            break

Expected output: against a provider that returns one 429 then a 200, the page is retried (not dropped), the cursor advances exactly once, and the run terminates only when next_cursor is null — never on a short page.

Verification

Confirm correctness with boundary tests aimed at the throttling scenario specifically:

1. Retry-After priority. Mock a 429 with Retry-After: 7 and assert the client sleeps 7.0 s, ignoring the exponential curve. Mock a 429 with no header and assert the delay matches $\min(\beta^n + \text{jitter}, c)$ for the attempt number.
2. Jitter desynchronization. Compute backoff for ten simulated workers at the same attempt and assert the delays are distinct — no two workers retry within the same millisecond, so the herd does not re-form.
3. Short-page is not end-of-data. Return a 200 with an empty data array but a non-null next_cursor and assert the loop keeps paging. Only a null cursor may terminate the run.
4. Sliding-window bound. Drive limit + 5 requests into one window and assert exactly limit are admitted before time_until_slot() returns a positive wait; assert the count never exceeds limit within any rolling window seconds.
5. Circuit-breaker transitions. Feed threshold consecutive failures and assert state == "open", that no request is issued during cool-down, and that one success after half_open closes it.
6. Resume mid-pagination. Kill the loop after page 3, then restart; assert it resumes from the saved cursor, re-fetches nothing already committed, and that the deterministic idempotency key absorbs any overlap so no employee is double-posted.
7. Cutoff guard. Set pay_period_end close and a backoff that would push completion past it; assert the run logs a cutoff-risk alert before sleeping, so the FLSA Threshold Mapping timing is never silently breached.

Failure Modes

• Retry-After ignored in favour of local backoff. Symptom: the provider returns 429 for far longer than expected and eventually blocks the tenant. Root cause: the client computes its own (shorter) exponential delay and retries before the provider’s window clears. Remediation: always honour Retry-After first, as Step 2 does; the local curve is only a fallback for responses that omit it.
• Cursor advanced before the page committed. Symptom: a backoff or breaker trip mid-page loses records, and the resumed run skips them — silently underpaying a batch. Root cause: saving next_cursor before the page’s records are durably handed downstream. Remediation: persist the cursor strictly after the page commits, so a crash between fetch and commit only ever re-fetches a page, never skips one.
• Synchronized retries re-trip the limit. Symptom: a fleet recovers from one 429, then all workers fail again at the same instant. Root cause: fixed exponential backoff with no jitter, so every worker’s schedule is identical. Remediation: add bounded uniform jitter to every delay and cap the curve, so recovery spreads across the window instead of stacking on one tick.

Related

• REST API Payroll Sync — the parent pattern: OAuth token lifecycle, cursor pagination, schema enforcement, and idempotency this guide throttles on top of.
• Async Batch Processing for Large Payroll Files — back-pressure and resumable checkpoints for overlapping paginated I/O with database writes at volume.
• CSV Ingestion Pipelines — the degraded flat-file path when an endpoint stays throttled past the retry budget.
• Fallback Routing Strategies — the tier hierarchy that catches records the channel delivers but validation rejects.

Frequently Asked Questions