Backtesting Threshold Changes Against Historical Runs

A proposed change to a wage-base cap, bonus ceiling, or overtime trigger is only safe to promote once its dollar impact has been measured against real payroll history, record by record, rather than approximated by inspecting a handful of test employees. This pattern is part of the Threshold Tuning Workflows topic within the Payroll Calculation Engines & Validation Rules framework, and it specifies how to replay closed historical pay runs against a candidate rule set, diff the candidate output against what was actually disbursed, and gate promotion on a bounded, explainable delta.

Problem Framing

A naive backtest reruns a handful of familiar employees through the new rule, sees no obvious blow-up, and ships. That approach fails for exactly the population it should protect: employees clustered near the old boundary, whose classification flips under the new one. Backtesting exists to replace “looks fine” with a quantified, per-record impact report before a single future pay period is affected.

Backtesting is deliberately distinct from shadow-running, its sibling pattern in this topic. Shadow-running executes the candidate rule in parallel with the live production system going forward, so it only surfaces a discrepancy once a new pay period actually runs — useful for catching integration drift, but slow to build confidence. Backtesting instead replays pay runs that already closed, pulled from the append-only ledger described in Building a Checksum Chain for Payroll Events, so the full impact across months of history is available before the candidate rule ever touches a live disbursement.

Deterministic replay is the load-bearing property: the same historical inputs, run against the same candidate rule identified by the same rule_hash, must produce the same output on every execution. If replay is not reproducible, a diff computed once cannot be trusted to still hold when the promotion decision is actually made. Per-record impact is:

$\Delta_i = f_{\text{candidate}}(x_i, d_i) - y_i^{\text{baseline}}$

where $x_i$ is the record’s input snapshot, $d_i$ its effective_date, and $y_i^{\text{baseline}}$ the output the historical rule actually produced. The no-regression gate promotes only when both the worst single-record impact and the share of the population it touches stay bounded:

\text{promote} \iff \max_i \lvert \Delta_i \rvert \le \tau \ \wedge\ \frac{\left|\{\, i : \lvert \Delta_i \rvert > \tau \,\}\right|}{n} \le 0.02

Prerequisites & Data Requirements

Backtesting only works if history was captured losslessly at run time. Reconstructing a baseline from “what the rule looks like today” is not backtesting — it is comparing two unknowns. Every field below must exist on the historical record before replay starts.

Two preconditions sit outside the schema:

1. History is retained in an append-only ledger, not a mutable table that a later correction could silently overwrite. The checksum-chained event log is the canonical source, and its retention window must cover at minimum the 90-to-365-day backtest range under review — commonly kept far longer to satisfy the four-year retention under 26 CFR § 31.6001-1.
2. The tolerance band and population cap are fixed before the replay runs, not chosen after seeing the results. A tolerance picked to make an inconvenient diff pass defeats the purpose of the gate.

Step-by-Step Implementation

Step 1 — Model the historical record and diff types. Every monetary field is Decimal; inputs is rejected if any value is a float.

from dataclasses import dataclass
from datetime import date
from decimal import Decimal
from typing import Callable, Dict, List
import logging

logger = logging.getLogger("payroll.threshold_tuning.backtest")

CENTS = Decimal("0.01")
DEFAULT_TOLERANCE = Decimal("0.00")
MAX_AFFECTED_RATIO = Decimal("0.02")


@dataclass(frozen=True)
class HistoricalRecord:
    employee_id: str
    pay_period_id: str
    effective_date: date
    inputs: Dict[str, Decimal]
    baseline_output: Decimal
    baseline_rule_hash: str

    def __post_init__(self) -> None:
        if not isinstance(self.baseline_output, Decimal):
            raise TypeError("baseline_output must be Decimal")
        if any(not isinstance(v, Decimal) for v in self.inputs.values()):
            raise TypeError("all input values must be Decimal")


@dataclass(frozen=True)
class ReplayDiff:
    employee_id: str
    pay_period_id: str
    baseline_output: Decimal
    candidate_output: Decimal
    delta: Decimal

Step 2 — Define the candidate rule against the record’s own effective_date. The rule function is date-aware by signature so a replay cannot accidentally resolve today’s thresholds against yesterday’s history — the failure mode covered below.

def candidate_rule_fn(
    inputs: Dict[str, Decimal], effective_date: date,
) -> Decimal:
    """Compute the candidate output for one historical record.

    `effective_date` selects the threshold version that would have applied
    on that day under the candidate rule set — never datetime.now().
    """
    wage_base_cap = (
        Decimal("176100.00") if effective_date.year >= 2026 else Decimal("168600.00")
    )
    taxable = min(inputs["ytd_taxable_wages"], wage_base_cap)
    return (taxable * Decimal("0.062")).quantize(CENTS)

Step 3 — Replay each record and diff against the stored baseline in Decimal. Expected output: for a record with ytd_taxable_wages = Decimal("180000.00") at effective_date=date(2026, 3, 1) and baseline_output = Decimal("10453.20") computed under the old $168,600 cap, the candidate output is Decimal("176100.00") * Decimal("0.062") = Decimal("10918.20"), a delta of Decimal("465.00").

def replay_against_candidate(
    records: List[HistoricalRecord],
    rule_fn: Callable[[Dict[str, Decimal], date], Decimal],
    candidate_rule_hash: str,
) -> List[ReplayDiff]:
    diffs = []
    for rec in records:
        candidate_output = rule_fn(rec.inputs, rec.effective_date)
        if not isinstance(candidate_output, Decimal):
            raise TypeError("candidate rule must return Decimal")
        delta = (candidate_output - rec.baseline_output).quantize(CENTS)
        diffs.append(ReplayDiff(
            employee_id=rec.employee_id,
            pay_period_id=rec.pay_period_id,
            baseline_output=rec.baseline_output,
            candidate_output=candidate_output,
            delta=delta,
        ))
        logger.info(
            "bkt_diff emp=%s pay_period=%s rule_hash=%s baseline=%s candidate=%s delta=%s",
            rec.employee_id, rec.pay_period_id, candidate_rule_hash,
            rec.baseline_output, candidate_output, delta,
        )
    return diffs

Step 4 — Aggregate the impact across the full replay set. Expected output for two records with deltas Decimal("465.00") and Decimal("0.00"): changed_records == 1, net_delta == Decimal("465.00"), max_abs_delta == Decimal("465.00").

@dataclass(frozen=True)
class ImpactReport:
    total_records: int
    changed_records: int
    affected_employee_ids: List[str]
    net_delta: Decimal
    max_abs_delta: Decimal
    promote: bool


def aggregate_impact(diffs: List[ReplayDiff], tolerance: Decimal) -> ImpactReport:
    net_delta = Decimal("0")
    max_abs_delta = Decimal("0")
    affected: List[str] = []
    changed = 0
    for d in diffs:
        net_delta += d.delta
        if d.delta != Decimal("0"):
            changed += 1
        abs_delta = abs(d.delta)
        max_abs_delta = max(max_abs_delta, abs_delta)
        if abs_delta > tolerance:
            affected.append(d.employee_id)
    return ImpactReport(
        total_records=len(diffs),
        changed_records=changed,
        affected_employee_ids=affected,
        net_delta=net_delta.quantize(CENTS),
        max_abs_delta=max_abs_delta,
        promote=False,
    )

Step 5 — Gate promotion on tolerance and affected population. Expected output: with tolerance = Decimal("0.00") and any nonzero delta present, affected_ratio exceeds 0, so a single seeded change with a full population of 1 record yields promote=False — the gate is intentionally strict until the reviewer widens tolerance deliberately.

def gate_on_tolerance(report: ImpactReport, tolerance: Decimal) -> ImpactReport:
    affected_ratio = (
        Decimal(len(report.affected_employee_ids)) / Decimal(report.total_records)
        if report.total_records else Decimal("0")
    )
    promote = report.max_abs_delta <= tolerance and affected_ratio <= MAX_AFFECTED_RATIO
    logger.info(
        "bkt_gate total=%s changed=%s affected=%s max_abs_delta=%s promote=%s",
        report.total_records, report.changed_records,
        len(report.affected_employee_ids), report.max_abs_delta, promote,
    )
    return ImpactReport(
        total_records=report.total_records,
        changed_records=report.changed_records,
        affected_employee_ids=report.affected_employee_ids,
        net_delta=report.net_delta,
        max_abs_delta=report.max_abs_delta,
        promote=promote,
    )

Verification

Confirm the replay engine and the gate with cases that isolate determinism, not just arithmetic:

1. Identical rule, zero diff. Replay baseline_rule_hash as its own candidate — feed the historical rule function back through replay_against_candidate — and assert every delta == Decimal("0"). Any nonzero delta here means the reconstructed rule does not match what actually ran, and the ledger’s baseline_rule_hash should be treated as untrusted.
2. Seeded change, expected delta. Raise the wage-base cap by a known amount for a single synthetic record and hand-compute the expected delta; assert aggregate_impact reproduces that exact Decimal to the cent. This catches sign errors and rounding-order mistakes that a large batch can hide.
3. Replay determinism. Run replay_against_candidate on the same records list three times and assert byte-identical ReplayDiff sequences, including order. Nondeterminism here — from unordered dict iteration or a wall-clock read inside rule_fn — invalidates every downstream diff.
4. Gate strictness at the boundary. Feed max_abs_delta == tolerance exactly and assert promote is True (the gate uses <=); feed one cent over and assert promote is False.

Failure Modes

• Replaying against today’s rule instead of the historical one. Root cause: the candidate function (or a dependency it calls, such as a tax-bracket lookup) reads the live configuration keyed by datetime.now() rather than the record’s own effective_date, so every historical record is silently evaluated as if it happened today. Remediation: thread effective_date explicitly through every rule and sub-lookup the candidate touches, and add a unit test that asserts the candidate’s resolved threshold for a two-year-old effective_date differs from today’s when the two are known to differ.
• Nondeterministic inputs or rule evaluation. Root cause: the candidate rule iterates a set or unordered dict whose enumeration order varies between runs, or depends on a random tie-breaker for records with identical timestamps, so two replays of the same history produce different ReplayDiff sequences. Remediation: sort every input collection by an explicit key before iterating, and require the candidate rule to be a pure function of (inputs, effective_date) with no hidden state; the determinism test in Verification exists specifically to catch this before promotion.
• Ignoring the affected edge population. Root cause: the aggregate net_delta nets out near zero because roughly equal numbers of employees gain and lose under the candidate rule, masking large offsetting individual swings concentrated at the old boundary. Remediation: gate on max_abs_delta and affected_ratio in addition to net_delta, and always emit affected_employee_ids in the impact report so a reviewer can inspect the actual population rather than trusting a single summed number.

Frequently Asked Questions

Related

• Shadow-Running New Tax Tables Before Promotion — the live-parallel counterpart that runs after backtesting has bounded the historical impact.
• Threshold Tuning Workflows — the parent topic defining the versioned, effective-dated configuration this replay tests.
• Validating Federal Tax Bracket Updates — the annual bracket-edge change that this replay pattern is commonly run against.
• Weighted-Average Regular Rate with Nondiscretionary Bonuses — a rate calculation whose historical inputs are frequently the subject of a backtest.