Shadow-Running New Tax Tables Before Promotion

A withholding table promoted directly from validation into production has never faced a single real pay run, so the first evidence of a miscalibrated bracket edge or a mismapped filing-status column is a disbursement that already left the building and a correction cycle that follows it. This pattern is part of the Threshold Tuning Workflows topic within the Payroll Calculation Engines & Validation Rules framework, and it runs the candidate table against every current pay run in parallel with the live production table — computing but never disbursing the shadow result — so drift surfaces before promotion instead of after a Form 941-X and W-2c correction.

Problem Framing

Shadow-running is deliberately narrower than it sounds. It is live parallel compute on the current pay run, not a replay of history: the candidate table sees the same gross wages, filing statuses, and allowance elections that are moving through production right now, at the same moment production sees them. That is the load-bearing distinction from the sibling pattern in Backtesting Threshold Changes Against Historical Runs, which replays already-settled batches offline. Backtesting proves a candidate table against last quarter’s known population; shadow-running proves it against this run’s actual population — new hires, mid-year filing-status changes under IRS Publication 15-T, and bonus timing that a historical replay cannot reconstruct because it never happened in the archived batch.

A naive implementation breaks in three specific ways:

• The shadow computation shares a code path with disbursement. If the shadow branch calls the same function that writes to the disbursement ledger — even behind a flag — a config error flips the flag and the “shadow” table pays real employees at a candidate rate under IRC § 3402. Shadow compute must be structurally incapable of disbursing, not merely configured not to.
• The comparison mixes rounding states. Diffing an unrounded shadow value against an already-quantized production value manufactures phantom drift from rounding noise alone, or — worse — masks real drift that rounding happens to absorb on one side and not the other.
• Promotion happens before the canary completes. A candidate table that shows zero drift against three canary records is not validated; it has not yet seen the wage brackets, supplemental-income cases, or negative-adjustment records that only appear once a meaningful sample has passed through.

The gate that governs promotion compares per-record drift against a fixed tolerance and only proceeds once canary coverage is statistically sufficient:

\Delta_i = \left| W_{\text{shadow},i} - W_{\text{prod},i} \right|, \qquad \text{PROMOTE} \iff \frac{\#\{i : \Delta_i > \tau\}}{n} \le \beta \ \wedge\ n \ge n_{\min}

where $\tau$ is the per-record drift tolerance in cents, $\beta$ is the maximum acceptable breach rate, and $n_{\min}$ is the minimum canary sample size before a decision is trusted at all. The $n_{\min}$ term matters as much as $\tau$: a candidate table that shows zero breaches against fifty canary records has not yet been tested against the wage bands, supplemental-income cases, and negative retroactive adjustments that only appear once a few hundred pay runs have passed through it, so an early PROMOTE reading is not evidence of correctness — it is evidence of an insufficient sample. Every comparison, breach or not, is written to the append-only comparison store with the same structured key=value logging used elsewhere in the threshold-tuning engine, so a post-promotion audit can reconstruct exactly which employees were in scope, what each table produced, and when the gate changed its decision — the same evidentiary standard a DOL audit evidence package requires for any other payroll calculation.

Prerequisites & Data Requirements

Shadow-running requires a dual-table configuration and a comparison store that exist independently of both tables’ evaluation code, so a shadow run can be toggled, ramped, or killed without a deployment.

Two preconditions sit outside any single function:

1. The shadow branch has no code path to the disbursement ledger. This is a structural guarantee, not a runtime check — the function that computes shadow withholding must not accept a ledger client as a dependency at all.
2. Canary membership is deterministic per employee, not per run. The same employee_id must land in or out of the canary population consistently across every pay run in the observation window, or drift statistics from run to run are not comparable.

Step-by-Step Implementation

Step 1 — Model the shadow configuration and comparison record. Frozen dataclasses keep the candidate table’s identity and tolerance immutable for the duration of a run, and every comparison is a fully typed, appendable record.

import hashlib
import logging
from dataclasses import dataclass
from decimal import Decimal, ROUND_HALF_UP
from typing import Callable, List, Optional

logger = logging.getLogger("payroll.threshold_tuning.shadow")
CENTS = Decimal("0.01")


@dataclass(frozen=True)
class ShadowConfig:
    candidate_table_id: str
    canary_percentage: int   # 0-100
    tolerance: Decimal
    active: bool = True

    def __post_init__(self) -> None:
        if not 0 <= self.canary_percentage <= 100:
            raise ValueError("canary_percentage must be within 0-100")
        if not isinstance(self.tolerance, Decimal):
            raise TypeError("tolerance must be Decimal")


@dataclass(frozen=True)
class ComparisonRecord:
    employee_id: str
    pay_run_id: str
    production_withholding: Decimal
    shadow_withholding: Decimal
    drift: Decimal
    breached: bool

Step 2 — Assign canary membership deterministically. A stable hash of employee_id — never a random draw — guarantees the same employee is in or out of the shadow population on every run. Expected output: calling this twice with the same employee_id and canary_percentage always returns the same boolean.

def in_canary_scope(employee_id: str, canary_percentage: int) -> bool:
    """Deterministically assign an employee to the shadow canary population."""
    digest = hashlib.sha256(employee_id.encode()).hexdigest()
    bucket = int(digest[:8], 16) % 100
    return bucket < canary_percentage


# Same employee, same bucket, every call:
assert in_canary_scope("E-70142", 10) == in_canary_scope("E-70142", 10)

Step 3 — Compute both tables per record, and let only production reach disbursement. The shadow branch takes no ledger dependency, so it is structurally incapable of disbursing; a shadow lookup failure is caught locally and never blocks or mutates the production result. Expected output: production is logged and returned even when the shadow lookup raises.

def shadow_run_record(
    employee_id: str,
    pay_run_id: str,
    gross_wages: Decimal,
    production_table_id: str,
    shadow_cfg: ShadowConfig,
    lookup_fn: Callable[[Decimal, str], Decimal],
) -> Optional[ComparisonRecord]:
    """Disburse production withholding; compute-and-compare shadow withholding."""
    if not isinstance(gross_wages, Decimal):
        raise TypeError("gross_wages must be Decimal")

    production = lookup_fn(gross_wages, production_table_id).quantize(
        CENTS, rounding=ROUND_HALF_UP
    )
    logger.info(
        "shd_disburse emp=%s run=%s table=%s amount=%s",
        employee_id, pay_run_id, production_table_id, production,
    )
    # ^ the only line in this function permitted to reach the disbursement ledger.

    if not shadow_cfg.active or not in_canary_scope(employee_id, shadow_cfg.canary_percentage):
        return None

    try:
        shadow = lookup_fn(gross_wages, shadow_cfg.candidate_table_id).quantize(
            CENTS, rounding=ROUND_HALF_UP
        )
    except Exception:
        logger.error(
            "shd_compute_fail emp=%s run=%s table=%s",
            employee_id, pay_run_id, shadow_cfg.candidate_table_id,
        )
        return None

    drift = abs(shadow - production)
    breached = drift > shadow_cfg.tolerance
    record = ComparisonRecord(
        employee_id=employee_id,
        pay_run_id=pay_run_id,
        production_withholding=production,
        shadow_withholding=shadow,
        drift=drift,
        breached=breached,
    )
    logger.info(
        "shd_compare emp=%s run=%s prod=%s shadow=%s drift=%s breached=%s",
        employee_id, pay_run_id, production, shadow, drift, breached,
    )
    return record

Step 4 — Diff on identically quantized values, never on a mix. Both production and shadow are quantized with the same ROUND_HALF_UP step before the subtraction in Step 3, so the drift reflects a real difference in the tables — the percentage-method lookup itself — rather than rounding-order noise. Expected output for a $1,500.00 gross wage with production returning Decimal("218.00") and the candidate table returning Decimal("223.00"): drift == Decimal("5.00"), breached is True at tolerance = Decimal("0.01").

Step 5 — Aggregate canary comparisons and evaluate the promotion gate. The gate requires a minimum sample before it trusts any breach rate, then decides HOLD, PROMOTE, or ROLLBACK.

@dataclass
class PromotionGate:
    min_canary_samples: int
    max_breach_rate: Decimal   # e.g. Decimal("0.01") == 1%

    def evaluate(self, comparisons: List[ComparisonRecord]) -> str:
        n = len(comparisons)
        if n < self.min_canary_samples:
            logger.warning("shd_gate_hold reason=insufficient_samples n=%s", n)
            return "HOLD"

        breaches = sum(1 for c in comparisons if c.breached)
        breach_rate = (Decimal(breaches) / Decimal(n)).quantize(
            Decimal("0.0001"), rounding=ROUND_HALF_UP
        )
        if breach_rate > self.max_breach_rate:
            logger.warning(
                "shd_gate_rollback breaches=%s n=%s rate=%s", breaches, n, breach_rate
            )
            return "ROLLBACK"

        logger.info("shd_gate_promote n=%s breach_rate=%s", n, breach_rate)
        return "PROMOTE"

Step 6 — Ramp the canary and roll back without a deploy. Promotion and rollback are both configuration changes to ShadowConfig and production_table_id, never code changes. Expected output: after a ROLLBACK decision, production_table_id is untouched and shadow_cfg.active is flipped to False, halting shadow compute instantly.

shadow_cfg = ShadowConfig(
    candidate_table_id="2026-v2", canary_percentage=10, tolerance=Decimal("0.01")
)
comparisons: List[ComparisonRecord] = []

for emp_id, run_id, gross in [
    ("E-1001", "R-9001", Decimal("1500.00")),
    ("E-1002", "R-9001", Decimal("2200.00")),
]:
    rec = shadow_run_record(emp_id, run_id, gross, "2026-v1", shadow_cfg, lookup_stub)
    if rec is not None:
        comparisons.append(rec)

gate = PromotionGate(min_canary_samples=500, max_breach_rate=Decimal("0.01"))
decision = gate.evaluate(comparisons)
# decision == "HOLD" until at least 500 canary comparisons have accumulated;
# ramp canary_percentage upward (10 -> 25 -> 50 -> 100) only after a sustained
# zero-breach PROMOTE window at each step.

Verification

1. Production is unaffected by a shadow failure. Force lookup_fn to raise for shadow_cfg.candidate_table_id only. Assert shadow_run_record still returns None for that record without raising, and that shd_disburse was logged with the correct production amount before the exception occurred.
2. Drift is detected and reported. Configure lookup_fn so the candidate table returns production plus Decimal("5.00") with tolerance = Decimal("0.01"). Assert record.drift == Decimal("5.00") and record.breached is True.
3. Canary scope is stable and proportioned. Call in_canary_scope on the same employee_id twice and assert identical results. Generate 10,000 synthetic ids at canary_percentage=10 and assert the observed fraction in scope falls between 8% and 12% — hash-bucket noise, not a moving target.
4. Rollback restores the prior table without a deploy. Drive PromotionGate.evaluate to ROLLBACK with a synthetic breach rate above max_breach_rate. Assert production_table_id was never referenced by the gate and remains the table that was disbursing throughout; assert flipping shadow_cfg.active to False stops all subsequent shd_compare log lines on the next run.

Failure Modes

• The shadow branch accidentally disburses. Root cause: the shadow computation calls the same function used for production, and a flag or an environment misconfiguration routes its output to the disbursement ledger — employees are paid at a candidate rate that never cleared promotion. Remediation: give the shadow path a function signature with no ledger dependency at all, so there is no code path by which it can write a disbursement; add a test that patches the ledger client to raise if invoked from shadow context.
• Comparing a rounded value against an unrounded one. Root cause: the production amount is quantized to cents at disbursement time, but the shadow amount is diffed before its own quantization step, so the comparison reflects rounding-order noise rather than an actual table difference — sometimes manufacturing false drift, sometimes masking a real one. Remediation: quantize both sides with the identical Decimal rounding mode immediately before the subtraction, as in Step 3, and never diff a raw lookup result against an already-disbursed value.
• Promoting a candidate table without completing a canary. Root cause: the gate is evaluated against a handful of records instead of a statistically sufficient sample, so the table is promoted before it has seen the supplemental-income cases, negative adjustments, or mid-year filing-status changes that only appear at scale. Remediation: enforce min_canary_samples in the gate, and ramp canary_percentage in stages — never jump directly from a small canary to full-population disbursement.

Frequently Asked Questions

Related

• Backtesting Threshold Changes Against Historical Runs — the historical-replay counterpart to this live parallel-compute pattern.
• Threshold Tuning Workflows — the parent topic defining effective-dated configuration and fail-closed routing this pattern builds on.
• Federal Withholding Percentage-Method Lookups — the table format being evaluated in shadow before promotion.
• Validating Federal Tax Bracket Updates — the validation stage a candidate table passes before it is eligible for a shadow run.