Migrating a Payroll Feed from CSV to EDI 834: A Dual-Run Cutover Playbook

Switching a benefits-enrollment feed from a flat-file CSV drop to a carrier’s ASC X12 834 transmission is not a parser swap — it is a controlled migration with its own acceptance criteria, and skipping the parity window is how an open-enrollment cycle ships silently wrong elections to payroll. This scenario is the operational sequel to the CSV vs EDI 834 vs REST API Ingestion Tradeoffs topic within the Multi-Format Payroll Data Ingestion & Normalization framework, and it assumes the decision to move off CSV has already been made — the carrier now supports X12 and the CSV export was always a workaround for a source that speaks EDI natively.

Problem Framing

A naive cutover treats this as a cutover date on a calendar: turn off the CSV job, turn on the 834 listener, done. That breaks for three reasons specific to a live enrollment feed.

First, the two formats do not carry the same shape of truth. CSV columns are whatever the carrier’s export tool happened to name them; the 834’s INS, HD, and DTP segments carry the same enrollment facts through a versioned X12 grammar, as covered in EDI 834 Parsing. A field-by-field mapping has to exist and be tested before a single 834 record is trusted, because a plan code or action code that looks equivalent across formats can encode a subtly different business rule.

Second, a straight cutover has no fallback. If the 834 feed drops a dependent’s termination date that the CSV always carried correctly, the first sign of trouble is a wrong deduction on a live paycheck, not a caught test failure. There is no interval during which both feeds are compared against each other, so there is nothing to roll back to except a manual reconstruction from carrier support tickets.

Third, “the file parses” is not the same claim as “the enrollment data is correct.” An 834 interchange can pass every structural check — balanced ST/SE counts, a clean 999 acknowledgment — and still misclassify an INS03 action code, invert a coverage window, or carry a premium amount in a different AMT qualifier than the one your mapping expects. Structural validity and business correctness are different gates, and only a parity comparison against a trusted source catches the second kind of error.

The fix is to never let the 834 feed become the source of truth in one step. Both the CSV and 834 adapters are run in parallel against the same enrollment periods, each is normalized to the identical canonical schema, and the two canonical sets are diffed record-for-record. Only when that diff is empty for a sustained window does the 834 feed take over, and the CSV path stays wired and ready to reactivate until decommissioning is a deliberate, separate decision.

Prerequisites & Data Requirements

Before the dual-run window opens, the CSV-to-834 field mapping must be written down and reviewed by whoever owns the carrier relationship — not inferred from sample files. Both adapters must already emit the same canonical record shape used across the ingestion framework, keyed on employee_id (or member id), plan_code, and coverage_effective date, following the same Data Boundary Definitions contract every other vector normalizes to.

Two preconditions gate the start of the dual-run window itself: the carrier must confirm the 834 trading-partner setup (AS2 or VAN transport, functional-acknowledgment handling) is live in a non-production environment that mirrors production cadence, and the mapping table above must have a signed-off owner for every row — an unmapped column is a blocker, not a “map it later.”

Step-by-Step Implementation

Step 1 — Define the shared canonical record and both adapters. Both feeds must emit exactly this shape so the diff step in Step 3 is a structural equality check, not a fuzzy comparison.

from dataclasses import dataclass
from decimal import Decimal
from typing import Optional


@dataclass(frozen=True)
class CanonicalEnrollment:
    """Same shape emitted by the CSV adapter and the 834 adapter."""

    employee_id: str
    plan_code: str
    action_code: str
    coverage_effective: str
    coverage_termination: Optional[str]
    premium_amount: Decimal
    is_dependent: bool

Expected output: CanonicalEnrollment(employee_id="E-4417", plan_code="PPO500", action_code="001", coverage_effective="2026-08-01", coverage_termination=None, premium_amount=Decimal("184.50"), is_dependent=False) from either source for the same real-world election.

Step 2 — Run both feeds against the same enrollment cycles. Tag every canonical record with the cycle it belongs to and the vector that produced it, and land both outputs in the same partitioned store rather than two separate systems that a human has to reconcile by hand. The Parsing EDI 834 files with Python guide covers building the 834 side of this adapter; the CSV side is the existing production loader, unmodified.

import logging

logger = logging.getLogger("payroll.migration.dual_run")


def tag_and_store(records, source_vector: str, cycle_id: str, sink: list) -> None:
    """Attach provenance and append to the shared dual-run store."""
    for record in records:
        sink.append({"cycle_id": cycle_id, "source_vector": source_vector, "record": record})
        logger.info(
            "dual_run_ingest cycle=%s vector=%s emp=%s plan=%s",
            cycle_id, source_vector, record.employee_id, record.plan_code,
        )

Expected output: after one cycle, sink contains two tagged copies of every enrollment event — one source_vector="csv", one source_vector="edi834" — with no record dropped from either side.

Step 3 — Diff the two canonical sets for the cycle. Because both adapters emit the identical dataclass, the comparison is a keyed equality check, not a tolerance-based reconciliation — money is Decimal on both sides, so an exact match is the only acceptable outcome.

from dataclasses import asdict
from typing import Dict, List

logger = logging.getLogger("payroll.migration.parity")


def _index_by_key(records: List[CanonicalEnrollment]) -> Dict[str, CanonicalEnrollment]:
    return {f"{r.employee_id}:{r.plan_code}:{r.coverage_effective}": r for r in records}


def diff_canonical_sets(
    csv_records: List[CanonicalEnrollment],
    edi_records: List[CanonicalEnrollment],
    cycle_id: str,
) -> Dict[str, list]:
    """Compare two canonical enrollment sets and return every discrepancy."""
    csv_index = _index_by_key(csv_records)
    edi_index = _index_by_key(edi_records)

    missing_in_edi = sorted(set(csv_index) - set(edi_index))
    missing_in_csv = sorted(set(edi_index) - set(csv_index))
    mismatched = []

    for key in sorted(set(csv_index) & set(edi_index)):
        if csv_index[key] != edi_index[key]:
            mismatched.append(
                {"key": key, "csv": asdict(csv_index[key]), "edi834": asdict(edi_index[key])}
            )

    logger.info(
        "parity_diff cycle=%s missing_in_edi=%s missing_in_csv=%s mismatched=%s",
        cycle_id, len(missing_in_edi), len(missing_in_csv), len(mismatched),
    )
    return {
        "missing_in_edi": missing_in_edi,
        "missing_in_csv": missing_in_csv,
        "mismatched": mismatched,
    }

Expected output for a clean cycle: {"missing_in_edi": [], "missing_in_csv": [], "mismatched": []}. Any non-empty list names the exact employee_id:plan_code:coverage_effective key that failed, which is what a carrier ticket references.

Step 4 — Gate cutover on sustained parity, not a single clean run. One empty diff can be luck — a light enrollment day with no edge cases. Require the diff to be empty across several consecutive cycles before advancing.

def evaluate_parity_gate(diff_history: List[Dict[str, list]], min_clean_cycles: int = 3) -> bool:
    """Return True only when the most recent N diffs are all fully empty."""
    if len(diff_history) < min_clean_cycles:
        return False
    recent = diff_history[-min_clean_cycles:]
    clean = all(
        not diff["missing_in_edi"] and not diff["missing_in_csv"] and not diff["mismatched"]
        for diff in recent
    )
    logger.info("parity_gate clean_cycles_required=%s result=%s", min_clean_cycles, clean)
    return clean

Expected output: evaluate_parity_gate(diff_history) returns True only after three consecutive cycles each returned the empty-list result from Step 3; a single mismatched cycle anywhere in the window resets the count.

Step 5 — Cut over behind a reversible flag. Cutover flips which adapter’s output feeds the deduction engines; it does not delete the CSV adapter or its job schedule.

def select_source_vector(parity_gate_passed: bool, force_csv: bool = False) -> str:
    """Single switch point: everything downstream reads this, nothing else."""
    if force_csv or not parity_gate_passed:
        return "csv"
    return "edi834"

Expected output: select_source_vector(parity_gate_passed=True) returns "edi834"; select_source_vector(parity_gate_passed=True, force_csv=True) still returns "csv" — the manual override always wins, which is the rollback mechanism in Step’s Verification below.

Verification

1. Parity diff is empty before cutover. Run diff_canonical_sets across the required min_clean_cycles window and assert every returned dict has all three lists empty; evaluate_parity_gate must return True before select_source_vector is ever allowed to return "edi834" in production configuration.
2. Rollback restores CSV. Flip force_csv=True after cutover and assert select_source_vector immediately returns "csv" regardless of the stored parity-gate history — the override must not require re-running the gate, because a rollback under incident pressure cannot wait on a batch job.
3. A seeded mismatch is caught, not averaged away. Inject one record into the 834 canonical set with a premium_amount off by Decimal("0.01") from its CSV counterpart and assert diff_canonical_sets places its key in mismatched — never silently passes because every other record in the cycle matched.
4. Dependent-flag mapping is exercised explicitly. Since is_dependent is derived differently per format (INS02 on the 834 side, a boolean column on the CSV side), seed one subscriber and one dependent record through both adapters and assert the canonical is_dependent values agree — this is the single highest-risk row in the mapping table.

Failure Modes

• Parity gate bypassed under a deadline. A carrier-imposed cutover date arrives before three clean cycles have accumulated, and the team advances the switch manually “just this once.” Root cause: treating the parity window as a soft recommendation instead of a hard gate enforced in code. Remediation: make select_source_vector read parity_gate_passed from the stored gate evaluation, not from a human’s assertion, so advancing past an unmet gate requires an explicit, logged override rather than a silent skip.
• Rollback path never exercised before it’s needed. The force_csv flag exists in code but the CSV adapter’s job schedule was disabled at cutover instead of left idling, so flipping the flag returns "csv" with no data behind it. Root cause: decommissioning too early — Step 5 flips the read path, but the CSV feed must keep running silently through cutover until Phase 4. Remediation: keep the CSV job scheduled and its output landing in the dual-run store through the entire cutover phase; only stop it at the deliberate decommission step.
• False mismatches from divergent idempotency keys. The CSV adapter keys on a content hash while the 834 adapter keys on the ISA13 interchange number, so diff_canonical_sets’ business key (employee_id:plan_code:coverage_effective) is fine, but a team that reuses each adapter’s own idempotency key as the diff key gets spurious mismatches. Root cause: conflating the replay/idempotency key with the parity-comparison key — they solve different problems. Fix: diff on the business-meaning key shown in Step 3, and keep each adapter’s native idempotency key internal to its own duplicate-suppression logic.

Frequently Asked Questions

Related

• Choosing Between Batch and Real-Time Payroll Sync — measuring a source’s real latency before committing either feed to an SLA.
• CSV vs EDI 834 vs REST API Ingestion Tradeoffs — the parent topic’s decision rubric this migration puts into practice.
• Reconciling EDI 834 Enrollments to Payroll Deductions — the downstream check that matches 834 premiums against actual payroll deductions post-cutover.
• Parsing EDI 834 Files with Python — the implementation this migration’s 834 adapter is built on.