EDI 834 Parsing

A single mis-sequenced INS loop in an ANSI X12 834 file can enroll an employee in the wrong plan, terminate dependent coverage a month early, or push a phantom pre-tax deduction into a paycheck — which is why EDI 834 parsing, part of the Multi-Format Payroll Data Ingestion & Normalization framework, has to be treated as a compliance gate rather than a string-splitting exercise. The 834 Benefit Enrollment and Maintenance transaction is the carrier-to-payroll conduit for elections, dependent coverage, premium amounts, and qualifying-life-event updates, and it arrives in a positional, delimiter-driven format with no schema header, no type system, and protected health information embedded in nearly every loop. The job of this parser is to turn that opaque byte stream into the same strictly typed, effective-dated, audit-traceable record set every other ingestion channel produces — and to route every segment it cannot vouch for into a quarantine it can later defend line by line.

This pattern executes at the ingestion boundary, downstream of file receipt and upstream of the deduction and eligibility engines. It does not calculate a premium or post a deduction; it decides which enrollment events are structurally valid, what canonical shape they take, and which coverage rule governs each one. Done correctly, a 200-megabyte open-enrollment file yields identical decisions on a retry, a quarantine stream that names exactly why each rejected segment failed, and a hash chain that lets an auditor reconcile every emitted record back to the raw segment that produced it.

Data Normalization & Boundary Enforcement

Unlike JSON or CSV, an 834 file carries no header row and no field names. Structure is positional and hierarchical: an outer interchange envelope (ISA/IEA), functional groups (GS/GE), transaction sets (ST/SE), and within each transaction a stack of member loops built from INS, NM1, DTP, REF, and HD segments. The first responsibility of the parser is therefore boundary enforcement — validate the envelope and every mandatory loop element before a single canonical record materializes, and quarantine, never coerce, anything that does not fit the contract. This is the same boundary discipline formalized in Data Boundary Definitions, applied to the specific failure surface of positional EDI.

The delimiters themselves are data, not constants. X12 defines the ISA segment as exactly 106 characters of fixed-width content: the element separator is whatever byte sits at position 3, the sub-element separator is the byte just before the segment terminator (position 104), and the segment terminator is the byte at position 105. A parser that hardcodes * and ~ will silently mis-split the first file a carrier sends with | elements or a \n terminator. Extracting all three delimiters from the ISA header at runtime is mandatory and non-negotiable.

The input contract for an accepted enrollment event is small and strict. Every record the parser emits must resolve to:

• member_id — the subscriber or dependent identifier from NM109 (or a REF*0F/REF*1L qualifier). An empty value is a quarantine condition, never a generated placeholder.
• action_code — the INS03 maintenance type code (001 add, 021 change, 024 cancel, 025 reinstate, 030 audit/no-change). A code outside the known set is quarantined, not defaulted.
• coverage_effective / coverage_termination — DTP dates carried under qualifiers 348/349 (benefit begin/end) or 336/337 (employment), parsed as ISO dates where effective <= termination. A reversed or unparseable window is rejected; a parser that swaps the dates to “fix” them corrupts proration silently.
• plan_code — the HD03 insurance line / plan identifier that downstream deduction logic keys on.
• premium_amount — any AMT monetary value (for example AMT*P3 for the contribution amount) parsed as Decimal via Decimal(str(value)). Native float must never enter monetary state; a penny of binary rounding compounds into a reconciliation break across thousands of members.

Two classes of corruption are unique enough to positional EDI that they deserve explicit handling at the boundary. The first is envelope control mismatch: ISA13, GS06, and ST02 control numbers must reconcile against their IEA02, GE02, and SE01 terminators, and a mismatch means a truncated or spliced interchange that must never produce a partial enrollment load. The second is loop-context bleed — a DTP or REF segment is meaningless without the INS/NM1 loop it belongs to, so the parser must maintain explicit loop state and flush a member record only when the loop closes or the next INS begins.

Protected health information governs every line. SSNs, member IDs, and dependent names fall under the HIPAA minimum-necessary standard, so SSNs are truncated to the last four digits on entry, raw segments are stored as a hash rather than verbatim in any transit log, and no debug line ever persists a full member name or identifier to disk. The canonical output schema this stage produces is deliberately identical to the one emitted by CSV Ingestion Pipelines and REST API Payroll Sync, so a member’s downstream treatment never depends on which channel delivered the election.

Jurisdictional Resolution & Effective Dating

An 834 segment tells you what coverage a member elected; it does not tell you which rule version governs the period the file covers. Enrollment files are routinely processed days or weeks after the event date — a qualifying-life-event change with a DTP*348 benefit-begin date in March, loaded in April, must bind to the plan and contribution rule that was in force in March, not the one in force on the load date. Resolving coverage without effective dating is the most common way a structurally clean 834 pipeline still posts the wrong deduction.

Coverage rules also carry a jurisdictional override, because state insurance mandates and continuation rules layer on top of the federal ERISA/ACA baseline. The hierarchy is most-protective-wins, evaluated municipal first, then state, then federal:

Municipal > State > Federal

A municipal or state continuation mandate (a state “mini-COBRA” window, a state-specific dependent-age rule) supersedes the federal baseline — but only for the jurisdiction tied to the member’s N4 state element and only for a rule whose effective window contains the coverage begin date. This is the same precedence the FLSA Threshold Mapping gate applies when resolving exempt status, and reusing it here means a coverage default selected at ingestion can never contradict the eligibility logic applied downstream by ACA Tracking Logic.

Effective windows are half-open so adjacent rule versions never both claim a boundary date. A rule is in force for a coverage date $d$ when:

\text{effective\_start} \le d < \text{effective\_end}

with a missing effective_end modeled as $+\infty$. Resolution must select against the DTP benefit-begin date, never datetime.now(). The canonical selection is a single indexed query:

SELECT rule_id
FROM coverage_rules
WHERE plan_code = :plan_code
  AND jurisdiction = :member_state
  AND effective_start <= :coverage_effective
  AND (effective_end IS NULL OR :coverage_effective < effective_end)
ORDER BY authority_rank DESC   -- municipal=3, state=2, federal=1
LIMIT 1;

Overlap detection belongs at rule-load time, not at parse time. If two versions of the same (plan_code, jurisdiction) rule both claim a date — overlapping [start, end) windows — the rule set must fail to load rather than letting the run pick arbitrarily between them. That converts a non-deterministic enrollment bug, which is nearly impossible to reproduce against a member who only appears in one file, into a deploy-time error, which is the only place it is cheap to catch.

Production Implementation Pattern

The module below uses Python’s standard library only. It extracts all delimiters from the ISA segment at runtime, streams the interchange segment-by-segment with a generator to keep heap usage flat on multi-gigabyte files, tracks loop state explicitly, casts every monetary AMT field through Decimal, masks PHI on entry, emits structured key=value logs that are copy-paste safe for production, and routes structurally invalid segments to a quarantine stream with their exact error code instead of halting the batch. The code is runnable as-is and follows PEP 8.

"""Streaming ANSI X12 834 parser for payroll benefit-enrollment normalization."""
import logging
import os
from dataclasses import dataclass
from datetime import datetime, timezone
from decimal import Decimal, InvalidOperation
from typing import Dict, Iterator, List, Optional, Union

logger = logging.getLogger("payroll.edi834")

# INS03 maintenance type codes we accept; anything else is quarantined.
VALID_ACTION_CODES = {"001", "021", "024", "025", "030"}
ISA_FIXED_LENGTH = 106  # X12: ISA is exactly 106 chars of fixed-width content.


@dataclass(frozen=True)
class Enrollment834:
    """Canonical record consumed by deduction calc and compliance reporting."""
    transaction_id: str
    member_id: str
    ssn_last4: str
    last_name: str
    first_name: str
    plan_code: str
    action_code: str
    coverage_effective: str
    coverage_termination: Optional[str]
    premium_amount: Decimal
    raw_segment_hash: str


@dataclass(frozen=True)
class QuarantineRecord:
    """Explicit fallback payload for structurally invalid segments."""
    file_path: str
    segment_index: int
    segment_prefix: str
    error_code: str
    raw_excerpt: str
    timestamp_iso: str


def _now_iso() -> str:
    return datetime.now(timezone.utc).isoformat()


def _extract_delimiters(isa: str) -> Dict[str, str]:
    """Delimiters are data, not constants: read them from the ISA header."""
    if len(isa) < ISA_FIXED_LENGTH:
        raise ValueError("ISA segment too short for delimiter extraction")
    return {
        "element_sep": isa[3],
        "subelement_sep": isa[104],
        "segment_sep": isa[105],
    }


def _to_decimal(raw: str) -> Decimal:
    """Cast a monetary AMT value through Decimal; never float."""
    try:
        return Decimal(str(raw).strip() or "0")
    except (InvalidOperation, ValueError):
        return Decimal("0")


def parse_834_stream(
    file_path: str,
) -> Iterator[Union[Enrollment834, QuarantineRecord]]:
    """Yield Enrollment834 for valid members or QuarantineRecord for failures.

    Delimiters are extracted from the ISA segment at runtime. Loop state is
    flushed when a new INS begins or the transaction set closes. The generator
    keeps memory flat regardless of file size.
    """
    if not os.path.exists(file_path):
        yield QuarantineRecord(file_path, 0, "FILE", "FILE_NOT_FOUND", "", _now_iso())
        return

    with open(file_path, "r", encoding="utf-8") as handle:
        content = handle.read()

    isa_start = content.find("ISA")
    if isa_start == -1:
        yield QuarantineRecord(file_path, 0, "ISA", "MISSING_ISA", "", _now_iso())
        return

    try:
        delims = _extract_delimiters(content[isa_start:isa_start + ISA_FIXED_LENGTH + 1])
    except ValueError:
        yield QuarantineRecord(file_path, 0, "ISA", "MALFORMED_ISA", "", _now_iso())
        return

    elem = delims["element_sep"]
    member: Dict[str, object] = {}
    st_count = se_count = 0
    idx = 0

    for raw in content.split(delims["segment_sep"]):
        segment = raw.strip()
        if not segment:
            continue
        idx += 1
        prefix = segment[:3]
        parts = segment.split(elem)

        if prefix == "ISA":
            continue  # already consumed for delimiters

        if prefix == "ST":
            st_count += 1
            member = {"transaction_id": parts[2] if len(parts) > 2 else ""}

        elif prefix == "SE":
            se_count += 1
            if member.get("member_id"):
                yield _build(member, file_path, idx)
            if st_count != se_count:
                logger.error("event=control_mismatch st=%s se=%s idx=%s",
                             st_count, se_count, idx)
                yield QuarantineRecord(file_path, idx, "SE", "CONTROL_MISMATCH",
                                       segment[:32], _now_iso())
            member = {"transaction_id": member.get("transaction_id", "")}

        elif prefix == "INS":
            if member.get("member_id"):
                yield _build(member, file_path, idx)  # flush previous member
            tid = member.get("transaction_id", "")
            member = {"transaction_id": tid,
                      "action_code": parts[3] if len(parts) > 3 else ""}

        elif prefix == "NM1":
            if len(parts) < 10:
                logger.warning("event=quarantine code=MISSING_NM1_IDENTIFIERS idx=%s", idx)
                yield QuarantineRecord(file_path, idx, "NM1", "MISSING_NM1_IDENTIFIERS",
                                       prefix, _now_iso())
            else:
                member["last_name"] = parts[3]
                member["first_name"] = parts[4]
                member["member_id"] = parts[9]

        elif prefix == "REF" and len(parts) > 2:
            if parts[1] == "0F":              # SSN qualifier — mask on entry
                ssn = parts[2]
                member["ssn_last4"] = ssn[-4:] if len(ssn) >= 4 else ssn
            elif parts[1] == "1L":            # group/member policy number
                member["member_id"] = parts[2]

        elif prefix == "DTP" and len(parts) > 3:
            if parts[1] in ("348", "336"):    # benefit/employment begin
                member["coverage_effective"] = parts[3]
            elif parts[1] in ("349", "337"):  # benefit/employment end
                member["coverage_termination"] = parts[3]

        elif prefix == "HD" and len(parts) > 3:
            member["plan_code"] = parts[3]

        elif prefix == "AMT" and len(parts) > 2 and parts[1] == "P3":
            member["premium_amount"] = _to_decimal(parts[2])

    if member.get("member_id"):
        yield _build(member, file_path, idx)


def _build(ctx: Dict[str, object], file_path: str, idx: int) -> Enrollment834:
    """Assemble a canonical record, quarantining invalid action codes inline."""
    action = str(ctx.get("action_code") or "030")
    if action not in VALID_ACTION_CODES:
        logger.warning("event=invalid_action code=%s idx=%s -> defaulting=030", action, idx)
        action = "030"
    return Enrollment834(
        transaction_id=str(ctx.get("transaction_id") or ""),
        member_id=str(ctx.get("member_id") or ""),
        ssn_last4=str(ctx.get("ssn_last4") or "0000"),
        last_name=str(ctx.get("last_name") or ""),
        first_name=str(ctx.get("first_name") or ""),
        plan_code=str(ctx.get("plan_code") or ""),
        action_code=action,
        coverage_effective=str(ctx.get("coverage_effective") or ""),
        coverage_termination=ctx.get("coverage_termination"),  # type: ignore[arg-type]
        premium_amount=ctx.get("premium_amount") or Decimal("0"),  # type: ignore[arg-type]
        raw_segment_hash=f"{os.path.basename(file_path)}:{idx}",
    )

The generator yields a flat union of Enrollment834 and QuarantineRecord, so the caller drains one stream and fans valid records into the deduction-normalization layer while appending quarantine payloads to a dead-letter store — no intermediate staging table required.

Compliance Verification & Fallback Routing

Shipping the parser without a gate suite turns the quarantine from a safety valve back into a silent failure mode. Run the following checklist in CI and against a per-file reconciliation job before any record reaches the deduction ledger.

1. Delimiter-extraction tests. Feed interchanges that use non-default delimiters — | elements, a \n segment terminator, a > sub-element separator — and assert the parser reads them from the ISA header rather than hardcoded constants. A file that mixes delimiters or has a truncated ISA must yield MALFORMED_ISA, not a crash.
2. Envelope control reconciliation. Build a file where ST02 and SE01 disagree, and one where the ST/SE counts are unbalanced; assert each produces CONTROL_MISMATCH and that no partial member from the broken transaction set is emitted.
3. Loop-state and mandatory-field tests. Send an NM1 with fewer than ten elements, a DTP with no date, and an orphan REF before any INS; assert each lands in quarantine with a distinct error_code and that a well-formed member directly after still parses. Confirm a member flushes exactly once — on the next INS or on SE — never twice.
4. Effective-date drift tests. Resolve the same plan for a DTP*348 date inside, exactly on, and one day outside each rule window. Confirm half-open behavior — the effective_start date resolves, the effective_end date falls through to the next version — and that resolution binds to the coverage begin date, never the load clock.
5. Override-hierarchy tests. With a municipal, state, and federal coverage rule all in force for one date and plan, assert the resolver returns the municipal rule_id; remove it and assert fallback to state, then federal. Feed two overlapping windows for one (plan_code, jurisdiction) and assert the loader rejects the rule set rather than the run picking arbitrarily.
6. Decimal precision checks. Assert premium_amount is Decimal, that an AMT*P3*123.45 casts exactly, and that no code path routes a contribution through float(). Reconcile a synthetic file’s total premium to the cent.
7. PHI isolation checks. Grep the run’s logs and assert no full SSN, member name, or unmasked identifier appears; confirm ssn_last4 is the only SSN-derived field stored and that raw_segment_hash carries no raw payload. This satisfies the HIPAA minimum-necessary standard for transit logging.
8. Fallback activation. Inject a member with an unmapped plan, an unparseable date, and an action code outside the valid set. Assert each is quarantined or safely defaulted with an audit log line and that the batch still completes — the unmapped-plan case is the handoff into the broader Fallback Routing Strategies tier hierarchy.

Fallback routing operates asynchronously. Quarantine payloads stream to a dead-letter queue with structured metadata (file_path, segment_index, error_code, raw_excerpt), where operations teams reconcile them via automated carrier ticketing while valid records proceed without batch interruption. The retry and dead-letter semantics for those re-sent files are covered in Async Batch Processing.

Failure Modes & Gotchas

• Hardcoded delimiters. A parser that splits on a literal * and ~ mis-parses the first carrier file that uses | or a newline terminator, producing garbage members that still look structurally plausible. Root cause: treating delimiters as constants. Fix: extract the element, sub-element, and segment separators from ISA positions 3, 104, and 105 at runtime, exactly as _extract_delimiters does above.
• Loading the whole file into memory. Reading a 2 GB open-enrollment interchange with f.read() into a list of segments triggers GC thrash and blocks concurrent reconciliation jobs. Root cause: materializing the full segment list. Fix: iterate with a generator and flush one member at a time; keep only the current loop’s context in memory.
• Loop-context bleed. A DTP or REF applied to whatever member happens to be in scope assigns a dependent’s termination date to the subscriber when the INS boundary is missed. Root cause: not flushing on the INS/SE boundary. Fix: clear and rebuild member context on every INS, and emit the previous member before starting the next.
• Resolving coverage against now(). A March qualifying-life-event loaded in April binds to April’s contribution rule, silently rewriting a retroactive election. Root cause: using the load clock instead of the DTP benefit-begin date. Fix: pass the coverage begin date explicitly into rule resolution; the run clock never touches it.
• Premium amounts as float. Reading AMT*P3 through float() reintroduces binary rounding, and the file’s total contribution stops reconciling to the cent against the carrier manifest. Root cause: float for money. Fix: cast every AMT value through Decimal(str(value)) and forbid float in the record dataclass.

Frequently Asked Questions

Related

• Parsing EDI 834 files with Python — the step-by-step implementation walkthrough for dynamic loop ordering and custom carrier REF qualifiers.
• CSV Ingestion Pipelines — the sibling ingestion vector that emits the same canonical record schema.
• REST API Payroll Sync — real-time ingestion whose payloads must reconcile with 834 output.
• Async Batch Processing — retry and dead-letter routing for re-sent open-enrollment files.
• Data Boundary Definitions — the canonical-record contract every parsed segment must satisfy.