Deriving the SDTM LB Domain from Central Lab Feeds

The Laboratory (LB) domain is where two data sources collide: a central-lab vendor ships results as a CSV or HL7 feed in whatever units the analyzer produced, while the EDC carries a subset of the same labs entered by site staff, and the two disagree on units, visit labels, and which draws even happened. This guide is the practical companion to the CDISC SDTM Transformation Pipelines guide and sits inside the wider Clinical Data Architecture & EDC Standards reference, and it is written for clinical data managers and Python ETL engineers who must turn a messy lab feed into a conformant, unit-standardized LB dataset without inventing a single result. Each step below isolates one responsibility — unit harmonization into LBSTRESN, reference-range flagging into LBNRIND, and feed-versus-EDC reconciliation — and carries its regulatory rationale.

The LB Derivation Flow #

The pipeline lands the raw central-lab feed, standardizes each result into SI-consistent units, derives the normal-range indicator, reconciles against EDC-entered labs, and gates the frame before it reaches a monitored environment.

Root Cause: Why Central-Lab Feeds Fight the LB Model #

The LB domain is a Findings-class dataset with a strict --TESTCD/--ORRES/--STRESC grain, and central-lab feeds are built for a different purpose. An analyzer reports glucose in mg/dL at one lab and mmol/L at another; the same feed can carry hematology in 10^9/L and chemistry in conventional units, so a single LBORRESU-to-LBSTRESU map is never enough. Reference ranges arrive per-result as low and high bounds (LBORNRLO, LBORNRHI) that vary by lab, sex, and age, which means LBNRIND cannot be a static lookup — it must be recomputed against the range shipped with each record. On top of that, the EDC holds site-entered labs for the same subjects and visits, and SDTM must represent one coherent LB even when the two sources disagree. Results below the limit of quantification (< 0.5) and qualitative results (POSITIVE, TRACE) are not numbers at all, so LBSTRESN must stay null while LBSTRESC preserves the character result. Every one of these is a place a silent integrity defect enters — the derivation is deterministic precisely so it does not.

Step-by-Step Implementation #

Step 1: Land the Feed and Anchor Provenance #

Parse the central-lab CSV or HL7 feed with types declared explicitly, and hash the raw payload before any transform so the source is reconstructable. Numeric-looking columns stay as strings at the boundary, because a result like <0.5 is not a float and coercing it early destroys information.

# ALCOA+ (Original): hash the raw lab row before any transform so the
# as-received value can be reconstructed and defended at inspection.
import hashlib
import pandas as pd

def land_lab_feed(csv_path: str) -> pd.DataFrame:
    df = pd.read_csv(csv_path, dtype=str, keep_default_na=False)  # never coerce results yet
    df["STUDYID"] = "CDP-204"
    df["DOMAIN"] = "LB"
    df["source_hash"] = df.apply(
        lambda r: hashlib.sha256(
            "|".join(str(r[c]) for c in df.columns).encode("utf-8")
        ).hexdigest(),
        axis=1,
    )
    return df

The same read-only landing discipline used across ingestion is documented in Pandas DataFrames for Clinical Data Cleaning; the feed is evidence, so it is parsed, never edited in place.

Step 2: Map Test Codes, Category, and Specimen to Controlled Terminology #

Bind each analyzer test name to its CDISC LBTESTCD/LBTEST pair through a versioned dictionary, and set LBCAT (for example CHEMISTRY, HEMATOLOGY, URINALYSIS) and LBSPEC (specimen, such as SERUM or URINE). Map by code, never by column order, and route an unknown test to review rather than dropping it.

# ALCOA+ (Consistent): map analyzer names to LB Controlled Terminology
# through a versioned dictionary; an unmapped test is surfaced, not dropped.
LB_CT_VERSION = "SDTM CT 2026-03-28"
LB_TEST_MAP = {
    "GLUCOSE":    {"LBTESTCD": "GLUC", "LBTEST": "Glucose",    "LBCAT": "CHEMISTRY",  "LBSPEC": "SERUM"},
    "ALT (SGPT)": {"LBTESTCD": "ALT",  "LBTEST": "Alanine Aminotransferase", "LBCAT": "CHEMISTRY", "LBSPEC": "SERUM"},
    "HEMOGLOBIN": {"LBTESTCD": "HGB",  "LBTEST": "Hemoglobin",  "LBCAT": "HEMATOLOGY", "LBSPEC": "BLOOD"},
}

def map_lab_tests(df):
    unknown = sorted(set(df["LBTEST_RAW"]) - LB_TEST_MAP.keys())
    resolved = df.join(df["LBTEST_RAW"].map(LB_TEST_MAP).apply(pd.Series))
    resolved["lb_ct_version"] = LB_CT_VERSION
    return resolved, unknown  # route `unknown` tests to a data-management query

Step 3: Harmonize Units into LBSTRESN/LBSTRESU with a Logged Factor #

This is the heart of the domain: the as-collected LBORRES/LBORRESU pair must be converted to standardized LBSTRESC/LBSTRESN/LBSTRESU. Drive every conversion from a factor table keyed by test and source unit, and log the factor applied to each row so the arithmetic is reproducible. A missing factor is a hard stop, not a silent pass-through.

# 21 CFR Part 11 (Accurate): every unit conversion applies a table-driven
# factor and records it, so LBSTRESN is a reproducible, auditable derivation.
from decimal import Decimal, InvalidOperation

# (LBTESTCD, from-unit) -> (factor, standardized unit)
CONV = {
    ("GLUC", "mg/dL"):  (Decimal("0.0555"), "mmol/L"),
    ("GLUC", "mmol/L"): (Decimal("1"),      "mmol/L"),
    ("HGB",  "g/dL"):   (Decimal("10"),     "g/L"),
}

def standardize_units(row: dict) -> dict:
    raw = row["LBORRES"]
    row["LBSTRESC"] = raw                      # character result always preserved
    try:
        value = Decimal(raw)                   # numeric branch only
    except (InvalidOperation, TypeError):
        row["LBSTRESN"] = None                 # non-numeric handled in Step 5
        row["LBSTRESU"] = row["LBORRESU"]
        row["conv_factor"] = None
        return row
    key = (row["LBTESTCD"], row["LBORRESU"])
    if key not in CONV:
        raise KeyError(f"No conversion factor for {key}")  # fail loud, never guess
    factor, std_unit = CONV[key]
    row["LBSTRESN"] = float(value * factor)
    row["LBSTRESC"] = format(value * factor, "f")
    row["LBSTRESU"] = std_unit
    row["conv_factor"] = str(factor)           # logged for audit reconstruction
    return row

Step 4: Derive LBNRIND and Carry LBFAST from the Shipped Range #

With a standardized numeric value in hand, derive the normal-range indicator by comparing LBSTRESN against the reference bounds shipped with the record. Convert the bounds with the same factor so the comparison is unit-consistent, and set LBNRIND to LOW, NORMAL, or HIGH only when both the value and the range are numeric.

# ALCOA+ (Accurate): LBNRIND is computed against the range shipped per result,
# in the standardized unit; ambiguous cases stay null rather than guessing NORMAL.
def derive_nrind(row: dict) -> dict:
    val = row.get("LBSTRESN")
    lo, hi = row.get("LBSTRNRLO"), row.get("LBSTRNRHI")   # already unit-converted
    row["LBFAST"] = "Y" if row.get("FASTING_FLAG") == "1" else "N"
    if val is None or lo is None or hi is None:
        row["LBNRIND"] = None                # no range or non-numeric -> leave blank
        return row
    if val < lo:
        row["LBNRIND"] = "LOW"
    elif val > hi:
        row["LBNRIND"] = "HIGH"
    else:
        row["LBNRIND"] = "NORMAL"
    return row

Step 5: Reconcile the Feed Against EDC-Entered Labs #

Join the central-lab feed to the EDC labs on subject, visit, and test, and classify every pair: feed-only, EDC-only, or matched-with-discrepancy. Reconciliation does not overwrite the central result — the central lab is the system of record for the value — but a mismatch or an orphaned draw becomes a query, and a below-LOQ or non-numeric result is flagged for coding.

# ALCOA+ (Complete, Consistent): reconcile feed vs EDC by key; discrepancies
# and orphaned draws become queries, never silently reconciled away.
LOQ_PATTERN = r"^\s*[<>]"

def reconcile(feed: pd.DataFrame, edc: pd.DataFrame) -> pd.DataFrame:
    keys = ["USUBJID", "VISITNUM", "LBTESTCD"]
    merged = feed.merge(edc, on=keys, how="outer", suffixes=("_feed", "_edc"), indicator=True)
    merged["recon_status"] = merged["_merge"].map({
        "left_only": "FEED_ONLY", "right_only": "EDC_ONLY", "both": "MATCHED"})
    # Non-numeric / below-LOQ results carry a character result but null LBSTRESN
    merged["needs_query"] = (
        merged["LBSTRESC_feed"].str.match(LOQ_PATTERN, na=False)
        | (merged["recon_status"] == "EDC_ONLY")          # draw the site logged but the lab never reported
    )
    return merged

Unmapped tests and reconciliation discrepancies feed the discrepancy workflow described in Automated Clinical Query Generation, so an orphaned draw becomes a query rather than a hole in the safety data.

Step 6: Gate on Conformance Before Promotion #

Assert the LB grain, controlled-terminology membership, and the numeric-versus-character invariant with a schema contract, and archive the report as OQ evidence. The critical rule the gate enforces: a null LBSTRESN must always be accompanied by a non-empty LBSTRESC, so a below-LOQ result is never mistaken for missing data.

# GxP test artifact: the pandera contract is the executable LB conformance
# spec; its pass/fail report is archived as OQ evidence for the run.
import pandera.pandas as pa

lb_schema = pa.DataFrameSchema({
    "LBTESTCD": pa.Column(str, pa.Check.str_matches(r"^[A-Z0-9]{1,8}$")),
    "LBCAT":    pa.Column(str, pa.Check.isin(["CHEMISTRY", "HEMATOLOGY", "URINALYSIS"])),
    "LBSTRESC": pa.Column(str, nullable=False),          # character result always present
    "LBNRIND":  pa.Column(str, pa.Check.isin(["LOW", "NORMAL", "HIGH"]), nullable=True),
    "LBDTC":    pa.Column(str, pa.Check.str_matches(r"^\d{4}-\d{2}-\d{2}")),  # ISO 8601
})

def gate(df):
    return lb_schema.validate(df, lazy=True)

The same conformance-gate pattern anchors every SDTM domain in this guide and is generalized in Validation and CSV Frameworks.

Verification and Audit Trail #

Confirming an LB derivation is itself a regulated activity. Capture the following evidence on every run so the transformation can be reconstructed and defended at inspection:

Run the derivation twice on identical feed input and assert byte-identical output with pandas.testing.assert_frame_equal; a stable result is your determinism proof for the audit file.

Edge Cases and Vendor-Specific Gotchas #

Log the conversion factor, not just the result. An inspector will ask how 142 mg/dL became 7.88 mmol/L. If you only store the standardized value, you cannot prove the arithmetic. Persist conv_factor and the CONV-table version on the row so the derivation is reconstructable; a change to the factor table is a change-control event, not a silent code edit.

Unscheduled visits break rigid visit windows. Central labs report draws that never map cleanly to a protocol visit — repeat draws, early-termination labs, retests. Do not force these into the nearest VISITNUM; carry VISIT as UNSCHEDULED with the actual LBDTC, and let the reconciliation flag the timing rather than corrupting a scheduled record.

Below-LOQ and qualitative results are not numbers. A result of <0.5 or TRACE must keep LBSTRESN null while LBSTRESC holds the character value, and it should raise a coding query. Coercing <0.5 to 0 or 0.5 fabricates a measurement and is a data-integrity finding; the character result plus a query is the defensible path.

Frequently Asked Questions #

Related #

• CDISC SDTM Transformation Pipelines — the parent guide covering the shared SDTM transformation architecture this LB build follows.
• Clinical Data Architecture & EDC Standards — the reference architecture this domain-derivation layer sits within.
• Mapping Vital Signs to the SDTM VS Domain — a sibling Findings-class build with its own unit-conversion and status handling.
• Handling SUPPQUAL Supplemental Qualifiers in SDTM — where non-standard lab fields that do not fit a core LB variable are parked.
• Automated Clinical Query Generation — where unmapped tests, below-LOQ results, and reconciliation discrepancies become queries.