CDISC SDTM Transformation Pipelines: Domain-by-Domain Mapping in Python

Every regulatory submission stands or falls on one transformation: the move from operational EDC capture into the Study Data Tabulation Model (SDTM), the CDISC standard the FDA requires for tabulation datasets. This guide solves the engineering problem of building that transformation as a domain-by-domain, traceable Python pipeline — one that routes a raw EDC extract by observation class, derives USUBJID and --SEQ deterministically, aligns every coded value to controlled terminology, emits a conformant define.xml, and refuses to ship anything that fails a Pinnacle 21 conformance gate. It sits within the broader reference design of Clinical Data Architecture & EDC Standards, where the EDC is an immutable, read-only source of truth and traceability from raw record to submission variable is the property auditors test first. Get the pipeline architecture right and a study team can regenerate its entire SDTM package from source on demand; get it wrong and every downstream ADaM dataset, table, and listing inherits an undefendable lineage.

SDTM Transformation at a Glance #

The pipeline reads a raw EDC extract, routes each record to the SDTM domain that matches its observation class, applies a per-domain mapper, harmonizes controlled terminology while generating define.xml metadata, and passes the whole package through Pinnacle 21 before it is allowed to become a submission deliverable.

Concepts and Prerequisites #

SDTM is not a file format but a content model, and the pipeline only makes sense once its three organizing ideas are clear. First, the SDTM Implementation Guide (SDTMIG) organizes every observation into one of three general observation classes: Interventions (what was given to or done to the subject — exposure, concomitant medications, procedures), Events (what happened to the subject — adverse events, medical history, disposition), and Findings (what was measured or observed — labs, vital signs, ECG, questionnaires). The class dictates the variable skeleton, so classifying a record correctly is the first real decision the pipeline makes. As described in the CDISC SDTM standard, a Findings domain like LB or VS is built around a --TESTCD/--ORRES/--STRESN grain, an Events domain like AE around a verbatim term plus a decode, and Interventions around dose and route. Second, two variables are structurally universal: USUBJID, the unique subject identifier that must be identical for a subject across every domain in the study, and the --SEQ sequence number that makes each record unique within a subject and domain. Third, controlled terminology (CT) — the versioned NCI/EVS codelists CDISC publishes quarterly — governs the permissible values of coded variables, and drift between CT versions is one of the most common sources of silent conformance failure.

The observation class is the first branch in the mapper, because it fixes the variable skeleton before any value is mapped:

Special-purpose domains (DM, CO, SE, SV) sit outside the three classes with their own fixed structures — DM in particular carries one record per subject and no --SEQ, a shape the Building the SDTM DM Domain from EDC Demographics guide works through in full.

Around those content rules sit the deliverables and gates. define.xml (the Case Report Tabulation Data Definition, define-XML v2.1) is the machine-readable metadata that describes each dataset, variable, codelist, and derivation to the reviewer; it is a required part of the submission, not an afterthought. Relationship datasets carry what the two-dimensional domains cannot: SUPP-- (Supplemental Qualifiers) holds non-standard variables that have no home in the parent domain, and RELREC records relationships between records across domains. Finally, Pinnacle 21 (the community and enterprise validator that implements the FDA and CDISC conformance rules) is the compliance gate: its report is the artifact an FDA reviewer effectively re-runs, so the pipeline treats a non-clean report as a build failure. The two regulatory anchors are the FDA Study Data Standards guidance, which mandates SDTM, and 21 CFR Part 11 with the ALCOA+ data-integrity model, which requires the whole transformation to be attributable, reproducible, and reconstructable.

The implementation below assumes a version-pinned runtime committed to a lockfile:

Implementation: Routing Raw Records by Observation Class #

The first stage classifies each raw record and dispatches it to the mapper that owns its target domain. Routing is data-driven — the mapping spec names each source form and the domain it becomes — so adding a domain never means editing the router. Keeping the dispatch explicit rather than inferred is what makes the pipeline auditable: every record’s destination domain is a decision recorded against a versioned spec, not a heuristic.

# ALCOA+ requirement: Attributable + Traceable — every raw record is routed to
# exactly one SDTM domain by a versioned spec, never by column-name guessing.
import pandas as pd

DOMAIN_CLASS = {          # SDTM general observation class per domain
    "DM": "special-purpose",
    "AE": "events",
    "LB": "findings",
    "VS": "findings",
}


def route_records(extract: pd.DataFrame, spec: dict) -> dict[str, pd.DataFrame]:
    """Split a raw EDC extract into one frame per target SDTM domain."""
    form_to_domain = spec["form_to_domain"]            # {FORM_OID: domain}
    routed: dict[str, list] = {d: [] for d in DOMAIN_CLASS}
    unrouted = []
    for _, row in extract.iterrows():
        domain = form_to_domain.get(row["FORM_OID"])
        if domain is None:
            unrouted.append(row["FORM_OID"])           # surfaced, never dropped
            continue
        routed[domain].append(row)
    if unrouted:
        raise ValueError(f"Unrouted forms — spec incomplete: {sorted(set(unrouted))}")
    return {d: pd.DataFrame(rows).reset_index(drop=True) for d, rows in routed.items()}

An unrouted form raises rather than silently disappearing, because a form that maps to no domain is either a spec gap or an unexpected EDC build change — both states an engineer must see before the run continues. The DOMAIN_CLASS table is more than documentation: downstream mappers assert their expected class, so a Findings domain can never accidentally be built with the Events variable skeleton.

Implementation: Deriving USUBJID and --SEQ Deterministically #

The two universal variables are derived once, centrally, so every domain agrees on them. USUBJID is composed from study, site, and subject identifiers in a fixed order; because it is the join key across DM, AE, LB, VS, and every other domain, computing it inconsistently is the single fastest way to break referential integrity across a submission. --SEQ is assigned per subject and domain after a deterministic sort, so the same input always yields the same sequence numbers — a property Pinnacle 21 uniqueness checks and reviewers both depend on.

# ALCOA+ requirement: Consistent — USUBJID is identical for a subject across ALL
# domains, and --SEQ is a reproducible, gap-tolerant per-subject sequence.
import pandas as pd


def derive_usubjid(df: pd.DataFrame, studyid: str) -> pd.DataFrame:
    df = df.copy()
    df["STUDYID"] = studyid
    # Fixed composition order: STUDYID-SITEID-SUBJID. Never reorder these parts.
    df["USUBJID"] = (
        studyid + "-" + df["SITEID"].astype(str) + "-" + df["SUBJID"].astype(str)
    )
    return df


def assign_seq(df: pd.DataFrame, domain: str, sort_keys: list[str]) -> pd.DataFrame:
    df = df.sort_values(["USUBJID", *sort_keys], kind="mergesort").copy()
    seq_var = f"{domain}SEQ"
    # groupby.cumcount is stable given the mergesort above; +1 for 1-based --SEQ.
    df[seq_var] = df.groupby("USUBJID").cumcount() + 1
    return df.reset_index(drop=True)

The mergesort is deliberate — it is the only stable sort in pandas, so records that tie on the sort keys keep their input order and the assigned --SEQ is reproducible run to run. Deriving USUBJID from string parts with an explicit separator (rather than an opaque hash) keeps it human-legible for reviewers while still being deterministic. The same disciplined frame handling that feeds these derivations is covered in Pandas DataFrames for Clinical Data Cleaning, and the per-domain mechanics live in the companion deep-dives for Building the SDTM DM Domain from EDC Demographics and Mapping Adverse Events to the SDTM AE Domain.

Controlled Terminology, define.xml, and Relationship Datasets #

With records routed and keyed, the transform harmonizes coded values against controlled terminology and captures the metadata that describes what it produced. CT harmonization is a strict lookup against a pinned codelist version — the same CT release stamped into define.xml — so a verbatim value that misses the codelist is tagged and routed for review rather than imputed. Variables that have no standard home go to SUPP-- as name/value pairs keyed back to the parent record, and cross-record relationships (an AE caused by a concomitant medication, say) are recorded in RELREC. The define.xml generation reads the same spec that drove the mapping, so the metadata can never drift from the data it describes.

# 21 CFR Part 11 relevance: define.xml is the reviewer-facing metadata contract;
# it is generated from the SAME spec that drove the mapping, so it cannot drift.
def harmonize_ct(series: pd.Series, codelist: dict[str, str]) -> pd.Series:
    # Miss -> explicit tag routed to query generation; never a silent default.
    return series.map(lambda v: codelist.get(v, f"__UNMAPPED__:{v}") if pd.notna(v) else v)


def build_suppqual(df: pd.DataFrame, domain: str, qnams: dict[str, str]) -> pd.DataFrame:
    """Non-standard variables -> SUPP-- as one QNAM/QVAL row per value."""
    supp = []
    for qnam, qlabel in qnams.items():
        present = df[df[qnam].notna()]
        for _, r in present.iterrows():
            supp.append({
                "STUDYID": r["STUDYID"], "RDOMAIN": domain, "USUBJID": r["USUBJID"],
                "IDVAR": f"{domain}SEQ", "IDVARVAL": r[f"{domain}SEQ"],
                "QNAM": qnam, "QLABEL": qlabel, "QVAL": str(r[qnam]),
                "QORIG": "CRF", "QEVAL": "SPONSOR",
            })
    return pd.DataFrame(supp)

Emitting SUPP-- as explicit QNAM/QVAL/IDVAR/IDVARVAL rows keyed on --SEQ is exactly the structure Pinnacle 21 expects, and it is what lets a reviewer walk from a supplemental qualifier back to its parent record. RELREC is built the same way — a pair of rows naming the two related records by domain, IDVAR, and IDVARVAL — so a concomitant medication given in response to an adverse event links CM to AE without either domain absorbing the other. Per-domain supplemental patterns — multi-select race in SUPPDM, partial-date imputation flags in SUPPAE — are worked through in the domain guides linked above and in ADaM Dataset Derivation for Analysis, which consumes these SDTM datasets.

End-to-end traceability is the property that ties the whole pipeline together: every submission variable must be walkable back to the raw EDC value it came from. The pipeline delivers that by hashing the raw source payload before any transform, stamping the spec_version and ct_version into each row’s lineage metadata, and preserving the source keys (FORM_OID, USUBJID, and the pre---SEQ sort keys) alongside the derived variables. When an FDA reviewer questions a single LBSTRESN value years after lock, that chain — raw hash, spec version, source keys, derivation rule — is what lets the study team reproduce the exact number rather than re-deriving an approximation. Traceability is therefore not a reporting afterthought but a first-class output of every stage, which is why the router refuses to drop records and the mappers tag rather than impute.

Configuration and Parameterization #

The mapping is data, not code. Domain assignment, per-variable source mapping, codelist references, and the pinned CT version live in a version-controlled spec so a clinical data manager can revise a mapping through a reviewed pull request without redeploying the engine — and the spec’s git history becomes change-control evidence. Hardcoding any of this inside the transform is the surest route to an unauditable pipeline.

# config/sdtm_spec.yml — committed; every mapping change is a reviewed diff.
spec_version: "SDTM-2026.2"
sdtmig_version: "3.4"
ct_version: "2026-03-28"          # pinned; must match define.xml exactly
timezone: "UTC"
locale: "C"                       # deterministic date + number parsing
form_to_domain:
  DEMOG: DM
  AE_LOG: AE
  LABRESULT: LB
  VITALS: VS
domains:
  VS:
    class: findings
    seq_sort: [VSTESTCD, VSDTC]
    variables:
      VSORRES:  {source: RESULT_VALUE}
      VSORRESU: {source: RESULT_UNIT, codelist: UNIT}
      VSTESTCD: {source: TEST_CODE, codelist: VSTESTCD}
    supp:
      VSPERF: "Vital Signs Performed"

Secrets and environment-specific endpoints (EDC_API_BASE, PINNACLE21_CLI) map through environment variables so the YAML stays credential-free and safely committable. The ct_version stamped here must equal the version recorded in define.xml; a mismatch is itself a failure the pipeline should raise, because it means the data was coded against a different codelist than the metadata advertises.

Testing and Validation #

GxP expectations require the pipeline to carry its own regression evidence, and SDTM gives that evidence two complementary layers. The first is Pinnacle21-style conformance checks — structural and CT rules asserted as declarative pandera contracts per domain — that prove a frame is submission-shaped before it is ever written. The second is rule-level unit tests: one focused test per derivation rule (USUBJID composition, --SEQ uniqueness, AGE derivation, CT membership) so a broken rule fails in isolation with a legible message. Both run over hashed, synthetic fixtures — never live patient data — and the artifacts are retained as OQ evidence.

# GxP test artifact: per-domain conformance contract + rule-level unit tests,
# archived as OQ evidence. Fixtures are synthetic, hashed, never live data.
import pandas as pd
import pandera as pa
from pandera import Column, Check

VS_CONTRACT = pa.DataFrameSchema(
    {
        "STUDYID":  Column(str, nullable=False),
        "DOMAIN":   Column(str, Check.eq("VS")),
        "USUBJID":  Column(str, nullable=False),
        "VSSEQ":    Column(int, Check.gt(0)),
        "VSTESTCD": Column(str, Check.str_matches(r"^[A-Z0-9]{1,8}$")),
        "VSDTC":    Column(str, Check.str_matches(r"^\d{4}(-\d{2}(-\d{2})?)?"), nullable=True),
    },
    strict="filter",
)


def test_vsseq_is_unique_within_subject(vs_frame):
    dup = vs_frame.duplicated(["USUBJID", "VSSEQ"]).sum()
    assert dup == 0, f"{dup} duplicate USUBJID/VSSEQ pairs — --SEQ not unique"


def test_usubjid_is_stable_across_domains(dm_frame, vs_frame):
    # Every VS subject must resolve to a DM subject: referential integrity.
    orphans = set(vs_frame["USUBJID"]) - set(dm_frame["USUBJID"])
    assert not orphans, f"VS records with no DM subject: {orphans}"

Wire both layers into CI so a non-conforming or non-deterministic change cannot merge, then run the packaged datasets through the Pinnacle 21 CLI as the final gate. Cross-domain referential checks — every VS subject resolving to a DM subject — belong in this layer, mirroring the CDISC ODM vs CDASH Schema Mapping conformance discipline the upstream acquisition layer already applies.

Production Gotchas and Failure Modes #

• CT drift between IG/CT versions. A codelist term that was valid under one CT release is retired or recoded in the next, so a mapping that passed last quarter fails Pinnacle 21 this quarter. Remediation: pin ct_version in the spec, stamp it into define.xml, and fail the run on any mismatch between the two — never let the codelist float.
• Timezone and partial ISO-8601 dates. --DTC values arrive as partial dates (2026-03, 2026) or with inconsistent timezone offsets, and naive parsing either crashes or silently invents a day. Remediation: keep partials as truncated ISO-8601 strings (never zero-fill to -01), pin TZ=UTC, and record any imputation as a flag in SUPP--, not in the --DTC itself.
• --SEQ non-uniqueness. Assigning --SEQ before a stable sort, or across the wrong grouping, produces duplicate USUBJID/--SEQ pairs that Pinnacle 21 rejects outright. Remediation: sort with kind="mergesort" on the spec’s seq_sort keys and assign with groupby.cumcount()+1, then assert uniqueness in CI.
• Silent type coercion. Reading a lab result column as float turns "<0.5" into NaN and a leading-zero subject id into an integer, destroying data. Remediation: read every source column as nullable string (dtype="string"), and coerce to numeric only inside the mapper with explicit handling of non-numeric qualifiers into --STRESC.
• Unmapped verbatim imputed to a default. Defaulting an unknown coded value to the first codelist entry fabricates clinical meaning. Remediation: tag misses __UNMAPPED__ and route them to query generation; never substitute.

Compliance Checklist #

Use this as the change-management gate before promoting an SDTM build to a validated environment:

• Every raw record is routed to exactly one domain by the versioned spec; unrouted forms halt the run (ALCOA+ Attributable).
• Raw source payload is hashed (SHA-256) before any transform, and the hash is retained (Original).
• USUBJID is composed identically across every domain and reconciles to DM (Consistent).
• --SEQ is assigned after a stable sort and is unique within each USUBJID/domain (Complete).
• Controlled terminology is harmonized against a pinned CT version; misses are tagged and queried, never imputed (Accurate).
• ct_version in the spec equals the CT version stamped in define.xml.
• Partial and timezone-bearing --DTC dates are preserved as truncated ISO-8601; imputations recorded in SUPP-- (Accurate).
• SUPP-- and RELREC carry IDVAR/IDVARVAL keys that resolve to parent records.
• define.xml is generated from the same spec as the data and validates against the define-XML schema.
• Pinnacle 21 conformance report is clean (no rejects) and archived as the release artifact (Enduring, Available).
• Per-domain pandera contracts and rule-level unit tests pass in CI.
• Mapping spec changes are reviewed pull requests; the diff history is archived as IQ/OQ evidence.

Related #

• Clinical Data Architecture & EDC Standards — the parent reference architecture this transformation pipeline sits within.
• CDISC ODM vs CDASH Schema Mapping — the acquisition-standard layer that feeds clean, controlled-terminology-aligned input into SDTM.
• Building the SDTM DM Domain from EDC Demographics — the special-purpose Demographics domain, USUBJID and reference dates worked end to end.
• Mapping Adverse Events to the SDTM AE Domain — MedDRA coding, partial AE dates, and the Events-class variable skeleton.
• ADaM Dataset Derivation for Analysis — the analysis datasets these SDTM domains feed, including treatment-emergent flags.
• Pandas DataFrames for Clinical Data Cleaning — the frame-handling and type-coercion discipline the mappers rely on.