Handling SUPPQUAL Supplemental Qualifiers in SDTM

Every SDTM build reaches the same crossroads: an EDC form carries fields the standard domains have no home for — an “other, specify” free-text box, a local severity scale, a site comment — and they cannot simply be dropped. The supplemental-qualifier model (SUPP--) is where these leftover variables live, keyed back to their parent record through IDVAR and IDVARVAL, and getting that link wrong is one of the most common findings a validator will raise. This guide is the practical companion to the CDISC SDTM Transformation Pipelines guide and sits inside the wider Clinical Data Architecture & EDC Standards reference, and it is written for clinical data managers and Python ETL engineers who need a deterministic, defensible procedure for moving non-standard EDC fields into a conformant SUPP-- dataset that links cleanly back to --SEQ. Each step isolates one responsibility — decide, build, link, split, and validate — with its regulatory rationale alongside the code.

The SUPP-- Build and Link Flow #

The pipeline decides which leftover fields belong in a supplemental qualifier, reshapes them into the SUPP-- structure, links each back to its parent --SEQ through IDVAR/IDVARVAL, splits over-length values, and gates against the validator ruleset.

Root Cause: Why Non-Standard Fields Need a Separate Model #

SDTM domains have a fixed variable list, and a study’s EDC almost always collects more than those variables can hold. Rather than inventing non-standard columns inside a domain — which breaks conformance — the standard provides the supplemental-qualifier model: a separate SUPP-- dataset that hangs name/value pairs off a parent record. The parent link is the whole point and the whole difficulty. A SUPP-- row does not repeat the parent’s keys inline; instead it names the parent’s identifying variable in IDVAR (almost always the domain sequence number, --SEQ, such as AESEQ or LBSEQ) and stores that variable’s value in IDVARVAL. Get IDVAR wrong — point it at USUBJID or a variable that is not unique within the subject-domain — and the qualifier attaches to the wrong record or to many records at once, which a validator flags immediately. On top of that, QVAL is capped at 200 characters, so a long free-text comment must be split across sequential records, and QORIG (the origin of the value) draws from controlled values. The model is deterministic precisely because a mislinked qualifier is a silent, subject-level data-integrity defect.

Step-by-Step Implementation #

Step 1: Triage Leftover Fields — SUPP-- or Core Variable #

Before building anything, decide where each unmapped EDC field belongs. A field that corresponds to an existing SDTM variable — even under a different name — goes in the core domain, not SUPP--. Only genuinely non-standard data (an “other, specify” text, a protocol-specific flag, a sponsor annotation) becomes a supplemental qualifier. Record the decision so it is reviewable.

# ALCOA+ (Consistent): the SUPP-- vs core-variable decision is table-driven
# and reviewable, so no non-standard field is parked or promoted ad hoc.
CORE_TARGET = {          # fields that actually belong in a core SDTM variable
    "AE_SERIOUS_FLAG": ("AE", "AESER"),
    "AE_OUTCOME":      ("AE", "AEOUT"),
}
SUPP_FIELDS = {          # genuinely non-standard -> supplemental qualifier
    "AE_OTHER_SPEC":   {"QNAM": "AEOTHSP", "QLABEL": "Other Action Taken, Specify", "QORIG": "CRF"},
    "AE_LOCAL_GRADE":  {"QNAM": "AELGRADE", "QLABEL": "Local Severity Grade",        "QORIG": "CRF"},
}

def triage(field: str):
    if field in CORE_TARGET:
        return ("CORE", *CORE_TARGET[field])   # route to the core domain build
    if field in SUPP_FIELDS:
        return ("SUPP", SUPP_FIELDS[field])
    raise KeyError(f"Unclassified EDC field: {field}")  # never silently drop

An unclassified field is a hard stop and becomes a query in the discrepancy workflow described in Automated Clinical Query Generation, so nothing is lost by omission.

Step 2: Build the SUPP-- Structure #

Reshape each chosen field into the fixed SUPP-- variable set. The dataset name is SUPP plus the parent domain (SUPPAE, SUPPLB), RDOMAIN names the parent domain, and each row carries USUBJID, the QNAM short name, its human-readable QLABEL, the QVAL value, the QORIG origin, and optionally QEVAL (the evaluator, for assessor-derived values).

# 21 CFR Part 11 (Attributable): each SUPP-- row records the origin (QORIG)
# and, where applicable, the evaluator (QEVAL) of the qualifier value.
import pandas as pd

def build_supp(parent: pd.DataFrame, field: str, spec: dict, rdomain: str) -> pd.DataFrame:
    rows = parent[parent[field].astype(bool)]          # only records that carry a value
    return pd.DataFrame({
        "STUDYID":  rows["STUDYID"].values,
        "RDOMAIN":  rdomain,                            # parent domain, e.g. "AE"
        "USUBJID":  rows["USUBJID"].values,
        "QNAM":     spec["QNAM"],
        "QLABEL":   spec["QLABEL"],
        "QVAL":     rows[field].astype(str).values,
        "QORIG":    spec["QORIG"],                      # controlled: CRF, DERIVED, ASSIGNED, ...
        "QEVAL":    spec.get("QEVAL"),                  # e.g. INVESTIGATOR when assessor-derived
    })

Step 3: Link Each Qualifier Back to Its Parent --SEQ #

This is the step that makes or breaks the dataset. Set IDVAR to the parent domain’s sequence variable (AESEQ, LBSEQ) and IDVARVAL to that record’s sequence value, so each SUPP-- row points at exactly one parent record. Join on the natural key the parent record already has, then carry --SEQ across — never guess or reuse a row index.

# ALCOA+ (Accurate): IDVAR names the parent --SEQ and IDVARVAL carries its
# value, so every qualifier resolves to exactly one parent record.
def link_to_parent(supp: pd.DataFrame, parent: pd.DataFrame, seq_var: str) -> pd.DataFrame:
    # `parent` already holds AESEQ/LBSEQ unique within USUBJID + RDOMAIN
    keyed = parent[["USUBJID", seq_var, "_natural_key"]]
    supp = supp.merge(keyed, on=["USUBJID", "_natural_key"], how="left", validate="m:1")
    if supp[seq_var].isna().any():
        raise ValueError("Unresolved parent --SEQ; qualifier would orphan")  # fail loud
    supp["IDVAR"] = seq_var                            # e.g. "AESEQ"
    supp["IDVARVAL"] = supp[seq_var].astype(int).astype(str)
    return supp.drop(columns=[seq_var, "_natural_key"])

The validate="m:1" guard is deliberate: many qualifiers may attach to one parent, but a qualifier must never fan out to many parents. If it does, the merge raises rather than silently duplicating.

Step 4: Split QVAL at the 200-Character Limit #

QVAL is capped at 200 characters. A longer free-text value must be split across sequential SUPP-- records that share the same QNAM and parent link, distinguished only by ordering, so the full text is reconstructable in order without truncation.

# ALCOA+ (Complete): over-length QVAL is split across sequential records,
# never truncated, so the full free-text value survives to submission.
QVAL_MAX = 200

def split_long_qval(supp: pd.DataFrame) -> pd.DataFrame:
    out = []
    for _, row in supp.iterrows():
        text = row["QVAL"]
        if len(text) <= QVAL_MAX:
            out.append(row.to_dict())
            continue
        chunks = [text[i:i + QVAL_MAX] for i in range(0, len(text), QVAL_MAX)]
        for n, chunk in enumerate(chunks, start=1):
            piece = row.to_dict()
            piece["QVAL"] = chunk
            piece["QNAM"] = f"{row['QNAM']}{n}"         # e.g. AECOM1, AECOM2 per split convention
            out.append(piece)
    return pd.DataFrame(out)

Step 5: Gate Against the Validator Ruleset #

Assert the SUPP-- structure and its parent linkage with a schema contract that mirrors the Pinnacle 21 SUPP rules, and archive the report as OQ evidence. The critical checks: IDVAR names a real sequence variable, every IDVARVAL resolves to a parent record, QORIG is a controlled value, and no QVAL exceeds the limit.

# GxP test artifact: the pandera contract mirrors the Pinnacle 21 SUPP rules;
# its pass/fail report is archived as OQ evidence for the run.
import pandera.pandas as pa

supp_schema = pa.DataFrameSchema({
    "RDOMAIN":  pa.Column(str, pa.Check.str_matches(r"^[A-Z]{2}$")),
    "IDVAR":    pa.Column(str, pa.Check.str_matches(r"^[A-Z]{2}SEQ$")),   # must name --SEQ
    "IDVARVAL": pa.Column(str, pa.Check.str_matches(r"^\d+$")),
    "QNAM":     pa.Column(str, pa.Check.str_length(1, 8)),
    "QVAL":     pa.Column(str, pa.Check.str_length(1, 200)),              # 200-char cap
    "QORIG":    pa.Column(str, pa.Check.isin(["CRF", "DERIVED", "ASSIGNED", "PROTOCOL", "EDT"])),
})

def gate(df):
    return supp_schema.validate(df, lazy=True)

The same conformance-gate pattern anchors every SDTM domain in this guide and is generalized in Validation and CSV Frameworks; where a qualifier expresses a relationship between two domains rather than a value, model it with RELREC instead of forcing it into SUPP--.

Verification and Audit Trail #

Confirming a SUPP-- build is itself a regulated activity. Capture the following evidence on every run so the transformation can be reconstructed and defended at inspection:

Run the build twice on identical parent input and assert byte-identical output with pandas.testing.assert_frame_equal; a stable result is your determinism proof for the audit file.

Edge Cases and Vendor-Specific Gotchas #

IDVAR must reference --SEQ, not the subject. The most common SUPP finding is an IDVAR pointing at USUBJID or a non-unique variable. USUBJID is not unique within a domain, so the qualifier attaches to every record for that subject rather than one. Always set IDVAR to the parent sequence variable (AESEQ, LBSEQ) and confirm each IDVARVAL resolves to a single parent row before promotion.

QVAL cannot exceed 200 characters, so split rather than truncate. A long investigator comment silently truncated at 200 characters loses clinical content and is a data-integrity finding. Split the text across sequential records sharing the parent link, following the study’s split-naming convention, so the full value reconstructs in order. Never let the transform drop the overflow.

QORIG and QEVAL draw from controlled values. QORIG describes where the value came from (CRF, DERIVED, ASSIGNED, PROTOCOL) and must use a controlled term, not free text; a derived qualifier marked CRF misrepresents provenance. Populate QEVAL only when an assessor produced the value — for example INVESTIGATOR — and leave it blank for collected data, so the origin story stays accurate.

Frequently Asked Questions #

Related #

• CDISC SDTM Transformation Pipelines — the parent guide covering the shared SDTM transformation architecture this SUPP-- build follows.
• Clinical Data Architecture & EDC Standards — the reference architecture this qualifier-handling layer sits within.
• Deriving the SDTM LB Domain from Central Lab Feeds — a sibling build whose leftover local lab fields land in SUPPLB.
• Mapping Vital Signs to the SDTM VS Domain — a sibling build whose non-standard vitals annotations land in SUPPVS.
• Validation and CSV Frameworks — the conformance-gate patterns and Pinnacle 21 rules this build validates against.