ADaM Dataset Derivation for Statistical Analysis

The Analysis Data Model (ADaM) is where clinical data engineering stops being about faithful representation and starts being about analysis: a statistician must be able to run a single procedure against an ADaM dataset and reproduce a number in the clinical study report, and a reviewer must be able to trace that number backward — variable by variable, row by row — to the tabulated Study Data Tabulation Model (SDTM) record and, through it, to the original case report form entry. This page solves the core derivation problem that sits between those two demands: how to build ADaM datasets that are simultaneously analysis-ready and traceable, so that derived analysis flags, treatment assignments, and change-from-baseline values never sever the audit chain that regulators require. It is part of the Clinical Data Architecture & EDC Standards reference design, and it consumes the tabulated output of the CDISC SDTM Transformation Pipelines as its sole input. Every derivation described here is metadata-driven and one proc away from a result, because that is the standard an FDA statistical reviewer applies when they open your define-XML.

ADaM Derivation at a Glance #

The derivation resolves standardized SDTM domains into a subject-level spine, fans that spine out into structured analysis datasets for measurements and events, and drives the tables, listings, and figures that populate the submission — with a traceability annotation carried back to every source record.

Concepts and Prerequisites #

ADaM is defined by four principles that constrain every line of derivation code, and violating any one of them is a define-XML finding rather than a style preference. Traceability to SDTM means a reviewer can follow any analysis value back to the tabulation record it derives from, which is why derived rows carry SRCDOM, SRCVAR, and SRCSEQ. Analysis-ready means the dataset needs no further manipulation before a statistical procedure runs against it — populations, treatment assignments, and derived endpoints are already present as columns. Metadata-driven means the derivation is described by an analysis-metadata specification (the define-XML-style document) that a human reviews, not buried in imperative code. And one proc away from results means a PROC (or its pandas equivalent, a single groupby/aggregation) reproduces a reported number without a preparatory data step. The CDISC ADaM standard codifies these, and the FDA Study Data Technical Conformance Guide makes them a submission expectation.

These four principles are not independent — traceability is what makes the other three defensible. A dataset can be analysis-ready and metadata-driven and still be worthless to a reviewer if they cannot see how a derived endpoint relates to the collected data, because at that point the number is an assertion rather than a reconstruction. This is the deep difference between SDTM and ADaM: SDTM is a faithful, minimally-derived tabulation of what was collected, whereas ADaM is permitted — expected — to carry derived and imputed values, and the price of that latitude is that every derivation must announce itself. A last-observation-carried-forward value, a treatment-emergent classification, a windowed visit average: each is legitimate, but each must be visibly derived, stamped with the metadata that lets a second programmer regenerate it from the same SDTM input. That is why the lineage variables and the DTYPE marker recur throughout this page; they are the mechanism that keeps ADaM’s analytical freedom compatible with regulatory reconstructability.

Two structures do almost all the work. The Subject-Level Analysis Dataset (ADSL) is a one-record-per-subject spine holding treatment variables and population flags; every other ADaM dataset merges to it by USUBJID. The Basic Data Structure (BDS) is a vertical, one-record-per-parameter-per-timepoint structure for measurements, keyed by PARAMCD. Events use the Occurrence Data Structure (OCCDS), one record per occurrence. The relevant variable families are worth fixing before any code:

The implementation assumes a version-pinned runtime committed to a lockfile, so a derivation re-run years later reproduces the same numbers:

Building the ADSL Spine #

ADSL is derived first because everything else depends on it. The pattern merges SDTM DM (demographics) with EX (exposure) and DS (disposition), then computes the treatment variables and population flags that downstream datasets carry by reference. The cardinal rule is one row per subject: any merge that can fan out — exposure has many records per subject — must be aggregated to a single value before it touches the spine, or the one-record-per-subject invariant breaks silently.

# ALCOA+ requirement: Attributable + Accurate — ADSL is the analysis spine; every
# subject appears exactly once, and planned vs actual treatment are kept distinct.
import pandas as pd


def derive_adsl(dm: pd.DataFrame, ex: pd.DataFrame, ds: pd.DataFrame) -> pd.DataFrame:
    # First and last dosing dates collapse many EX rows to one per subject.
    ex = ex.sort_values(["USUBJID", "EXSTDTC"], kind="mergesort")
    exposure = ex.groupby("USUBJID", as_index=False).agg(
        TRTSDT=("EXSTDTC", "first"),          # first exposure = treatment start
        TRTEDT=("EXENDTC", "last"),           # last exposure = treatment end
        TRT01A=("EXTRT", "first"),            # ACTUAL treatment, from what was given
    )
    adsl = dm.merge(exposure, on="USUBJID", how="left")
    adsl["TRT01P"] = adsl["ARM"]              # PLANNED treatment, from randomization

    # Population flags are additive booleans rendered as Y/N, never overlapping columns.
    dosed = adsl["TRTSDT"].notna()
    adsl["SAFFL"] = pd.Series(dosed).map({True: "Y", False: "N"})           # any dose
    adsl["RANDFL"] = adsl["ARM"].notna().map({True: "Y", False: "N"})        # randomized
    adsl["ITTFL"] = adsl["RANDFL"]                                          # ITT = randomized
    adsl["TRT01PN"] = adsl["TRT01P"].map({"Placebo": 0, "Drug 50mg": 1, "Drug 100mg": 2})
    return adsl.drop_duplicates("USUBJID").reset_index(drop=True)

The distinction between TRT01P (planned, sourced from the randomized ARM) and TRT01A (actual, sourced from what EX records were actually given) is not cosmetic: safety analyses run on actual treatment, efficacy on planned, and conflating them is one of the most consequential derivation errors in a submission. Note also the numeric companion TRT01PN: statistical procedures order treatment groups by a numeric code rather than by an alphabetic string, so every character analysis variable that participates in a by grouping needs a paired ...N numeric version with a pinned decode. The population flags follow the same discipline — they are rendered as Y/N character values but must be internally coherent, since a subject flagged into the efficacy population who is not in the safety population is a contradiction the conformance checks should reject. The full subject-level derivation — population exclusions, demographic groupings, and stratification factors — is detailed in Deriving the ADSL Subject-Level Analysis Dataset.

BDS Derivation: Baseline, Change, and DTYPE #

The Basic Data Structure holds one row per parameter per timepoint. Deriving it involves three coupled operations: flag the baseline record with ABLFL, carry that baseline value onto every post-baseline row as BASE, and compute change (CHG) and percent change (PCHG). Where an analysis needs a value that was never observed — a last-observation-carried-forward imputation or a visit average — a new row is inserted and stamped with DTYPE so a reviewer can tell derived rows from collected ones. Silently mixing observed and imputed values in the same undifferentiated column is a data-integrity finding.

# ALCOA+ requirement: Original + Complete — observed and derived rows are distinct;
# DTYPE marks any row that was computed rather than collected, preserving traceability.
import numpy as np
import pandas as pd


def derive_bds_baseline(adlb: pd.DataFrame) -> pd.DataFrame:
    adlb = adlb.sort_values(["USUBJID", "PARAMCD", "ADT"], kind="mergesort").copy()
    adlb["DTYPE"] = pd.NA                      # null DTYPE = an observed, collected row

    # ABLFL: the last non-missing pre-treatment record per subject-parameter.
    pre = adlb["ADT"] < adlb["TRTSDT"]
    last_pre = (adlb[pre & adlb["AVAL"].notna()]
                .groupby(["USUBJID", "PARAMCD"]).tail(1).index)
    adlb["ABLFL"] = np.where(adlb.index.isin(last_pre), "Y", pd.NA)

    # BASE broadcasts the flagged baseline AVAL across all rows of the group.
    base = (adlb[adlb["ABLFL"].eq("Y")]
            .set_index(["USUBJID", "PARAMCD"])["AVAL"].rename("BASE"))
    adlb = adlb.merge(base, on=["USUBJID", "PARAMCD"], how="left")

    post = adlb["ADT"] >= adlb["TRTSDT"]
    adlb["CHG"] = np.where(post, adlb["AVAL"] - adlb["BASE"], np.nan)
    adlb["PCHG"] = np.where(post & adlb["BASE"].ne(0),
                            100 * (adlb["AVAL"] - adlb["BASE"]) / adlb["BASE"], np.nan)
    return adlb

Change-from-baseline is only ever computed for post-baseline rows, and PCHG guards against a zero baseline rather than emitting an inf. The parameter grain — PARAM as human-readable label, PARAMCD as the short analysis key, AVAL numeric and AVALC character — must stay one-to-one; a PARAMCD that maps to two different PARAM strings will not pass conformance. The BDS vertical shape is what makes this structure so flexible: rather than one wide column per lab test, every measurement is a row identified by its PARAMCD, so a single analysis dataset can hold chemistry, hematology, and derived ratios side by side, and a new parameter is added as data rather than as a schema change. The timing variables complete the grain — ADT as the analysis date, ADY as the study day relative to treatment start, and the AVISIT/AVISITN pair that windows nominal visits so an early or late collection still maps to the protocol-scheduled timepoint it belongs to. When a value must be carried or averaged into a visit that was missed, the derived row is inserted with the appropriate DTYPE, and the character/numeric visit pair keeps it sortable alongside the observed rows without pretending it was collected.

OCCDS Derivation and Traceability Variables #

Events do not have a value to measure, so they use the Occurrence Data Structure: one row per occurrence, merged to ADSL for treatment context and stamped with occurrence flags. The adverse-event dataset ADAE is the canonical case — it merges SDTM AE to ADSL, derives the treatment-emergent flag from the AE start date against the treatment start date, and flags first occurrences (AOCCFL, AOCCPFL). Crucially, every ADaM row in either structure carries its lineage variables so the back-link in the diagram is real, not aspirational.

# ALCOA+ requirement: Traceable — every derived row names the SDTM domain, variable,
# and sequence number it came from, so a reviewer can reconstruct it from tabulation.
import pandas as pd


def attach_lineage(df: pd.DataFrame, src_domain: str,
                   src_var: str, seq_col: str = "AESEQ") -> pd.DataFrame:
    df = df.copy()
    df["SRCDOM"] = src_domain          # e.g. "AE" — the source SDTM domain
    df["SRCVAR"] = src_var             # e.g. "AETERM" — the source variable
    df["SRCSEQ"] = df[seq_col]         # the source record's --SEQ, an exact pointer
    return df


def merge_to_adsl(occ: pd.DataFrame, adsl: pd.DataFrame) -> pd.DataFrame:
    # USUBJID is the only safe merge key; carry treatment and dates from the spine.
    carry = ["USUBJID", "TRT01A", "TRT01P", "TRTSDT", "TRTEDT", "SAFFL"]
    merged = occ.merge(adsl[carry], on="USUBJID", how="inner", validate="many_to_one")
    merged["TRTA"] = merged["TRT01A"]          # actual treatment for safety analysis
    merged["TRTP"] = merged["TRT01P"]
    return merged

The validate="many_to_one" on the merge is a guardrail, not decoration: it raises if ADSL somehow carries a duplicate USUBJID, catching a broken spine at the point of use rather than letting one subject’s events multiply. The OCCDS structure differs from BDS in a way that shapes the whole derivation: there is no AVAL, because an adverse event is not a measurement with a value — it is an occurrence classified against the treatment timeline. What BDS spends on baseline and change, OCCDS spends on emergence and occurrence flags: was the event treatment-emergent, was it the subject’s first occurrence, the first of its preferred term. The lineage variables carry through identically in both structures, so a reviewer opening ADAE can point at any treatment-emergent flag and follow SRCDOM/SRCVAR/SRCSEQ back to the exact AE record and its --SEQ. The full treatment-emergent logic, partial-date imputation, and occurrence flags live in Deriving the ADAE Adverse-Event Analysis Dataset.

Configuration and Parameterization #

ADaM derivation is metadata-driven by mandate, so the analysis rules — parameter definitions, population criteria, treatment decode, and imputation policy — live in a version-controlled analysis-metadata specification, the machine-readable ancestor of the define-XML that ships with the submission. A biostatistician revises a derivation through a reviewed pull request; the engine reads the spec and never hardcodes an endpoint.

# config/adam_adlb.yml — the analysis-metadata spec; every change is a reviewed diff
# and its git history is part of the define-XML change-control evidence.
dataset: ADLB
structure: BDS
source_domain: LB
key: [USUBJID, PARAMCD, AVISITN]
parameters:                       # PARAMCD -> PARAM, one-to-one and pinned
  ALT:  "Alanine Aminotransferase (U/L)"
  AST:  "Aspartate Aminotransferase (U/L)"
  CREAT: "Creatinine (umol/L)"
baseline:
  flag: ABLFL
  definition: "last non-missing record with ADT < TRTSDT"
derivations:
  - target: CHG
    dtype: null                   # observed, not imputed
  - target: AVAL
    dtype: LOCF                   # imputed rows are stamped, never silent
    method: "last observation carried forward within AVISIT window"
populations:
  analysis_flag: SAFFL            # which ADSL flag subsets this analysis

Environment-specific paths (SDTM_INPUT_PATH, DEFINE_XML_OUT) map through environment variables so the YAML holds no host detail and commits cleanly. The dataset and its metadata version must match the version stamped into the define-XML the run produces; a mismatch means data was derived under a spec different from the one under review, and the pipeline should raise on it.

Testing and Validation #

The regulated norm for ADaM is independent double-programming: a second programmer, working only from the analysis-metadata spec, produces the dataset independently, and the two outputs are compared for equality. A difference is a defect in one program or an ambiguity in the spec — either way it is resolved before the number reaches a table. Alongside that, dataset-level conformance checks assert the structural invariants that make a dataset “ADaM” at all.

# GxP test artifact: double-programming comparison + structural conformance,
# both retained as OQ evidence that the derivation is reproducible and correct.
import pandas as pd
import pandera as pa
from pandera import Column, Check

ADSL_CONTRACT = pa.DataFrameSchema(
    {
        "USUBJID": Column(str, nullable=False, unique=True),   # one row per subject
        "TRT01P": Column(str, nullable=False),
        "TRT01A": Column(str, nullable=True),
        "SAFFL":  Column(str, Check.isin(["Y", "N"])),
        "ITTFL":  Column(str, Check.isin(["Y", "N"])),
    },
    strict="filter",
)


def test_adsl_conforms(adsl):
    ADSL_CONTRACT.validate(adsl, lazy=True)          # collect all violations at once


def test_double_programming_agreement(adsl_primary, adsl_qc):
    # Independent derivation must match the production derivation exactly.
    pd.testing.assert_frame_equal(
        adsl_primary.sort_index(axis=1), adsl_qc.sort_index(axis=1)
    )

Wire both into CI so a non-conforming or non-reproducing derivation cannot merge. The same tabulation discipline that produces the SDTM inputs is documented in CDISC SDTM Transformation Pipelines; keeping the contracts aligned end to end means a fixture that passes tabulation stays valid through analysis.

Production Gotchas and Failure Modes #

• Planned vs actual treatment conflated. Using one treatment column for both safety and efficacy analyses silently mis-assigns subjects who were randomized to one arm but dosed with another. Remediation: derive TRT01P from ARM and TRT01A from EX as separate columns, and pick the correct one per analysis population.
• Imputation hidden in observed rows. Overwriting a missing AVAL with an LOCF value in place makes an imputed number indistinguishable from a collected one. Remediation: insert a new row and stamp DTYPE (e.g. LOCF, AVERAGE); a null DTYPE must always mean observed.
• ADSL merge fan-out. Merging an un-aggregated EX or LB frame onto ADSL multiplies subject rows and inflates population counts. Remediation: aggregate to one row per subject before the merge and assert validate="many_to_one"/uniqueness on USUBJID.
• Broken baseline flag. Flagging more than one ABLFL="Y" per subject-parameter double-counts the baseline and corrupts every CHG. Remediation: select the single last non-missing pre-treatment record, and assert at most one flag per USUBJID/PARAMCD group.
• Severed lineage. Dropping SRCDOM/SRCVAR/SRCSEQ during a reshape leaves derived rows with no path back to SDTM. Remediation: carry lineage columns through every transform and assert their non-null presence in the conformance contract.

Compliance Checklist #

Use this as the change-management gate before promoting an ADaM derivation to a validated environment:

• ADSL holds exactly one record per USUBJID; uniqueness is asserted in CI (Complete).
• TRT01P (planned) and TRT01A (actual) are derived from distinct sources and never conflated (Accurate).
• Population flags SAFFL/ITTFL/EFFFL/RANDFL are Y/N and internally consistent (Consistent).
• Exactly one ABLFL="Y" per subject-parameter; BASE, CHG, PCHG derive only for post-baseline rows.
• Every imputed or derived row carries a non-null DTYPE; observed rows carry a null DTYPE (Original).
• Every ADaM row carries SRCDOM, SRCVAR, and SRCSEQ back to its SDTM source (Attributable, Traceable).
• The analysis-metadata spec version matches the version stamped in the produced define-XML.
• Independent double-programming produces a byte-identical dataset; the comparison is archived (Enduring, Available).
• pandera structural conformance and double-programming tests both pass in CI before merge.
• Imputation policy is documented in the spec, reviewed by pull request, and reflected in DTYPE.

Related #

• Clinical Data Architecture & EDC Standards — the parent reference architecture this analysis layer sits within.
• CDISC SDTM Transformation Pipelines — the tabulation stage that produces the SDTM domains ADaM derives from.
• Deriving the ADSL Subject-Level Analysis Dataset — the full subject-spine derivation with populations and stratification.
• Deriving the ADAE Adverse-Event Analysis Dataset — the OCCDS build with treatment-emergent and occurrence flags.
• CDISC ODM vs CDASH Schema Mapping — the upstream acquisition-standard contract that feeds tabulation.