Generating Annotated CRF PDFs with Python

The submission-style annotated CRF (aCRF) is the blank case report form with each field’s SDTM variable and codelist written beside it, and reviewers still expect it as a PDF — but hand-annotating a 200-page blank CRF in Acrobat is slow, error-prone, and the moment a protocol amendment moves a field the whole document is stale. This page is the tactical companion to CDISC CRF Annotation for Discrepancy Tracking, and it sits inside the broader Clinical Query Generation & Discrepancy Management section where the annotation map is treated as the single source of truth. It is written for the clinical data engineer who wants to generate the aCRF PDF programmatically: draw an SDTM-variable overlay with reportlab, merge it onto the blank CRF pages with pypdf, and verify that every annotated field landed — so the PDF a regulator reads is produced from the same versioned map that drives SDTM mapping and edit checks, and can never disagree with them.

The Overlay-and-Merge Flow #

The generator never edits the blank CRF; it draws annotations onto a transparent overlay sized to the same page geometry, then merges the two PDFs so the source pages stay pristine and the annotations sit on top.

Root cause: why annotation drifts from the metadata #

A hand-annotated aCRF drifts because it is authored in a different tool than the one that governs the data. The blank CRF is a PDF; the SDTM targets live in an annotation map and the collection semantics live in the CDISC Operational Data Model (ODM). When those three are maintained independently, a protocol amendment that renames a field updates the ODM and the map but leaves the PDF showing the old variable — and no automated check catches it, because a PDF label is opaque text no gate reads. The consequence is not cosmetic: a reviewer who reconciles the aCRF against the define.xml finds a variable in one that the other does not name, and the mismatch becomes a query on your submission rather than on the data. Generating the PDF from the map inverts the dependency: the annotations become a rendering of the same versioned source that drives SDTM mapping and edit checks, so a regenerate is the only way the document changes and it is always current with the map’s catalog_version. The aCRF stops being a document you maintain and becomes a build artifact you produce.

Step-by-step implementation #

Step 1: Load the annotation map and its page-coordinate table #

The annotation map supplies each field’s SDTM domain/variable/codelist; a companion coordinate table supplies where on the blank CRF that field sits (page index and a PDF-point x/y). Keep the coordinates in their own version-controlled file keyed by item_oid, because page geometry changes with the CRF layout, not with the SDTM mapping, and the two should version independently.

# 21 CFR Part 11 relevance: the aCRF is generated from the versioned map, so the
# document is attributable to an exact catalog_version (Attributable, Original).
import json

def load_inputs(map_path: str, coords_path: str) -> tuple[dict, dict, str]:
    doc = json.loads(open(map_path, encoding="utf-8").read())
    annotations = doc["annotations"]                 # keyed by item_oid
    coords = json.loads(open(coords_path, encoding="utf-8").read())  # item_oid -> {page,x,y}
    return annotations, coords, doc["catalog_version"]

Step 2: Resolve every annotation to a page and coordinate #

Join the map to the coordinate table on item_oid and build a per-page work list. Fail loudly on any annotated field that has no coordinate — a missing coordinate means an annotation that would never be drawn, which is exactly the silent-drop defect the aCRF exists to prevent.

# ALCOA+ requirement: Complete. An annotated field with no coordinate is a gap,
# not a skip — surface it before rendering so no SDTM target goes unshown.
from collections import defaultdict

def resolve_placements(annotations: dict, coords: dict) -> dict[int, list[dict]]:
    by_page: dict[int, list[dict]] = defaultdict(list)
    missing = []
    for item_oid, ann in annotations.items():
        pos = coords.get(item_oid)
        if pos is None:
            missing.append(item_oid)
            continue
        label = f"{ann['domain']}.{ann['variable']}"
        if ann.get("codelist"):
            label += f" ({ann['codelist']})"         # show CT codelist inline
        by_page[pos["page"]].append({"x": pos["x"], "y": pos["y"], "label": label})
    if missing:
        raise RuntimeError(f"annotations without coordinates: {sorted(missing)}")
    return by_page

Step 3: Draw the overlay with reportlab #

Create one overlay PDF whose page size exactly matches the blank CRF, and draw each annotation’s label at its resolved coordinate. Matching the page geometry is what keeps the text aligned when the layers merge; a mismatched page box shifts every annotation.

# ALCOA+ requirement: Legible. Draw annotations in a contrasting colour at a
# fixed size so the SDTM variable is unambiguously readable on the merged page.
from reportlab.pdfgen import canvas
from reportlab.lib.colors import HexColor
import io

def build_overlay(by_page: dict[int, list[dict]], page_size: tuple[float, float],
                  n_pages: int) -> io.BytesIO:
    buf = io.BytesIO()
    c = canvas.Canvas(buf, pagesize=page_size)
    c.setFillColor(HexColor("#b91c1c"))              # submission-convention red
    c.setFont("Helvetica-Bold", 8)
    for page_idx in range(n_pages):                  # one canvas page per CRF page
        for a in by_page.get(page_idx, []):
            c.drawString(a["x"], a["y"], a["label"])
        c.showPage()
    c.save()
    buf.seek(0)
    return buf

Step 4: Merge the overlay onto the blank CRF with pypdf #

Read the blank CRF, stamp each page with the matching overlay page, and write the annotated result. pypdf’s merge_page composites the overlay on top without touching the underlying content stream, so the source CRF stays byte-for-byte intact.

# 21 CFR Part 11 relevance: the blank CRF is evidence and is never edited in
# place; the overlay composites on top, preserving the source (Original, Enduring).
from pypdf import PdfReader, PdfWriter

def merge_overlay(blank_crf_path: str, overlay: io.BytesIO, out_path: str) -> int:
    base = PdfReader(blank_crf_path)
    over = PdfReader(overlay)
    writer = PdfWriter()
    for i, page in enumerate(base.pages):
        page.merge_page(over.pages[i])               # overlay sits above source
        writer.add_page(page)
    with open(out_path, "wb") as fh:
        writer.write(fh)
    return len(writer.pages)

Step 5: Verify coverage against the map #

Re-extract the text from the generated PDF and assert that every annotated domain.variable label appears. This is the executable proof the document is complete; archive its report as OQ evidence alongside the map’s catalog_version.

# OQ requirement: documented evidence that every annotation in the map was
# rendered into the final PDF (Complete, Available for inspection).
from pypdf import PdfReader

def verify_coverage(out_path: str, annotations: dict) -> list[str]:
    text = "\n".join(p.extract_text() or "" for p in PdfReader(out_path).pages)
    expected = {f"{a['domain']}.{a['variable']}" for a in annotations.values()}
    missing = sorted(lbl for lbl in expected if lbl not in text)
    if missing:
        raise RuntimeError(f"aCRF coverage gate failed, labels absent: {missing}")
    return sorted(expected)

Verification and audit trail #

The generated aCRF is a regulated artifact, so capture the evidence that lets it be reconstructed and defended at inspection.

Edge cases and vendor-specific gotchas #

Field-to-coordinate mapping breaks when the CRF layout changes. A new blank-CRF build reflows pages, so a coordinate keyed to page 12 now points at the wrong form. Keep coordinates in a file versioned against the CRF layout, and re-run Step 5 after any layout change — a coverage pass is not enough; spot-check that labels land on the correct field, not merely somewhere on the right page.

Codelist annotations need the CT version, not just the codelist id. Showing AESER (C66742) is only unambiguous if the reader knows which CT package C66742 came from. Render the annotation’s ct_version in a page footer or the document cover so the codelist reference resolves to an exact release.

The aCRF must be versioned with the protocol. An amendment that moves or retires a field invalidates the old PDF. Treat regeneration as part of the amendment’s release: bump the map, regenerate, and re-archive, so the aCRF’s catalog_version always matches the protocol version in force — never a stale PDF sitting beside a current map.

Frequently Asked Questions #

Related #

• CDISC CRF Annotation for Discrepancy Tracking — the annotation map this generator renders and the discrepancy logic it drives.
• Mapping EDC Forms to CDASH Standards Step by Step — how the SDTM bindings shown on the aCRF are produced.
• Cross-Form Data Validation Rules — the edit-check engine the annotated rules feed.
• Clinical Query Generation & Discrepancy Management — the parent section this deep-dive sits within.