Inside the REDCap Connector: How a Data Dictionary Becomes a Governed Graph

To trust a governance tool you should be able to predict what it does with your data — row by row, column by column. This deep dive opens the hood on the CoreModels REDCap connector: how the data dictionary CSV is parsed, how each REDCap concept lands in the governed graph, which facts ride vendor metadata rather than governed meaning, what triggers each audit rule, and precisely where the mapping is lossy.

The row, as the parser reads it

Take a small dictionary:

"Variable / Field Name","Form Name","Section Header","Field Type","Field Label","Choices, Calculations, OR Slider Labels","Field Note","Text Validation Type OR Show Slider Number","Text Validation Min","Text Validation Max","Identifier?","Branching Logic (Show field only if...)","Required Field?","Custom Alignment","Question Number (surveys only)","Matrix Group Name","Matrix Ranking?","Field Annotation"
"record_id","demographics","","text","Record ID","","","","","","","","","","","","",""
"first_name","demographics","","text","First Name","","","","","","y","","y","","","","",""
"sex","demographics","","radio","Sex","1, Male | 2, Female","","","","","","","y","","","","",""
"age","demographics","","text","Age","","","integer","","","","","","","","","",""

Parsing starts with the header, and the first design decision shows up immediately: columns are located by header keyword, not by position or exact name — a case-insensitive substring match, so a lightly renamed or reordered export still parses. Only three columns are hard requirements: Variable / Field Name, Form Name, and Field Type. If any is missing, parsing fails with a message that says exactly that, rather than producing a half-right model. The reader itself is RFC 4180-compliant — quoted cells, embedded commas and newlines, doubled quotes — which matters because real choice lists are full of commas.

From there, each row is classified:

• Rows with an empty variable or form name are skipped.
• descriptive rows are display-only text, not data. They're skipped — and counted, so twelve excluded descriptive rows are a recorded number, not a silent one.
• A duplicate variable name within the dictionary produces a declared lossiness record ("Duplicate variable name in the dictionary; the later row was ignored") — the import still succeeds, and tells you what it dropped.

From REDCap concepts to governed concepts

The structural mapping is small enough to state completely:

REDCap	Governed graph
Instrument (form)	Type node, labeled with the form name
Variable	Element on its Type
First variable in the dictionary	The record id — Unique + NotNull checks (REDCap's primary key)
Required Field? = y	NotNull check
Radio/dropdown/checkbox choices	Taxonomy + controlled-list relation
Text validation type	The element's data type (table below)

Each imported node also gets a vendor-identity marker — a mapsTo mixin value with the vendor key as the standard and the REDCap-side identity as the URI — which lets audits match graph nodes back to dictionary rows deterministically and keeps that identity queryable through exports.

Node ids themselves are minted, not copied: CoreModels ids must be camelCase alphanumerics, so visit_date on the follow_up instrument becomes the element id followUpVisitDate, while the REDCap names live on the Label, the mapsTo value, and the metadata. Because separators fold away, two genuinely different REDCap names can collapse onto one id — visit_1 and visit1 both mint visit1. Rather than silently merging two instruments, the import refuses and names the colliding identities so you can rename one in REDCap.

The type mapping is driven by the Text Validation Type column when present, falling back to the field type when not. In full:

Native value	Governed type	Approximated?
integer	Integer	no
number	Double	no
number_1dp, number_2dp	Double	yes — precision spec dropped
calc	Double	yes — a computed value, typed by convention
slider	Double	yes
yesno, truefalse	Boolean	no
date_* (any date/datetime validation)	DateTime	no
notes	String	no
text (or empty)	String	no
anything else	String	yes

"Approximated" isn't a judgment call at read time — it's a flag the mapping layer sets so the difference between "this is genuinely a Double" and "this is a slider we're representing as a Double" is never lost.

Choice lists: labels govern, codes ride along

A REDCap choice list like 1, Male | 2, Female encodes two things: the labels humans see and the codes that appear in exported data. The connector splits the responsibilities. The labels become the governed taxonomy — they are the meaning. The codes are preserved verbatim in vendor metadata as redcap.codes (pipe-joined, in order), so nothing needed for data-level reconciliation is lost. Two parsing details matter:

• Each choice splits on the first comma only, so labels containing commas survive intact.
• If a label repeats within one list, it enters the taxonomy once; if a code repeats, the field is flagged with redcap.duplicateCodes metadata — which is what later fires the audit warning.

What rides the vendor-metadata mixin

Not every dictionary fact belongs in governed meaning. REDCap bookkeeping — real and useful, but not semantics the whole organization shares — rides a per-vendor metadata mixin on the node, refreshed on every import (estate bookkeeping, explicitly exempt from the "import never mutates" rule that protects governed nodes):

Metadata key	Content
redcap.fieldType	The raw REDCap field type (text, radio, dropdown, checkbox, notes, …) — always recorded
redcap.identifier	Present (true) when Identifier? = y — the PHI mark
redcap.note	The Field Note text
redcap.branching	The branching logic expression, preserved verbatim
redcap.codes	The choice codes, pipe-joined, in list order
redcap.duplicateCodes	Present when a choice list reuses a code

The import also writes a state node recording the whole event — a source fingerprint (the first 16 hex characters of a SHA-256 over the exact CSV bytes) plus estate-level facts: the instrument count, field count, identifier-field count, and descriptive-row count. Two governance-relevant numbers — how many PHI fields the project carries, how much display-only content was excluded — are therefore recorded facts rather than things you re-derive by hand.

A data dictionary carries no lineage, so a REDCap import adds zero lineage edges; the response reports the zero rather than hiding the counter.

The REDCap audit rules, precisely

Beyond the vendor-neutral coverage and drift checks (dataset-unmapped, field-unmapped; dataset-removed, field-removed, field-type-drift, enum-constraint-removed, enum-narrowed, enum-widened, contract-drift), the connector contributes three conformance rules, and their trigger conditions are exact:

• phi-fields (Info) — fires once per instrument that contains at least one field with the redcap.identifier mark. Deliberately aggregated: one finding per instrument with the count in the message and the field names in detail, not one finding per field. A project with 40 flagged fields across 5 instruments produces 5 findings, each a complete inventory — human-readable and machine-parseable.
• text-no-validation (Info) — fires for a field whose REDCap field type is text and whose effective native type is still text. A text field with integer validation doesn't fire; a bare text field does. Unvalidated free text is where harmonization projects go to die, and the audit names every instance.
• duplicate-choice-codes (Warning) — fires for any field carrying the redcap.duplicateCodes flag from parsing. This is the only REDCap-specific rule at Warning severity because it has data-level consequences: exported values for that field are genuinely ambiguous.

The round trip: generate

The mapping also runs backwards. From the governed model the connector emits coremodels_data_dictionary.csv — an upload-ready dictionary with the full 18-column REDCap header. The reverse mappings: a governed taxonomy becomes a dropdown with codes minted sequentially (1, Label | 2, Label | …); Integer becomes text with integer validation; Double becomes text with number; Boolean becomes a yesno field; DateTime becomes text with date_ymd; a NotNull check becomes Required Field? = y. Variable and form names are sanitized to REDCap conventions — lowercase, alphanumerics and underscores.

The round trip is honest rather than magical, and its normalizations follow from the mappings above:

• All date validations regenerate as date_ymd. A field imported as date_mdy returns as DateTime, whose canonical REDCap rendering is date_ymd — the type is preserved, the display format is not.
• truefalse regenerates as yesno — both import as Boolean, and Boolean's rendering is yesno.
• Dropdown codes are minted, not read back: generation numbers taxonomy terms sequentially rather than consulting the original codes preserved on redcap.codes. Treat a generated dictionary as a scaffold for a new instrument, not a byte-faithful reconstruction.
• The Identifier? and branching-logic columns are emitted empty. Both facts survive on the governed side as vendor metadata, but generation does not re-emit them — re-apply PHI marks and branching in REDCap before deploying a generated instrument.
• Cross-form references have no REDCap slot at all (a data dictionary cannot express "this field points at that instrument"), so they're carried as plain text fields — a documented limit of the format rather than a hidden one.

Limits, stated plainly

• Columns the parser doesn't read. Section headers, text validation min/max, custom alignment, question numbers, matrix group names, matrix ranking, and field annotations are not captured into the governed model; generated dictionaries emit those columns empty. If your governance needs validation ranges, that is a boundary to know about.
• Additive imports don't mint value sets. A variable grafted onto an already-governed instrument gets its Element, but its choice list does not become a governed Taxonomy — promoting a value set stays a human act. The import declares a ConstraintRelaxation record naming the field.
• Snapshot cap. The parsed snapshot stored at import — what one-call re-audits run against — is capped at roughly 1.5 MB encoded. Oversized dictionaries import and audit fine with fresh artifacts but report snapshotStored: false with a lossiness record, and re-audit stays unavailable.
• No live connection. There is no REDCap API integration and your token never reaches CoreModels; LiveSync is declared but deferred. The design leans into REDCap's schema estate being one stable, portable file.
• Required-ness lives in checks. CoreModels has no schema-level required/nullable flag, so Required Field? = y becomes a NotNull check carried in metadata rather than a structural property — and generation reads it back from exactly there. The information survives the round trip; its representation is a documented convention.

Every one of these behaviors — keyword-matched headers, the first-variable record id, the label/code split, the aggregation of phi-fields — exists so the governed model is predictable from the dictionary and the dictionary is reconstructible from the governed model, with every gap between them written down. That's what we think vendor integration should mean.

Practical setup — extraction, import, audit, CI — is covered in the REDCap quickstart in the CoreModels docs (docs/quickstarts/redcap).