REDCap to CoreModels: Zero to First Audit with One CSV

Most schema estates take real work to extract: information-schema queries, catalog exports, manifest builds. A REDCap project is the pleasant exception — the entire schema estate is one stable file, the data dictionary CSV, and you almost certainly already know how to download it. That makes the distance from "nothing governed" to "first drift audit" unusually short. This tutorial walks the whole distance: export the dictionary, import it into a governed CoreModels project, run your first audit, and read what comes back. No REDCap credentials are ever shared with CoreModels at any point.

What you need

• A CoreModels project and its 32-character hex project id. We'll call it $PROJECT_ID.
• A CoreModels login token, $TOKEN. Import requires the Admin role on the project; the audit only needs Viewer.
• Your CoreModels API base URL. We'll use https://coremodels.example.com throughout — substitute your own host.
• Access to your REDCap project (either the web UI or an API token you keep on your own machine).

Step 1 — Export the data dictionary

The connector needs exactly one artifact, named data_dictionary. There are two ways to produce it.

From the REDCap UI: Project Setup → Data Dictionary → Download the current Data Dictionary. Save the CSV.

From the REDCap API, run locally — your REDCap token never leaves your machine and never reaches CoreModels:

curl -s -X POST "https://your-redcap.example.edu/api/" \
  -d token=$REDCAP_TOKEN -d content=metadata -d format=csv > data_dictionary.csv

Either route gives you the same file. The parser matches columns by header keyword rather than exact position, so lightly renamed exports still parse, and it reads RFC 4180 CSV properly — quoted choice lists with embedded commas are handled. Three columns must be present for the file to be recognized as a REDCap dictionary at all: Variable / Field Name, Form Name, and Field Type.

Step 2 — Import it into the governed graph

The import call is a POST with the CSV contents inlined as a JSON string. The cleanest way to build that body is jq --rawfile, which handles all the escaping:

jq -n --rawfile dd data_dictionary.csv \
  '{artifacts: {data_dictionary: $dd}}' > import-request.json

curl -sS -X POST \
  "https://coremodels.example.com/graph/integrations/redcap/import/$PROJECT_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  --data-binary @import-request.json

Import is additive: it creates governed nodes for what it finds and never mutates or deletes nodes that already exist. You can re-run it safely; already-governed instruments are skipped and counted, not overwritten. Applying a meaning change to the governed model is deliberately a human act — drift between the estate and the model is the audit's job, which is exactly what we set up next.

What the import writes, in CoreModels vocabulary:

• Each instrument (form) becomes a governed Type.
• Each variable becomes an Element on its Type, with the text validation type mapped to a real data type — integer → Integer, number → Double, date_* → DateTime, yesno/truefalse → Boolean, calc → Double.
• Radio, dropdown, and checkbox choices like 1, Male | 2, Female become Taxonomies. The human labels govern; the numeric codes ride along as redcap.codes metadata so nothing is lost.
• Required Field? = y becomes a NotNull check; the first variable in the dictionary — REDCap's record id — is recognized as the instrument's identity and gets Unique + NotNull.
• Every field marked Identifier? = y (REDCap's PHI flag) is inventoried as redcap.identifier metadata. Branching logic is preserved as metadata too.
• Purely presentational descriptive rows are skipped — and counted, so you can verify nothing silently vanished.

Reading the import response

{ "success": true, "vendor": "redcap", "projectName": "redcap-project",
  "datasetsAdded": 3, "datasetsSkippedExisting": 0, "fieldsAdded": 0,
  "lineageEdgesAdded": 0, "lineageEdgesSkipped": 0, "nodesEnriched": 67,
  "snapshotStored": true, "lossiness": [], "errors": [] }

Read those counters precisely, because one of them is easy to misread. datasetsAdded is the number of instruments newly governed — three here. fieldsAdded is not your variable count: it counts variables added to instruments that were already governed, so on a first import it is legitimately 0 — the variables of a brand-new instrument are written together with their Type. The counter that reflects the whole estate is nodesEnriched: one write per instrument plus one per variable, so 3 + 64 = 67. lineageEdgesAdded stays 0 for REDCap; a data dictionary carries no lineage. Two more fields deserve attention:

• lossiness is the honesty channel. A successful import can still report exactly what it approximated or dropped — for example, a duplicate variable name in the dictionary is recorded here (the later row is ignored). Empty means a clean, faithful import.
• snapshotStored: true means the parsed dictionary was persisted server-side. That snapshot is what powers one-call re-audits later, so you'll want it to be true. Very large dictionaries that exceed the storage cap (~1.5 MB encoded) report false with a lossiness record explaining it — fresh-artifact audits still work fine.

Step 3 — Run your first audit

The audit compares a data dictionary against the governed model and reports three sections: Coverage (what is and isn't governed), Drift (what changed since governance), and Conformance (research-data hygiene rules specific to REDCap). It is strictly read-only and needs only Viewer role. Setting recordHistory: true additionally appends this run to the project's rolling audit trail — useful from day one, because it seeds the status badge:

jq -n --rawfile dd data_dictionary.csv \
  '{artifacts: {data_dictionary: $dd}, recordHistory: true}' | \
curl -sS -X POST \
  "https://coremodels.example.com/graph/integrations/redcap/audit/$PROJECT_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  --data-binary @- | jq '{errorCount, warningCount, infoCount, codes}'

Since you just imported this exact dictionary, expect zero drift. What you will likely see immediately are the REDCap conformance findings:

{ "section": "Conformance", "severity": "Info", "code": "phi-fields",
  "subject": "demographics",
  "message": "Instrument carries 3 identifier (PHI) field(s) — inventory them before any export or agent access.",
  "detail": "first_name, last_name, mrn" }

phi-fields is the aggregated PHI inventory — one finding per instrument, naming the flagged fields. It's informational, and it's arguably the single most useful thing a first audit gives a research data manager: a machine-readable list of exactly which variables carry identifiers, per instrument, derivable by any downstream policy check. Two more REDCap-specific rules may appear: text-no-validation (Info — a free-text field with no validation type, which resists harmonization) and duplicate-choice-codes (Warning — a choice list reuses a code, making exported data ambiguous).

The response also carries three counters — errorCount, warningCount, infoCount — plus a findings array and a markdown field holding a human-readable report you can paste straight into a ticket or PR comment. The rule that matters most going forward: errorCount > 0 means the dictionary violates governed meaning. That is the fail signal you'll later wire into CI.

Step 4 — The badge and the status check

Because you recorded the audit run, the project now has a live status badge — a self-contained SVG reflecting the latest recorded run:

curl -sS -H "Authorization: Bearer $TOKEN" \
  "https://coremodels.example.com/graph/integrations/redcap/badge/$PROJECT_ID" \
  > redcap-audit.svg

Green means clean, yellow means warnings only, red means errors, gray means no recorded runs yet. And any time you want to confirm what's governed, the status endpoint returns the last-import state and the governed dataset count:

curl -sS -H "Authorization: Bearer $TOKEN" \
  "https://coremodels.example.com/graph/integrations/redcap/status/$PROJECT_ID"

Where this goes next

You now have a governed model of your REDCap project and a repeatable, read-only check against it. From here the loop tightens in three directions: wire the audit into CI so every dictionary change is gated before it ships; use the one-call reaudit endpoint to check the stored dictionary snapshot whenever the governed model changes; and close the circle with generate, which emits an upload-ready data dictionary CSV back out of the governed model. Every governed fact — including that PHI inventory — is also queryable by AI agents over MCP.

For the condensed version of everything above, see the REDCap quickstart in the CoreModels docs (docs/quickstarts/redcap).