Governing Your Warehouse Schema: A dbt + Databricks Walkthrough

Your dbt project and your Databricks catalog each think they know what orders.order_total means. CoreModels holds the governed answer in one place — and audits both estates against it, in CI, without ever holding a warehouse credential. Here is the whole loop, end to end, with real requests and real responses.

Why govern the warehouse at all?

Schema drift is not a hypothetical. A column quietly changes from DOUBLE to VARCHAR in a hotfix, a taxonomy loses a value nobody remembers approving, a contract that was supposed to be enforced never was. The failures show up weeks later, downstream, in a dashboard someone trusts. The root cause is always the same: the meaning of the schema lives in people's heads, while the shape of the schema lives in three tools that don't talk to each other.

CoreModels (by ARAMAI) takes a different position. The governed model — types, elements, taxonomies, references — lives in one graph. Your vendor estates (dbt, Databricks Unity Catalog, and eleven others) are audited against that graph using the build artifacts your tooling already produces. Import is additive and never mutates governed nodes; audit and generate never write anything. And when a translation between worlds can't be exact, we say so: every operation returns an explicit lossiness ledger rather than silently papering over the gap. Meaning changes stay human-gated; drift detection is automated.

This walkthrough runs the full loop for a dbt project materialized in Databricks: extract, import, audit, gate the PR, re-audit over time, and generate governed artifacts back out — first over HTTP, then the same flow over MCP for agent-driven governance. Replace https://coremodels.example.com with your CoreModels host throughout.

Step 1: Extract artifacts — no credentials required

CoreModels never connects to your warehouse. No connector holds live credentials — you upload artifacts your vendor tooling already produces.

For dbt, the artifacts are files dbt writes into target/:

dbt parse                # or compile/run/build — writes target/manifest.json
dbt docs generate        # optional — writes target/catalog.json

The manifest artifact is required (manifest schema v10–v12; any dbt command produces it). catalog and semantic_manifest are optional enrichers.

For Databricks Unity Catalog, the required information_schema artifact is one query in the SQL editor:

SELECT c.table_catalog, c.table_schema, c.table_name, t.table_type, t.comment AS table_comment,
       c.column_name, c.ordinal_position, c.full_data_type, c.is_nullable, c.comment
FROM system.information_schema.columns c
JOIN system.information_schema.tables t
  ON t.table_catalog = c.table_catalog AND t.table_schema = c.table_schema AND t.table_name = c.table_name
WHERE c.table_catalog = '<YOUR_CATALOG>' AND c.table_schema = '<YOUR_SCHEMA>';
-- Download the result as JSON from the SQL editor.

Optional Databricks artifacts: keys (flattened PK/FK rows) and lineage (system.access.table_lineage rows).

Step 2: Import the estate into the governed graph

Import runs on the interactive surface (your normal login token, Admin role on the project) and is strictly additive — re-importing never mutates existing governed nodes. The body is the same shape for every vendor: a map of artifact name to raw content.

# dbt: build the request body with jq, then import
jq -n --rawfile manifest target/manifest.json \
      --rawfile catalog  target/catalog.json \
  '{ artifacts: { manifest: $manifest, catalog: $catalog } }' \
| curl -s -X POST \
    "https://coremodels.example.com/graph/integrations/dbt/import/$PROJECT_ID" \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d @-

A real import response:

{ "success": true, "vendor": "dbt", "projectName": "jaffle_shop",
  "datasetsAdded": 9, "datasetsSkippedExisting": 0, "fieldsAdded": 42,
  "lineageEdgesAdded": 11, "lineageEdgesSkipped": 0, "nodesEnriched": 51,
  "snapshotStored": true, "lossiness": [], "errors": [] }

Datasets become Types, columns become Elements, accepted values become Taxonomies, relationships become references — plus lineage and vendor metadata. snapshotStored: true means the estate snapshot was stored for later re-audit; when an encoded snapshot exceeds the storage cap (~1.5 MB), import reports snapshotStored: false with a lossiness record — fresh-artifact audits still work, but reaudit has no stored snapshot to run against.

The Databricks side is the same call with its own artifacts:

jq -n --rawfile schema information_schema.json \
  '{ artifacts: { information_schema: $schema } }' \
| curl -s -X POST \
    "https://coremodels.example.com/graph/integrations/databricks/import/$PROJECT_ID" \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d @-

With both estates imported, one optional call links the datasets both vendors govern as the same physical relation, via reciprocal sameAs assertions (idempotent, Admin role):

curl -s -X POST \
  "https://coremodels.example.com/graph/integrations/reconcile/$PROJECT_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "vendorA": "dbt", "vendorB": "databricks", "spaces": null }'

Now your dbt model and the Delta table it materializes are one governed thing, seen through two lenses.

Step 3: Audit — what a finding actually looks like

Audit compares fresh artifacts against the governed graph. It is read-only (Viewer role suffices) and, by default, records nothing — the audit verb stays strictly read-only unless asked, via "recordHistory": true.

jq -n --rawfile manifest target/manifest.json \
  '{ artifacts: { manifest: $manifest }, recordHistory: false }' \
| curl -s -X POST \
    "https://coremodels.example.com/graph/integrations/dbt/audit/$PROJECT_ID" \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d @-

The report has three sections — Coverage, Drift, Conformance — and three severities: Error, Warning, Info. An abridged real response:

{
  "success": true,
  "vendor": "dbt",
  "projectName": "jaffle_shop",
  "errorCount": 1,
  "warningCount": 1,
  "infoCount": 0,
  "codes": { "field-type-drift": 1, "key-column-untested": 1 },
  "driftedObjects": ["model.jaffle_shop.orders"],
  "findings": [
    { "section": "Drift", "severity": "Error", "code": "field-type-drift",
      "subject": "model.jaffle_shop.orders.order_total",
      "message": "Field type drifted from the governed model.",
      "detail": "governed: Double, artifact: VARCHAR" }
  ],
  "markdown": "...",
  "historyRecorded": false,
  "lossiness": []
}

The markdown field is a PR-comment-ready human report; fingerprint (an artifact content hash) and metrics ride along too. The finding codes are stable and machine-checkable:

• Coverage — dataset-unmapped, field-unmapped, projection-unmapped
• Drift — dataset-removed, contract-drift, field-removed, field-type-drift, enum-constraint-removed, enum-narrowed, enum-widened
• dbt conformance — contract-not-enforced, contract-column-missing-type, key-column-untested, source-no-freshness
• Databricks conformance — table-no-comment, key-column-undeclared

One rule matters above all: errorCount > 0 means the change violates governed meaning. That is the CI gate.

Step 4: Wire the CI gate

CI uses the machine-to-machine surface at v1/..., where user API keys work and both routes are Viewer role — the audit never writes, so a Viewer key suffices. dbt parse is the cheapest way to produce a manifest in CI; no warehouse connection is needed for the audit itself.

With the GitHub Action (a composite action, "CoreModels Schema Audit"), set one repo secret (COREMODELS_API_KEY — a CoreModels user API key) and two repo variables (COREMODELS_API_URL, COREMODELS_PROJECT_ID), then:

jobs:
  schema-audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build the dbt manifest
        run: dbt parse   # no warehouse connection needed for the audit
      - name: CoreModels Schema Audit
        uses: ./.github/actions/coremodels-audit
        with:
          api-url: ${{ vars.COREMODELS_API_URL }}
          api-key: ${{ secrets.COREMODELS_API_KEY }}
          project-id: ${{ vars.COREMODELS_PROJECT_ID }}
          manifest-path: target/manifest.json
          fail-on: error          # 'warning' to gate harder
          record-history: "true"  # feeds the history trail + badge

The action writes the markdown report to the job summary, posts a sticky PR comment, emits an annotation for every Error and Warning finding, and fails the build when errorCount > 0 (or on warnings, with fail-on: warning). Outputs include error-count, warning-count, and drifted-objects for downstream steps.

No GitHub? The gate is one curl and one jq. On the v1 surface the response rides in an envelope, so the counts live under data.*:

RESPONSE=$(jq -n --rawfile manifest target/manifest.json \
    '{ artifacts: { manifest: $manifest }, recordHistory: true }' \
  | curl -s -X POST \
      "https://coremodels.example.com/v1/$PROJECT_ID/integrations/dbt/audit" \
      -H "Authorization: Bearer $COREMODELS_API_KEY" \
      -H "Content-Type: application/json" \
      -d @-)

if [ "$(echo "$RESPONSE" | jq '.data.errorCount')" -gt 0 ]; then
  echo "$RESPONSE" | jq -r '.data.markdown'   # human-readable report to the log
  exit 1                                      # the PR violates governed meaning
fi

Runs recorded from CI carry the trigger Ci, and they feed the status badge — a shields-style SVG from the latest recorded run (green for clean, yellow for warnings only, red for errors, gray when no run is recorded), embeddable in READMEs with a user API key:

curl -s -H "Authorization: Bearer $COREMODELS_API_KEY" \
  "https://coremodels.example.com/v1/$PROJECT_ID/integrations/dbt/badge" \
  > schema-audit-badge.svg

Step 5: The ongoing loop — reaudit and history

The gate catches vendor-side drift on every PR. But drift is two-sided: the governed model itself evolves. reaudit replays the estate snapshot stored at import time against the current governed model — no artifacts needed, and the run is always recorded:

curl -s -X POST \
  "https://coremodels.example.com/graph/integrations/dbt/reaudit/$PROJECT_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "projectName": null, "spaces": null }'

(projectName: null means "the latest stored snapshot".) The history endpoint then gives you drift over time — the recent audit trail across CI runs and reaudits:

curl -s -H "Authorization: Bearer $TOKEN" \
  "https://coremodels.example.com/graph/integrations/dbt/history/$PROJECT_ID"

Note the surface split: reaudit and history live on the interactive surface only; the v1 API-key surface carries audit and badge.

Step 6: Generate artifacts back out

Governance that only ever reads is half a loop. The generate verb produces vendor artifacts from the governed graph — still read-only against the graph, Viewer role:

curl -s -X POST \
  "https://coremodels.example.com/graph/integrations/dbt/generate/$PROJECT_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "typeNames": [], "targetVersion": null, "extra": {}, "spaces": null }'

For dbt this yields models/coremodels_contracts.yml: enforced model contracts with per-column data_type, not_null constraints, unique/not_null tests, accepted_values blocks generated from your governed taxonomies, and relationships tests from governed references. The default targets dbt ≥ 1.8 (the data_tests: key); pass "targetVersion": "1.7" to emit the legacy tests: key.

For Databricks, the same call against the databricks connector yields coremodels_delta_tables.sql: CREATE TABLE IF NOT EXISTS ... USING DELTA statements with NOT NULL from governed checks, one informational PRIMARY KEY per table, FOREIGN KEY ... REFERENCES from governed references, and taxonomy allowed values carried on column comments. Views are skipped — with declared lossiness, not silence.

That closes the loop: the governed model is no longer just a referee; it is the source your contracts and DDL are generated from.

Step 7: The same loop over MCP, for agents

Everything above is also exposed as MCP tools, so an AI agent can run governance conversationally. Connect a client to the OAuth-protected endpoint (Claude Code shown; any streamable-HTTP MCP client works):

claude mcp add --transport http coremodels https://go.coremodels.io/mcp

/mcp serves the read-only (Viewer-role) tools; write tools live on the admin endpoint (/mcp-admin) and additionally require write-level membership on the project — import_vendor_project and reconcile_vendor_projects both require the Admin role — and the per-project role check is the enforced authorization boundary regardless of endpoint.

The audit tool mirrors the HTTP surface. audit_vendor_project requires graphProjectId (32-char hex) and vendor, and takes either inline artifacts or artifactUrls for payloads too large to inline (URL fetches are SSRF-guarded: https only, no redirects, private ranges refused):

{
  "tool": "audit_vendor_project",
  "arguments": {
    "graphProjectId": "<32-char-hex-project-id>",
    "vendor": "dbt",
    "artifactUrls": { "manifest": "https://ci.example.com/artifacts/manifest.json" }
  }
}

The result is the same coverage/drift/conformance report — and the same contract: errorCount > 0 is the fail signal. Around it sit get_vendor_integration_status (registered connectors and capabilities; with a vendor, the project's last-import state), import_vendor_project (the additive estate import: datasets to Types, columns to Elements, accepted values to Taxonomies), and generate_vendor_artifacts (dbt schema.yml with enforced contracts, straight from the governed graph). An agent can import an estate, audit a proposed change, explain the findings in prose, and generate the corrected contract file — while the read/write split and project roles keep it from ever changing governed meaning without an explicitly authorized surface.

Where to go next

dbt and Databricks are two of thirteen connectors — Snowflake, BigQuery, Microsoft Fabric, AWS Glue, Confluent Schema Registry, Airbyte, Airflow, Neo4j, REDCap, Salesforce, and cBioPortal follow the same extract → import → audit → generate loop. The per-vendor quickstarts in the CoreModels docs give the exact extraction recipe and artifact names for each.