Fail the Build, Not the Dashboard: A CI Drift Gate for Databricks Unity Catalog

Somewhere in your lakehouse, a notebook is about to rebuild a table with a widened column type, and nothing will fail until a dashboard does — days later, in front of the wrong audience. The cheapest place to catch that is a pull request. This article turns the CoreModels schema audit into a merge-blocking CI gate for a Unity Catalog estate, then builds out the rest of the operational loop most drift setups forget: the status badge, the rolling history, the one-call re-audit for when the governed model itself changes, and the scheduled heartbeat that keeps watch between releases.

The endpoint CI calls

CI should never hold an interactive login, so the gate targets the machine-to-machine surface, which accepts CoreModels user API keys:

POST https://coremodels.example.com/v1/$PROJECT_ID/integrations/databricks/audit

The audit runs at Viewer role and never writes, so the key in your CI secrets needs no write access to anything. The response comes wrapped in the standard ApiResponse envelope: the report lives under data, and the gate keys on one number, data.errorCount.

The semantics decide when builds go red, so they are worth being precise about. Errors mean the extract contradicts governed meaning — a governed field removed (field-removed), a column retyped against the model (field-type-drift), an enum narrowed (enum-narrowed). Warnings are the connector's best-practice rules; for Databricks that is key-column-undeclared, a key-shaped column with no declared informational constraint. Info findings, such as table-no-comment, are documentation nudges. Failing on errorCount > 0 and letting warnings through keeps the signal crisp: red means meaning was violated, not that someone forgot a comment. Gate harder only once your estate is clean enough that warnings are actionable.

Getting the extract into CI

The audit needs one artifact: information_schema, the JSON rows of the documented Unity Catalog query. Two patterns work.

• Commit the extract with schema-change PRs. The PR then carries both the intended change and the evidence, and the audit runs on exactly what is proposed.
• Re-extract in a prior CI step against a staging catalog — for instance with databricks-sql-cli — so the pipeline always audits current reality.

Either way, the audit step itself needs nothing but curl and jq.

The gate, in full

name: coremodels-schema-audit
on:
  pull_request:
    paths: ["schemas/**"]

jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # A prior step places information_schema.json in the workspace —
      # committed with the PR, or re-extracted against a staging catalog.

      - name: CoreModels Schema Audit
        env:
          COREMODELS_API_URL: ${{ vars.COREMODELS_API_URL }}
          COREMODELS_API_KEY: ${{ secrets.COREMODELS_API_KEY }}
          COREMODELS_PROJECT_ID: ${{ vars.COREMODELS_PROJECT_ID }}
        run: |
          set -euo pipefail
          jq -n --rawfile info information_schema.json \
            '{artifacts: {information_schema: $info}, recordHistory: true}' > audit-request.json
          curl -sS -o audit-response.json -X POST \
            "$COREMODELS_API_URL/v1/$COREMODELS_PROJECT_ID/integrations/databricks/audit" \
            -H "Authorization: Bearer $COREMODELS_API_KEY" -H "Content-Type: application/json" \
            --data-binary @audit-request.json
          test "$(jq -r '.success' audit-response.json)" = "true"
          jq -r '.data.markdown' audit-response.json >> "$GITHUB_STEP_SUMMARY"
          test "$(jq -r '.data.errorCount' audit-response.json)" -eq 0

Reading it line by line: jq -n --rawfile wraps the extract as a string value inside the request body, escaping included. The first test distinguishes "the audit could not run" from "the audit ran and found problems" — a transport failure or unusable payload should fail the build loudly, not masquerade as a clean report. The markdown field goes to the job summary, so reviewers see the human-readable report — findings grouped by section, coded and subject-lined — without leaving the PR. The final test is the gate itself: any error finding fails the step.

recordHistory: true matters more than it looks. Every gated run is appended to the project's rolling audit trail with trigger ci, which is what makes the next two pieces work.

One note if you already use our composite GitHub Action elsewhere: its artifact inputs are manifest-shaped, built for build-artifact vendors. For a Unity Catalog extract the curl-and-jq step above is the recipe — same endpoint, same envelope, same gate semantics.

The badge

The same machine-to-machine surface serves an SVG badge rendered from the latest recorded run:

curl -sS -H "Authorization: Bearer $COREMODELS_API_KEY" \
  "https://coremodels.example.com/v1/$PROJECT_ID/integrations/databricks/badge" \
  -o databricks-audit.svg

Green means the last recorded run was clean, yellow means warnings only, red means errors, gray means nothing recorded yet. The route authenticates with the API key, so for a public README the practical pattern is the one above: fetch the badge in a scheduled job and publish the SVG wherever your docs live. Because the badge reflects recorded runs, a gate that records on every PR keeps it truthful for free.

The rolling history

The badge answers "now"; the history answers "lately":

curl -sS -H "Authorization: Bearer $TOKEN" \
  "https://coremodels.example.com/graph/integrations/databricks/history/$PROJECT_ID"

{ "success": true, "vendor": "databricks",
  "projects": [ { "projectName": "main", "runs": [
    { "at": "2026-08-03T09:12:44.6183920+00:00", "trigger": "ci",
      "errorCount": 0, "warningCount": 2, "infoCount": 3,
      "codes": { "key-column-undeclared": 2, "table-no-comment": 3 },
      "fingerprint": "9f2c4a1b0e77d3aa" },
    { "at": "2026-08-01T16:40:02.1170455+00:00", "trigger": "reaudit",
      "errorCount": 1, "warningCount": 2, "infoCount": 3,
      "codes": { "field-type-drift": 1, "key-column-undeclared": 2, "table-no-comment": 3 },
      "fingerprint": "9f2c4a1b0e77d3aa" } ] } ] }

Runs come back newest first, and trigger is one of four lowercase values: audit, ci, reaudit, scheduled. The trail is a rolling window — the most recent 50 runs per estate — so treat it as a drift signal, not an archive: if you need audit records kept forever, store the full audit response your gate already downloaded as a build artifact. Two further details make this trail genuinely useful. The fingerprint is a content hash of the audited artifacts: identical fingerprints with different finding counts mean the governed model moved, not the estate — exactly the situation in the sample above, where a field-type-drift error appeared and then cleared between two runs of the same extract. And codes gives you drift trends per rule without parsing findings. Note the surface boundary: history (and re-audit, next) live on the interactive surface with a login token; the API-key surface carries audit and badge.

Re-audit: the gate for the other direction

CI catches the estate drifting from the model. But models change too — someone tightens a type or removes a governed field in CoreModels. Re-audit checks the current governed model against the estate snapshot stored at import time, no fresh extract required:

curl -sS -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  "https://coremodels.example.com/graph/integrations/databricks/reaudit/$PROJECT_ID" \
  -d '{}'

The run is always recorded in the history (trigger reaudit), so governed-side changes leave the same audit trail estate-side changes do. The dependency to know about: re-audit needs the snapshot stored at import. Very large estates can exceed the snapshot storage cap (roughly 1.5 MB encoded), in which case import reports snapshotStored: false and re-audit has nothing to run against — fresh-artifact audits still work as usual.

The heartbeat

PR gates only fire when someone opens a PR. For the quiet weeks, CoreModels deployments have a scheduled server-side re-audit — a background worker, off by default and enabled by the operator through deployment configuration, per opted-in project — that periodically re-audits those projects against their stored snapshots and records the runs into the same history under trigger scheduled. No credentials, no vendor API calls, no CI minutes: it replays the last-known estate against the current model on a schedule, and a project with no stored snapshot is simply skipped. Combined with the PR gate you get both directions covered continuously: estate-side drift is caught at merge time, governed-side drift is caught by the heartbeat, and both write to one trail behind one badge.

The loop, assembled

One secret (a Viewer-scoped API key), one committed or re-extracted JSON file, and a five-line audit step buy you: a merge gate on governed meaning, a PR-visible report, a truthful badge, a queryable drift history with content fingerprints, and a scheduled heartbeat for the weeks nobody touches the schema. None of it holds a Databricks credential.

The extraction queries and the interactive-surface equivalents of these calls are in the Databricks quickstart in the CoreModels documentation.