Automating Glue Drift: The Gate, the Trail, and the Heartbeat

Nothing in your repository changes when a crawler retypes a column. That is the awkward fact about lake governance: the estate you need to watch does not live in git, so the usual "run it on pull requests" reflex leaves the interesting drift entirely unobserved. A Glue drift gate has to be driven by a clock, not by a commit — and once it is, three more mechanisms fall out of it almost for free: a rolling trail, a status badge, and a server-side heartbeat that watches the other direction of drift.

Everything below is read-only against AWS and read-only against your governed model. A Viewer-scoped CoreModels API key is all the automation needs, because the audit verb never writes governed meaning, and your AWS credentials never leave your own runner.

The gate

The machine-to-machine surface accepts user API keys and carries exactly what automation needs. The recipe is three moves: export the catalog, POST it, fail the job when data.errorCount is non-zero.

name: Glue catalog drift gate

on:
  schedule:
    - cron: "0 6 * * *"
  workflow_dispatch:

permissions:
  id-token: write
  contents: read

jobs:
  audit:
    runs-on: ubuntu-latest
    env:
      COREMODELS_API_URL: ${{ vars.COREMODELS_API_URL }}
      COREMODELS_API_KEY: ${{ secrets.COREMODELS_API_KEY }}
      COREMODELS_PROJECT_ID: ${{ vars.COREMODELS_PROJECT_ID }}
      GLUE_DATABASES: "lake reference_data"
    steps:
      - name: Configure AWS credentials
        # Your usual OIDC / credentials step. The role needs read access to the Glue catalog;
        # nothing about it is shared with CoreModels.
        run: aws sts get-caller-identity

      - name: Export the catalog
        run: |
          set -euo pipefail
          for DB in $GLUE_DATABASES; do
            aws glue get-tables --database-name "$DB" > "tables-$DB.json"
          done
          jq -s '.' tables-*.json > tables.json

      - name: CoreModels schema audit
        run: |
          set -euo pipefail
          jq -n --rawfile tables tables.json \
            '{artifacts: {tables: $tables}, recordHistory: true}' > audit-request.json

          status=$(curl -sS -o audit-response.json -w "%{http_code}" -X POST \
            "$COREMODELS_API_URL/v1/$COREMODELS_PROJECT_ID/integrations/glue/audit" \
            -H "Authorization: Bearer $COREMODELS_API_KEY" \
            -H "Content-Type: application/json" \
            --data-binary @audit-request.json)

          test "$status" = "200"
          test "$(jq -r '.success' audit-response.json)" = "true"

          jq -r '.data.markdown' audit-response.json >> "$GITHUB_STEP_SUMMARY"

          jq -r '.data.findings[] | select(.severity == "Error")
                 | "::error title=\(.code)::\(.subject) — \(.message)"' audit-response.json

          errors=$(jq -r '.data.errorCount' audit-response.json)
          echo "CoreModels Glue audit: $errors error(s)."
          test "$errors" -eq 0

It is one curl with two headers, so the same body runs unchanged from GitLab, Jenkins, or a cron job on a jump host. CoreModels also ships a packaged composite GitHub Action around this route, but it is built around dbt's artifact names and expects a manifest file on disk — for Glue, the explicit curl + jq step above is the recipe to copy.

What the gate is actually reading

The v1 surface wraps its report in the standard envelope, so every field lives under data.*. Three of them carry the logic.

success answers "did the audit run?". false means it could not proceed — a malformed export, an unknown vendor key, an artifact that is valid JSON but not shaped like get-tables output. Check it before the counts, because a build that skips a broken audit is worse than a build that fails one.

data.errorCount is the contract: greater than zero means the catalog violates governed meaning. In a Glue estate the Error-severity findings you will realistically see are dataset-removed (a governed table is gone from the catalog), field-removed (a governed column is gone), and field-type-drift (a column's Hive type changed since the last import — the comparison is case- and whitespace-insensitive, so VARCHAR(255) and varchar(255) are the same type, and decimal(19, 4) does not drift into decimal(19,4)). Add enum-narrowed to that list once you govern taxonomies on lake columns.

data.warningCount / data.infoCount are signal rather than violation: ungoverned tables (dataset-unmapped), Glue's semi-structured-column rule, and the table-no-description / classification-missing hygiene rules. The workflow above lets them pass.

Two more fields are there when you want tighter handling. data.driftedObjects lists the vendor identities that drifted — a compact "what changed" for a chat notification. data.codes is a map of finding code to count, which is how you write a policy that is stricter than "errors" but looser than "everything":

# Gate on warnings as well as errors:
test "$(jq -r '.data.warningCount' audit-response.json)" -eq 0

# Or: ignore hygiene noise, block only on structural drift.
jq -e '((.data.codes["field-type-drift"] // 0)
      + (.data.codes["dataset-removed"] // 0)
      + (.data.codes["field-removed"]   // 0)) == 0' audit-response.json

One scoping rule saves a lot of false alarms: dataset-removed only fires for governed datasets whose identity namespace appears in the export you sent. Glue identities are database.table, so auditing only lake never claims that reference_data's tables vanished. Export the databases you want gated; the rest are simply out of that comparison's scope.

Feeding the trail

Note recordHistory: true in the request body. It is false by default — the audit verb stays strictly read-only unless asked — but in a scheduled gate you want it on. Each run is appended to the project's rolling trail with trigger ci, and that trail is what turns point-in-time checks into a trend:

curl -sS -H "Authorization: Bearer $TOKEN" \
  "https://coremodels.example.com/graph/integrations/glue/history/$PROJECT_ID" \
  | jq -r '.projects[].runs[] | [.at, .trigger, .errorCount, .warningCount, .fingerprint] | @tsv'

2026-08-04T06:00:41.2210984+00:00   ci        0   1   3f9c1d5a7b2e4086
2026-08-03T06:00:38.9914772+00:00   ci        0   1   3f9c1d5a7b2e4086
2026-08-02T18:22:05.4471130+00:00   reaudit   2   1   3f9c1d5a7b2e4086

Each record keeps the timestamp, the trigger (audit, ci, reaudit, scheduled), the three counts, a per-code breakdown, and the artifact fingerprint — deliberately compact, and capped to a rolling window rather than kept forever. The fingerprint is the field that earns its keep during an incident: three runs above saw byte-identical catalog content, so the two errors on August 2 came from the governed model changing, not from the lake. Same fingerprint plus different findings means somebody edited meaning; different fingerprint plus same findings means the lake moved and your model absorbed it.

The badge

curl -sS -H "Authorization: Bearer $COREMODELS_API_KEY" \
  "$COREMODELS_API_URL/v1/$COREMODELS_PROJECT_ID/integrations/glue/badge" > glue-audit.svg

A shields-style SVG labeled glue audit, rendered from the latest recorded run: green for clean, yellow for warnings only, red for errors, gray for no recorded runs yet. The route is authenticated like every other v1 route, so the practical pattern is the one above — fetch it in the pipeline after the audit step and publish the SVG wherever your README or dashboard can reach it. And note the dependency: a pipeline that never sets recordHistory will look at a gray badge forever, no matter how many audits it runs.

The direction CI cannot see

Your gate fires when the catalog changes. Drift has a second direction: someone edits governed meaning in CoreModels — retypes an element, tightens a taxonomy — and now the last-known catalog may no longer conform. No crawler ran. No export exists. Re-audit answers exactly that question by replaying the audit engine over the snapshot stored at import time against the current governed model:

curl -sS -X POST \
  "https://coremodels.example.com/graph/integrations/glue/reaudit/$PROJECT_ID" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{}'

No artifacts, no AWS access, and the run is always recorded — if you asked the question, the trail shows the answer. Two constraints shape how you automate it. First, reaudit and history live on the interactive surface only; the API-key surface carries audit and badge. So this is a verb for a human at a terminal, or for the scheduler below — not something to bolt onto a CI key. Second, re-audit depends on the stored snapshot: a catalog whose encoded snapshot exceeds the storage cap (roughly 1.5 MB) imports with snapshotStored: false and a lossiness record saying so. Fresh-artifact audits keep working exactly as before; re-audit has nothing to run against.

The heartbeat

The last layer removes the human from that habit. CoreModels can run a scheduled server-side re-audit — a configuration-gated background job, off by default — that re-audits opted-in projects against their stored snapshots and records each run in the history under its own trigger, scheduled. Operators enable it per deployment with three settings: a flag to turn it on, a cron expression (daily in the early hours if unset), and the explicit list of project ids that opted in. Nothing is swept implicitly.

With all four pieces in place the loop is closed in both directions and needs no attention:

• the scheduled gate audits fresh catalog exports and fails loudly on errorCount > 0;
• the heartbeat re-audits the stored estate against the current model, catching meaning changes nobody re-exported for;
• both land in one trail with distinguishable triggers, so you can tell "the lake moved" from "we moved";
• and the badge summarizes the whole arrangement in a single color.

What to page on: errorCount > 0 on the gate, and any dataset-removed in a scheduled run — the second one usually means a table was dropped between crawls and nobody told the model. Everything else is a queue, not an incident.

The AWS Glue quickstart in the CoreModels documentation carries the complete gate recipe together with every route on both surfaces.