Catch Pipeline Drift Before It Ships: A CI Gate and Drift Loop for Airflow

Orchestration drift is the quiet kind of failure. Nobody deletes a pipeline; a schedule is flipped to manual during an incident and never flipped back, and for three weeks a downstream table renders yesterday's world with perfect confidence. Nobody removes a dataset from governance; a DAG is renamed in a refactor and the governed model now describes a pipeline that no longer exists. None of this throws an exception — which is exactly why it belongs in CI, where a machine checks it on every change.

CoreModels gives an Airflow estate four automation primitives: a CI-callable audit with a crisp pass/fail semantic, a rolling audit history, a one-call re-audit for when the governed model itself changes, and an SVG badge that shows the current state to anyone who opens the repo. This article wires up all four.

The contract: errorCount semantics

Everything below hangs on one number. An audit compares your deployment's artifacts against the governed model and returns severity counts:

• errorCount > 0 — the estate violates governed meaning (drift like dataset-removed or field-type-drift). Fail the build.
• warningCount > 0 — hygiene findings: Airflow's dag-no-owner, asset-unproduced, and paused-producer rules land here. Fail or tolerate, your call.
• infoCount — advisory (for Airflow, dag-no-description).

The audit is read-only and runs at Viewer role — a CI credential that can never write to the governed model is sufficient, and that is the credential you should mint.

The surface CI calls

CI uses the machine-to-machine surface, where user API keys work:

• POST https://coremodels.example.com/v1/{PROJECT_ID}/integrations/airflow/audit
• GET https://coremodels.example.com/v1/{PROJECT_ID}/integrations/airflow/badge

The v1 surface deliberately carries only those two verbs; reaudit and history live on the interactive surface with your normal login token. Responses on v1 are wrapped in an envelope, so the counts live under data.* — your gate reads data.errorCount, not errorCount.

The GitHub Actions gate

CoreModels ships a packaged composite action for the CI gate, but its file inputs are dbt-shaped (manifest-path, catalog-path), so for Airflow the honest recommendation is the raw curl + jq step — it is barely longer and shows exactly what happens. Prerequisites: a repo secret COREMODELS_API_KEY (a user API key with Viewer access), repo variables COREMODELS_API_URL and COREMODELS_PROJECT_ID, and a prior step that exports dags.json and datasets.json from the Airflow REST API of a staging or reference deployment.

- name: CoreModels Schema Audit
  env:
    COREMODELS_API_URL: ${{ vars.COREMODELS_API_URL }}
    COREMODELS_API_KEY: ${{ secrets.COREMODELS_API_KEY }}
    COREMODELS_PROJECT_ID: ${{ vars.COREMODELS_PROJECT_ID }}
  run: |
    set -euo pipefail
    jq -n --rawfile dags dags.json --rawfile ds datasets.json \
      '{artifacts: {dags: $dags, datasets: $ds}, recordHistory: true}' > audit-request.json
    curl -sS -o audit-response.json -X POST \
      "$COREMODELS_API_URL/v1/$COREMODELS_PROJECT_ID/integrations/airflow/audit" \
      -H "Authorization: Bearer $COREMODELS_API_KEY" -H "Content-Type: application/json" \
      --data-binary @audit-request.json
    test "$(jq -r '.success' audit-response.json)" = "true"
    jq -r '.data.markdown' audit-response.json >> "$GITHUB_STEP_SUMMARY"
    test "$(jq -r '.data.errorCount' audit-response.json)" -eq 0

Three deliberate choices in those last lines. First, check .success separately from the gate — a failed call (unknown vendor, unusable artifacts) should fail loudly, not read as a clean audit. Second, write data.markdown to the job summary before gating, so a red build carries its own explanation — the markdown is the complete audit report, findings and all, formatted for humans. Third, the gate itself is one integer comparison on data.errorCount.

Want a stricter gate? Add a warning check:

    test "$(jq -r '.data.warningCount' audit-response.json)" -eq 0

That promotes paused-producer — the silent-staleness hazard from the opening paragraph — from a note in a report to a blocked merge.

recordHistory and the rolling trail

The recordHistory: true in the request body matters. The audit verb is strictly read-only by default — it records nothing unless asked. With the flag set, each CI run is appended to the project's rolling audit history with trigger ci, alongside interactive runs (audit) and re-audits (reaudit).

The trail is read from the interactive surface:

GET https://coremodels.example.com/graph/integrations/airflow/history/{PROJECT_ID}
Authorization: Bearer $TOKEN

Each run record carries at, trigger, the three severity counts, per-code tallies, and the artifact fingerprint (a content hash of what was audited). The fingerprint turns the history into a diagnostic instrument: consecutive runs with the same fingerprint but different counts mean the governed model moved; a changed fingerprint means the estate moved. That distinction is usually the first question in any drift incident, and the trail answers it without archaeology.

Reaudit: the gate for the other direction

CI audits catch estate-side drift — the deployment changed under a stable model. The reverse also happens: someone updates the governed model (renames a Type, tightens a description of intent) and the last-known estate may no longer conform. That is what reaudit is for. At import time, CoreModels stores the parsed deployment snapshot; reaudit replays the same audit engine over that stored snapshot against the current governed model:

POST https://coremodels.example.com/graph/integrations/airflow/reaudit/{PROJECT_ID}
Authorization: Bearer $TOKEN
Content-Type: application/json

{}

No artifacts, no Airflow API calls, no credentials — the snapshot is already there. The run is always recorded in the history (trigger reaudit; recording is not optional for this verb, because a re-audit exists precisely to leave a trace). Wire it wherever governed-model changes happen — a webhook on your modeling workflow, or a manual step in your model-review checklist.

One honest cap: on very large deployments, when the encoded snapshot exceeds the storage limit (~1.5 MB), import reports snapshotStored: false with a lossiness record. Fresh-artifact audits — including the CI gate above — are unaffected; only reaudit has nothing stored to run against.

The heartbeat

A gate fires on pull requests and a reaudit fires on model changes, but estates also drift while nobody is changing anything — a DAG gets paused in production on a Friday. For that, CoreModels has a scheduled server-side heartbeat: a config-gated worker (Integrations:ScheduledReaudit, off by default) that periodically re-audits opted-in projects against their stored snapshots and records the runs into the same history with trigger scheduled. Off by default is a posture, not an accident: recurring background work against your project is something you switch on deliberately, not something you discover. If your deployment enables it, the history becomes a continuous drift signal with zero pipeline changes on your side.

The badge

Finally, make the state visible. The badge endpoint renders a shields-style SVG from the latest recorded run, labeled airflow audit:

![airflow audit](https://coremodels.example.com/v1/PROJECT_ID/integrations/airflow/badge)

Green means the last recorded run was clean; yellow, warnings only; red, errors; gray, no recorded runs yet. The v1 badge route works with a user API key, which is what makes it embeddable in a README; the same badge is also served on the interactive surface. Because the badge reads the recorded history, it composes with everything above: CI runs with recordHistory: true, reaudits, and heartbeat runs all move the badge; ad-hoc unrecorded audits do not.

The loop, assembled

• Every PR to the DAG repo: CI audit on the v1 route, gate on data.errorCount, report in the job summary, run recorded as ci.
• Every governed-model change: reaudit, always recorded as reaudit.
• Continuously (if enabled): the scheduled heartbeat records scheduled runs.
• Always visible: the badge reflects the latest recorded run; the history explains how it got there.

Estate drift, model drift, idle drift, and visibility — four primitives, one loop, and the paused producer from the opening paragraph never gets three quiet weeks again. The full extraction recipe and endpoint reference are in the Apache Airflow quickstart in the CoreModels integration docs.