General logo

Schema Drift Is Inevitable. Losing Track of It Isn't.

Your data estate changed this week — a column's type widened to unblock a load, a model's contract quietly stopped being enforced, an enum grew a value nobody reviewed. None of that is a failure of discipline; it is what living systems do, and the only real failure is not knowing. CoreModels is built around that distinction: drift is expected, and the product's job is to make sure it never goes unnoticed.

Schema Drift Is Inevitable. Losing Track of It Isn't.

Your data estate changed this week — a column's type widened to unblock a load, a model's contract quietly stopped being enforced, an enum grew a value nobody reviewed. None of that is a failure of discipline; it is what living systems do, and the only real failure is not knowing. CoreModels is built around that distinction: drift is expected, and the product's job is to make sure it never goes unnoticed.

Governance that runs once is decoration

Most schema governance follows the same arc. Someone runs an assessment, produces a document, and everyone agrees it is very good. Six months later the document describes an estate that no longer exists.

The problem isn't the assessment — it's the tense. An audit is a photograph, and drift is a film. Estates change daily: every dbt pull request, every warehouse migration, every schema-registry update is a chance for the physical estate to diverge from the meaning your organization agreed on. A check that runs episodically will always be describing the past.

So we built the audit to run as a loop, not an event. Four pieces make the loop: a CI gate that fails builds on divergence, a rolling history that answers "are we drifting?" over time, a re-audit to run whenever the governed model changes, and a badge that shows the current state where people actually look. All of it credential-free.

Make drift fail the build

The entry point is the audit: you hand CoreModels the artifacts your vendor tooling already produces — a dbt manifest, an information-schema export, a registry subject dump — and it compares them against the governed model. The comparison is strictly read-only. It writes nothing to your estate and, unless you explicitly ask it to record the run, nothing to CoreModels either.

The output is a report with three sections. Coverage tells you what exists in the estate but isn't governed at all — datasets and fields nobody has claimed. Drift tells you where the estate and the governed meaning have diverged: a field's type no longer matches, a dataset disappeared, an enum was narrowed or widened. Conformance flags vendor-specific hygiene, like a public dbt model whose contract isn't enforced, or a key column with no test. Every finding carries a stable code — field-type-drift, dataset-removed, enum-narrowed — a severity, and the specific object it names.

That structure is what makes the audit a gate rather than a memo. The rule is one line: if the error count is above zero, the change violates governed meaning, and the build fails. Our GitHub Action wires this up end to end — it calls the audit on every pull request, writes the human-readable report into the job summary, keeps a sticky comment on the PR, annotates each finding on the run, and fails the job on errors (or on warnings, if you want to gate harder).

The concrete moment looks like this: an engineer changes an order-total field from a numeric type to a string to work around a loader. Locally, everything passes — dbt is perfectly happy. But the pull request comes back red, with one finding: the field's type drifted from the governed model, governed type on one side, artifact type on the other. The conversation about whether that change should happen now takes place before merge, with the right people, instead of three weeks later in an incident channel. And because the audit reads a build artifact rather than the warehouse, a plain dbt parse in CI is enough — no warehouse connection required.

"Are we drifting?" is a trend question

A single audit answers "is this change safe?" The harder question — "are we drifting?" — is about direction, and direction needs history.

Every audit can record its run, and the CI action records by default, so a trail accumulates without anyone doing anything extra: which runs were clean, which found drift, what the counts were, and what triggered each run — the history distinguishes CI-triggered runs from interactive ones. The history endpoint turns governance from an anecdote into a series. You can see when a project last passed clean, whether error counts are climbing or falling, and whether that drift finding from last month was resolved or has just been sitting there getting comfortable.

This is deliberately opt-in on the audit itself. An audit someone runs ad hoc to investigate a hunch stays strictly read-only; recording is bookkeeping you ask for. The history should reflect your governance rhythm, not every exploratory poke.

When the model changes, re-audit the estate you already know

There's a second direction drift comes from, and most tooling ignores it: the estate can hold still while the model moves. Your stewards tighten a taxonomy, mark a field required, retire a dataset from the governed vocabulary. Every one of those decisions can put the model out of step with an estate that hasn't changed at all — and nobody wants to re-run extraction pipelines across thirteen kinds of infrastructure just to find out.

So they don't have to. When you import a vendor estate, CoreModels stores a snapshot of it. The re-audit verb replays that last-known estate against the current governed model — no fresh artifacts, no new extract, no ceremony. Change the model on Tuesday, re-audit on Tuesday, and know immediately which physical systems the decision just put out of conformance. Re-audit always records its run, because a check triggered by a governance decision is exactly the kind of event the history exists to remember.

One honest caveat, because we'd rather tell you than have you discover it: very large estates can exceed the snapshot storage cap. When that happens, the import says so plainly — with an explicit record of what wasn't kept — and fresh-artifact audits still work; only the stored-snapshot re-audit is unavailable. We don't silently degrade.

A badge that tells the truth

The last piece is the smallest and possibly the most effective: an SVG status badge, generated from the latest recorded run, embeddable in a README with a read-only API key. Green when the last run was clean, yellow when there were only warnings, red when there were errors — and gray when nothing has been recorded, because "unknown" and "passing" are different claims and a governance tool has no business confusing them.

A badge won't enforce anything. What it does is keep the state of the loop in front of the people who touch the code every day, which is where governance either lives or dies.

Still credential-free

Nothing in this loop requires handing CoreModels a connection string. The audit reads artifacts your tooling already emits; the re-audit reads a snapshot you already imported; the gate runs with a read-only key, because the audit never writes to your project. Every one of our connectors — from dbt and Snowflake to Airflow and REDCap — works this way, and the same audit is available to AI agents as a Model Context Protocol (MCP) tool with the same read-only guarantee. Your credentials stay in your infrastructure, where they belong.

From episodic to ongoing

Put the pieces together and the shape changes. Import an estate once. Audit on every pull request, and let the gate hold the line. Let the history accumulate so "are we drifting?" has an answer with a slope. Re-audit whenever the model itself moves. Keep the badge where everyone can see it.

That's the difference between governance as an event and governance as a loop. Drift will still happen — we promise you it will. You just won't lose track of it.

To wire the loop into your own estate, start with the per-vendor quickstarts in the CoreModels docs.