Every Check You Run Compares the PR to the Last Commit

Here is a two-minute experiment worth doing on a branch before you read further.

Take a model with an accepted_values test. Delete the test. Open a pull request.

Everything passes.

Not because your CI is badly configured — because every check in it is answering a different question than the one you think you asked. And once you see which question, you notice it applies to more than deleted tests.

The question your pipeline actually asks

Line up what runs on a PR and what each one compares against.

dbt build runs the tests that are in the branch. A deleted test is not a failing test; it is a test that no longer exists. There is nothing to fail.

Branch-versus-main tooling compares the PR to the previous commit. Excellent for catching what changed. But the test was removed in this PR, so the comparison sees a removal — and a removal is a legitimate kind of change, made deliberately all the time. Nothing distinguishes "we removed a test we no longer need" from "we removed the constraint that was holding a promise."

Your reviewers compare it to their memory. Which works for the areas they know well and fails silently everywhere else, in proportion to how long ago the decision was made and whether the person who made it still works here.

Three mechanisms, all correct at their jobs, none of them comparing the PR to what the team agreed the column means. That comparison has no home, because the agreement itself has no home — it lives in a Slack thread from March, a decision nobody wrote down, and the shared assumption that whoever wrote the test had a reason.

Why this is structural, not a discipline problem

The instinct is to fix it with process: require a second reviewer, add a checklist item, tighten the template.

That does not work, and it is worth being clear about why. Every one of those remedies asks a human to hold the prior agreement in their head at review time. The failure is not that people are careless — it is that the reference point does not exist as an artifact, so there is nothing to compare against even for a reviewer who is paying complete attention.

You can see this in which changes slip through. It is never the dramatic ones. It is:

• a permitted value quietly dropped from a list
• a column type widened from something specific to something permissive
• a test removed during a refactor because it was failing and the failure looked spurious
• a model renamed, its old name gone, and a downstream consumer that never had a ref

Each of these is small, locally justified, and passes review because it is locally correct. The damage is at a distance, in time and in team.

What a fourth comparison adds

The gate does one thing: on every PR, it compares the compiled manifest against the governed model — the place the agreement lives once you have somewhere to put it.

That makes a different class of finding possible, with names:

• a permitted-value set that was removed
• an enum that widened or narrowed
• a type that changed underneath a column
• a dataset that vanished

It is not more checks; it is a check with a different reference point. That is why it catches the PR that everything else passes.

The part that decides whether it survives

Turn it on failing only on errors, triage what it finds first, and record what you are deliberately not fixing.

That is not a caveat, it is the design. A gate that goes red on day one for forty things nobody has looked at gets switched off within a week, and the second attempt is much harder because everyone now remembers it as noisy. The recipe provisions a column for exemptions with a reason and a date precisely because "we know, not now" is a legitimate engineering answer and needs somewhere to live that is not someone's memory.

The honest sequence is: import the manifest you consider correct today as the baseline, triage, exempt what you are deferring, then turn the gate on. Starting from a known state is most of whether this works.

What it does not do

It does not replace dbt test, and nothing here is a better version of it — dbt runs the tests, and that job is dbt's. It does not catch bad data; a value that violates a rule in production is a test failure, not a governance finding. It does not know about agreements you never recorded — the governed model is the reference point, so a promise nobody wrote down is still invisible.

And it is worth saying plainly: this is a check on the meaning, not on the SQL. A PR can pass the gate and still be wrong in every way a code review exists to catch.

What actually changes

The specific change is small and repeats: the PR that would have quietly removed a constraint now stops, with a named finding, in front of the person who can say whether it was deliberate.

The larger one is that "reviewed" starts to mean something more than "two people looked at it." Right now the word carries an implication — that someone checked this against what we agreed — which none of your existing checks can support. Adding the fourth comparison makes the implication true.

The dbt CI guide that ships with the CoreModels integration docs covers wiring it, and starts with the same two-minute experiment.