Five Commitments We Make Before Touching Your Airbyte Catalog

Any tool that asks to sit between your data sources and your governed schemas is asking for a lot of trust. We think that trust should rest on verifiable commitments rather than assurances — specific behaviors, designed into the system, that you can test before you rely on them. Here are the five that govern how CoreModels handles an Airbyte catalog, and why each one exists.

1. Import is additive — your governed model is never overwritten

When you import a catalog, streams that aren't yet governed become governed Types, their properties become Elements, declared enums become taxonomies, and primary keys become uniqueness and not-null checks. But a node that already exists in the governed model is never mutated or deleted by an import. Re-import the same catalog, or a newer one, and the response tells you plainly what happened: how many datasets were added and how many were skipped because they already existed.

This matters because the alternative is quietly catastrophic. An import that "helpfully" updates existing nodes would let any upstream schema change rewrite your organization's agreed meaning without anyone consenting to it — which is precisely the failure governance exists to prevent. If the source now disagrees with the governed model, that disagreement is not resolved silently in either direction. It is surfaced, as drift, for a person to judge. The one thing a re-import does refresh is vendor bookkeeping metadata — sync modes, cursor configuration, import state — which describes the estate, not the meaning.

2. Changing meaning is a human act

The audit engine is very good at detecting that meaning has moved: a governed field whose type changed in the catalog, an enum that narrowed, a stream that disappeared. It is deliberately not in the business of deciding whether the move was right. Findings arrive with stable codes — field-type-drift, enum-narrowed, dataset-removed, and the rest — a severity, a subject, and a message; what happens next is a review, by people, with the evidence in front of them.

The role model enforces the same split. Importing requires Admin rights on the project. Auditing requires only Viewer rights, because it writes nothing. Even recording an audit run into the rolling history is opt-in, behind an explicit flag in the request — a read-only verb that left records behind by default wouldn't really be read-only. The single documented exception is the re-audit verb, which always records its run; it exists to build the drift trail, and it says so up front.

3. Lossiness is reported, never buried

Every translation between formalisms loses something, and the honest question is what happens to the remainder. Our answer: a successful import can still carry a lossiness report stating exactly what was approximated or dropped. Success and honesty are separate channels.

Airbyte offers two good examples. Object and array properties have no faithful primitive representation, so they are approximated — and then surfaced by the audit's untyped-fields finding, which counts and names them per stream, because semi-structured data entering the warehouse ungoverned should be visible, not laundered into a clean-looking model. And when a very large catalog exceeds the snapshot storage cap — around one and a half megabytes encoded — the import says so directly: the response reports that no snapshot was stored, with a lossiness record explaining why. Fresh-artifact audits still work; re-audit honestly doesn't, because its input honestly doesn't exist.

The same honesty extends to capability. Our Airbyte connector imports and audits; it does not generate catalogs, because a catalog describes what a source exposes and authoring it belongs to the source. Ask it to generate and you get an explicit error — never a plausible-looking artifact emitted silently. And there is no live connection to your Airbyte instance at all: you export the catalog with your own credentials, on your own side, and upload JSON. We never hold your Airbyte credentials.

4. Drift is evidence, not opinion

A one-off audit is a snapshot; governance needs a record. Every audit can be recorded into a rolling history, and each recorded run carries its timestamp, trigger, severity counts, finding codes, and the fingerprint of the artifact it examined — a content hash, so you can prove which catalog produced which verdict. The history endpoint turns "when did this start?" into a lookup. The SVG badge — green for clean, yellow for warnings, red for errors, gray when nothing is recorded — makes the latest verdict ambient.

Crucially, drift is checked in both directions. The live audit asks: do these fresh artifacts still conform to the governed model? The re-audit asks the mirror question: does the governed model still match the last-known estate? — by re-running the same audit engine over the catalog snapshot stored at import time against the current governed model. When someone tightens the model on a Wednesday, one call tells you whether the estate as last seen still conforms, with the run recorded.

5. Review, then deploy — the gate is yours

All of this converges in continuous integration. The machine-to-machine surface accepts user API keys and serves the audit and the badge; a CI job posts the freshly discovered catalog and fails the build when the error count is non-zero, with the Markdown report attached to the pull request. The gate's threshold, its placement, and the decision to merge remain entirely yours — CoreModels supplies the evidence, not the verdict.

That is the shape of the whole design: detection automated, meaning human-gated, remainders declared, history kept, and enforcement under your control. Governance tools should be easy to audit themselves — every commitment above is observable in the responses the system returns, starting with your first import.

The CoreModels Airbyte quickstart shows each of these behaviors live, from first import to CI gate.