Six Thousand Columns, and Nothing to Check Them Against

Run one query against system.information_schema in a mid-size Unity Catalog estate and what comes back is genuinely impressive: every table and view addressed as catalog.schema.table; every column with its full_data_type spelled out exactly as the catalog holds it, its ordinal position, its nullability flag, and a comment wherever somebody bothered. Hundreds of tables. Thousands of columns. Complete, accurate, current.

Now use it to answer the questions that actually eat your week.

• Which of these two customer tables are downstream teams supposed to build on?
• Is order_status allowed to hold the value that appeared last Thursday, or did it leak in from a change nobody reviewed?
• Is customer_id a key, or a column shaped like one?
• If we retype order_total next sprint, who has to agree, and what breaks?

It cannot answer any of them, and no amount of extra catalog metadata would change that. Every one of those questions is about an agreement, and a catalog is not a place where agreements live. That gap — between a perfect inventory of structure and a governed record of meaning — is why the Databricks Unity Catalog integration in CoreModels exists.

Three gaps that keep becoming the same three incidents

Documentation is optional, so it is mostly absent. Table and column comments in Unity Catalog are a good idea that nothing enforces. Estates therefore accumulate assets that resist every consumer: the analyst who has to ask in Slack, the engineer onboarding onto a domain, the AI assistant with nothing to reason from but a name. Our audit raises this as its own finding — code table-no-comment, severity Info — because it is cheap to fix and expensive to leave.

Intent goes undeclared. Unity Catalog supports primary-key and foreign-key constraints, and they are informational — read by the optimizer and by tools, not enforced by the engine. Declaring them is therefore a choice, and in most estates the choice defaults to "later". So you end up with id, customer_id, order_id columns carrying no declaration at all, and every consumer guessing. The guesses become joins that fan out, dedupe steps bolted on after the first bad number, and a table everyone treats as unique by one column when its real key spans three. The audit's key-column-undeclared warning exists for exactly this shape of column: Unity Catalog will happily record your intent, and nothing is recording it.

Change has no gate that understands meaning. A column gets retyped, a field disappears, a set of accepted values quietly widens. The platform is doing its job; it is not the platform's job to know a downstream aggregation assumed the old type. The change lands and is found at query time — the most expensive possible moment, usually by someone who was not in the room when it was made.

You cannot detect drift with only one side of the comparison

Here is the structural version. Drift detection is a comparison, and a comparison needs two sides. A catalog is a mirror: it always shows the estate exactly as it is right now, which means it can never tell you the estate moved. There is no before stored anywhere with the authority to say "this is what we agreed" — which is why schema post-mortems end in archaeology, reconstructing what a column used to mean from an old dashboard, a stale wiki page, and someone's memory of a decision made two quarters ago.

What the missing half actually is

CoreModels holds the agreed side, without ever holding your Databricks credentials. There is no live workspace connection by design: you run documented queries in the Databricks SQL editor and download the results as JSON. One extract against information_schema is required; two are optional — flattened PK/FK constraint rows, and rows from system.access.table_lineage. Those files are the entire interface.

From them, an import builds a governed model. Tables and views become governed Types identified by their catalog.schema.table name, with the materialization recorded — a managed table, an external table, a view and a materialized view are not the same thing and are not stored as if they were. Columns become Elements with full_data_type preserved verbatim rather than rounded to a generic "number". Comments become descriptions. is_nullable = NO becomes an explicit not-null check.

The optional extracts add the parts that are usually folklore. Primary-key rows are grouped by their constraint, so a three-column key marks each member not-null and records a composite-key marker on the table — it never fakes per-column uniqueness, the single dishonesty behind a whole family of bad joins. Foreign-key rows become governed references. Lineage rows become dependency edges with a direction that means something: the produced table depends on what it reads, and sources outside the extract are flagged as external rather than silently invented.

Crucially, import is additive. Existing governed nodes are never mutated. The governed model is the agreement; the estate is the reality; the interesting question is always where they differ.

The moment the pain actually stops

That question is the audit's whole job. Hand it a fresh information_schema extract and it reports across three sections: coverage (what exists but is not yet governed), drift (a governed field retyped, removed, or with its accepted values narrowed or widened), and conformance (the hygiene rules above). Findings are not prose — each carries a stable code, a severity, and an exact subject, down to main.sales.orders.customer_id.

Because the response also carries a machine-readable error count and a report shaped for a pull-request comment, the audit works as a build gate: an error count above zero fails the build. A status badge — green clean, yellow warnings, red errors, gray when nothing has been recorded yet — summarizes the last recorded run for anyone who just wants to glance.

So the retyped column stops being a broken dashboard found two weeks late. It becomes a red check on the pull request that proposed it, naming the column and stating both the governed type and the drifted one. The change can still happen. It just has to be decided.

The real shortage

Lakehouse teams are not short of metadata. They are short of a reference point: somewhere meaning is agreed, where changing it is a deliberate act, and against which reality can be checked continuously and cheaply. Unity Catalog gives you the estate's current truth. This is the other half.

If your last schema incident ended with "nobody knew that changed," that is the half you are missing. The Databricks Unity Catalog quickstart in the CoreModels docs walks through the extraction queries and the first import and audit end to end.