Follow One Table Across Your Stack

Pick a table. Say sales.orders.

An ingestion tool landed the raw rows. A transformation project built the model. Unity Catalog governs the Delta table it materializes into, with access control, discovery and lineage. An orchestrator schedules the job that refreshes it. Upstream, a schema registry holds the event contract feeding the chain; downstream, a BI tool exposes a version of it to people who have never heard of any of the above.

Now count the catalogs holding a partial truth about that one table. Each is internally consistent. Each is authoritative about its own layer. None can tell you whether all six describe the same business fact — because none can see the other five.

That is the landscape the Databricks connector lives in, and it explains what the connector is for.

What Unity Catalog is for, and where its scope ends

Unity Catalog is good at its job, and its job is the lakehouse: governing access, cataloging objects, recording lineage between them, giving every table a clean three-part identity. We do not compete with any of that. We consume it — through system.information_schema, through constraint rows, through system.access.table_lineage.

The scope boundary is the word "lakehouse". Almost nobody's estate is a lakehouse. It is a lakehouse plus a transformation layer plus a warehouse plus event streams plus SaaS systems, each with its own catalog, its own type system, and its own idea of what a table is. Meaning does not respect those boundaries, so governing each layer perfectly in that layer's own tool produces well-governed silos and no governed whole.

One neutral estate model, thirteen ways in

CoreModels treats this as a neutrality problem rather than a coverage problem.

The integration layer speaks to thirteen systems today: Databricks Unity Catalog alongside dbt, Snowflake, Google BigQuery, Microsoft Fabric, AWS Glue Data Catalog, Confluent Schema Registry, Airbyte, Apache Airflow, Neo4j, Salesforce, REDCap and cBioPortal. Every one of them parses into the same neutral estate model — datasets with fields and normalized checks, plus lineage edges — and nothing else. A connector may not introduce a new top-level concept.

So the Databricks connector's job is faithful translation into a shared vocabulary. Tables and views become governed Types identified by catalog.schema.table. Columns become Elements. Informational primary-key and foreign-key declarations become checks and governed references. Lineage rows become dependency edges.

What is genuinely Databricks-shaped is not flattened away — it rides as vendor metadata on the governed nodes: the verbatim full_data_type, the materialization (managed table, external table, view, materialized view), the physical name, the recorded checks. The core model stays neutral; the vendor detail stays queryable. That discipline is why no connector can contaminate the others, and why the next one is cheap.

Lineage that does not stop at the platform edge

The clearest payoff is lineage. Rows from system.access.table_lineage become Depends On edges pointing from the produced table to what it reads — the same edge kind a dbt ref produces when a dbt project is imported into the same governed project.

One vocabulary means one answer. "What does this table depend on, and what breaks if it changes?" traverses a single graph spanning the transformation layer and the lakehouse, instead of returning two partial answers from two UIs that you reconcile by hand. Sources appearing in the lineage extract but not the schema extract are flagged external rather than silently invented — the graph knows where its knowledge stops.

The same table, seen from two sides

Cross-estate reconciliation makes the composition explicit. Govern both a dbt project and a Databricks estate in one CoreModels project, and a reconcile operation finds the datasets the two vendors describe as the same physical relation — matched by their database.schema.table names — and links each pair with reciprocal sameAs assertions at dataset and field level.

The transformation-layer view and the storage-layer view of orders stop being two coincidentally similar records and become one governed entity seen from two angles. The operation is idempotent: re-running it refreshes the links, which matters because a reconciliation you are afraid to repeat is one you will not repeat.

Exits, in every direction

Neutrality is only credible if it is bidirectional.

Going back out to Databricks, the governed model generates Delta DDL: CREATE TABLE IF NOT EXISTS … USING DELTA with not-null from governed checks, one informational primary key per table, foreign keys from governed references, and governed allowed values carried on column comments. The same governed model generates dbt contract files for the transformation layer, and registry-ready Avro schemas for the streaming layer. One agreement, spoken in each system's native dialect.

Going out to no vendor at all, the governed model exports to open formats whose published specifications we implement — JSON Schema, Avro, LinkML, OWL, ShEx, Bitol's Open Data Contract Standard, Apache Ossie semantic models, Protocol Buffers, SQL DDL. Your meaning is no more trapped in our shape than in anyone else's, and every export declares its own lossiness rather than pretending the target could hold everything.

Why neutrality matters even if you are all-in on Databricks

The obvious objection is that a single-platform shop does not need a neutral layer. Three reasons we think the opposite.

Your table is already governed by more than one tool. You do not need a second platform to have a coordination problem — the transformation tool, the orchestrator and the catalog each hold a partial claim about orders today. Neutrality is not preparation for a hypothetical future; it is a description of your present.

Platform truth and agreed truth run on different clocks. Unity Catalog's contents change every time a job runs; that is what an operational catalog should do. An agreement should change only when people decide it should. Keeping it in a layer no pipeline can rewrite — where import is additive and disagreement surfaces for review — is a property the operational catalog cannot provide, because reflecting what just happened is its purpose.

The referee should not be on a team. We read Unity Catalog through artifacts you extract and upload, never through credentials we hold, and we write back only reviewable artifacts, never direct changes. CoreModels is not in your data path, does not replace your catalog, and does not touch your access control — which leaves it free to occupy the seat no platform tool can: the disinterested keeper of the agreement, with no stake in any one system's version being the real one.

Unity Catalog gives your lakehouse an excellent voice. A stack with six voices still needs a score. The Databricks Unity Catalog quickstart in the CoreModels docs is a practical way to bring the first one under one.