Databricks logoGovernance

Trust Is a Set of Powers Refused

Most tools that want to govern a data platform begin by asking for access to it. A service principal, a warehouse role, a token with just enough scope — and from that moment the honest answer to "what can this thing do to my estate?" is "you'd have to read the code."

Trust Is a Set of Powers Refused

Most tools that want to govern a data platform begin by asking for access to it. A service principal, a warehouse role, a token with just enough scope — and from that moment the honest answer to "what can this thing do to my estate?" is "you'd have to read the code."

So the fastest way to understand how CoreModels governs a Databricks Unity Catalog estate is the list of things it structurally cannot do. Trust is not a tone of voice. It is a set of powers refused, in code, on purpose.

It cannot reach your workspace

There is no live connection, and the absence is deliberate. The connector's declared capabilities are import, audit and generate; live sync is documented as declared-but-deferred, so nobody has to wonder whether a credential path exists and is merely switched off.

What flows in are artifacts your team produces and can inspect before sending: JSON results from documented queries run in the Databricks SQL editor. One extract from system.information_schema is required; two are optional — flattened primary-key and foreign-key constraint rows, and rows from system.access.table_lineage. When a security review asks what CoreModels sees, the answer is a file they can open.

It cannot overwrite what you already govern

Import is additive, and this is the most load-bearing invariant in the integration. Existing governed nodes are never mutated or deleted. Re-import a fresh extract and it adds what is new while reporting what it left alone — the response returns datasets added and datasets skipped as already-governed, side by side, in the same object.

Exactly one class of thing gets refreshed on every run: the vendor-metadata values riding on governed nodes — the verbatim full_data_type, the recorded checks, the materialization, the physical name — plus the record of the last import itself. That is estate bookkeeping, not governed meaning, and we name the distinction rather than letting it blur.

Roles back the invariant up. Import requires the Admin project role. Audit, re-audit, generate, status, history and badge all run at Viewer, because none of them writes governed meaning.

It cannot decide what a column means

Audit and generate are read-only verbs. The audit does not repair drift, does not file anything, and does not update the model to match reality. It reports.

Even recording a run is opt-in: recordHistory defaults to false, so the audit stays strictly read-only unless you ask for the run to be kept. One verb always records — re-audit, whose entire purpose is the trail — and the documentation says so rather than letting you discover it.

Which leaves the decision where it belongs. When estate and agreement disagree, a person reads the finding and chooses: the estate is wrong and gets fixed, or the meaning genuinely moved and someone with authority moves the governed model. The third option — the system quietly resolving the disagreement in favor of whatever the pipeline did last night — is not implemented, and that is the point.

It tells you what it approximated

A successful operation can still have lost something, and saying so is a first-class channel here, not a footnote. Lossiness records travel beside successful results in one shape — a kind, a path, a plain-language explanation — across four kinds: something dropped structurally, a type approximated, a constraint relaxed, meaning narrowed.

Three ordinary examples.

Composite keys are never faked. When a primary key spans several columns, its members are not individually unique, so we never record them that way: members get not-null checks and the table carries a composite-key marker. A convenient lie about uniqueness is precisely the sort of subtle wrongness governance exists to prevent.

Generated DDL declares what it skipped. Views are derived objects, so DDL generation covers tables only — and each skip comes back as a lossiness record, so the output never silently covers less than you assumed.

The snapshot store refuses honestly. Import persists the parsed estate so it can be re-audited later without fresh artifacts. On very large estates that snapshot exceeds the storage cap, and the import then returns snapshotStored: false with a record explaining the consequence: fresh-artifact audits still work; re-audit has no stored estate to run against. An honest refusal beats a silent degradation.

Drift is evidence, not an alarm

An alarm interrupts you and then vanishes. Evidence accumulates and can be cited.

Every audit finding is a small structured record: a section — coverage, drift or conformance — a severity, a stable kebab-case code, the exact subject, a message, and a detail carrying the specifics, such as the governed type beside the artifact's type. The response adds per-code counts, the drifted subjects, a fingerprint of the artifact content tying the run to one exact estate state, and a markdown report written for a pull-request comment.

Recorded runs accumulate into a rolling trail per vendor project, each entry keeping its counts, codes, fingerprint and trigger. The badge renders the latest: green clean, yellow warnings, red errors — and gray when nothing has been recorded at all. Gray is not green on purpose. A governance signal that reads "healthy" in the absence of evidence is worse than no signal.

Review, then deploy

The last question is how the governed model reaches the lakehouse. Not directly. Generation produces an artifact — one coremodels_delta_tables.sql script of CREATE TABLE IF NOT EXISTS … USING DELTA statements, with not-null from governed checks, one informational primary-key constraint per table, foreign keys from governed references, and governed allowed values on column comments.

That script goes into a pull request, is read by people who can reject it, and is deployed by your process with your permissions. Its header states the working agreement in two lines: generated by CoreModels, and meaning changes belong in CoreModels — regenerate rather than edit. Nothing here requires CoreModels to have standing in your deployment path, and it has none.

The posture, in one paragraph

Read only the artifacts you chose to share. Write additively, never over agreed meaning. Report honestly, including about our own limits. Keep every meaning-affecting act behind a human with a role. Emit changes as reviewable artifacts rather than applying them.

None of that is a promise about intent. Each is a property of the surface: which verb writes, which role it needs, what the response must disclose. You can check every one from outside — watch an import skip existing nodes, read a lossiness record, confirm an audit at Viewer role left nothing behind. Governance you cannot verify is branding.

The Databricks Unity Catalog quickstart in the CoreModels docs runs import, audit, re-audit and generate end to end, including an honest section on what is not supported by design.