dbt logoDeep dive

How a dbt Estate Maps into the Governed Graph

`model.jaffle_shop.orders` is a node in dbt's manifest. After a CoreModels import it is a governed Type with a vendor identity, a metadata mixin carrying its native types and tests, lineage edges to its parents, and — if a semantic model projects it — membership in a Component. This article is the precise account of that mapping: what each dbt construct becomes, what rides the metadata mixin, how native types are approximated (and where the exact string survives), which audit rules the connector contributes, and where the mapping is honestly lossy.

How a dbt Estate Maps into the Governed Graph

model.jaffle_shop.orders is a node in dbt's manifest. After a CoreModels import it is a governed Type with a vendor identity, a metadata mixin carrying its native types and tests, lineage edges to its parents, and — if a semantic model projects it — membership in a Component. This article is the precise account of that mapping: what each dbt construct becomes, what rides the metadata mixin, how native types are approximated (and where the exact string survives), which audit rules the connector contributes, and where the mapping is honestly lossy.

What the parser reads

The connector parses manifest.json (required; manifest schema v10–v12), catalog.json (optional), and semantic_manifest.json (optional) into a neutral estate snapshot: datasets with fields and normalized checks, lineage edges, and projections. Parsing is open-world by construction — dbt's v12 manifest gains fields without version bumps, so unknown keys are ignored rather than fatal, and oddities become declared lossiness rather than failures. Only an unusable manifest (invalid JSON, or valid JSON not shaped like a manifest, or a manifest with no models, seeds, snapshots, or sources at all) produces a hard error.

Tolerance is specific, not vague. Both v10 string-array refs and v11+ structured refs parse. Versioned model ids (model.myproject.orders.v2) resolve to their model name. Disabled nodes are counted and skipped. Ephemeral models are imported (generation later skips them — they cannot carry contracts). unit_tests and saved_queries are counted as facts but not modeled.

Datasets, fields, and identity

dbt constructBecomes in the graph
model / seed / snapshot / sourcea Type node, Label = the native name
columnan Element on its Type, with a primitive data type
accepted_values testa Taxonomy (controlled list) referenced by the column
relationships test / FK constraintan expected-type reference to the target Type
unique / not_null testsnormalized checks, recorded in the metadata mixin
parent_map entrya Depends On lineage relation
semantic model / exposurea Component (projection) with membership edges

Every imported Type carries its vendor identity as a queryable mapsTo value — standard dbt, URI the manifest's unique_id (e.g. model.jaffle_shop.orders). That identity is what lets an audit line up a fresh manifest against the governed graph, and it round-trips: generation uses it to recover the dbt model name (last id segment, with version suffixes like .v2 handled).

Beside the schema-shaped facts, each node gets a dbt Metadata mixin value refreshed on import: the unique id, resource kind, materialization, the physical database.schema.identifier, the model's access level, the contract-enforced flag, tags, meta, description, and the checks as JSON. This is the estate-bookkeeping channel — refreshing it on re-import is allowed precisely because it is bookkeeping, while the governed nodes themselves are never mutated by import. A per-project state node additionally records versions, generation timestamp, fingerprint, counts, and facts such as the adapter type and how many metrics, exposures, semantic models, and tests the manifest declared.

Type mapping: family-based, honestly approximate

dbt's data_type strings are whatever the target warehouse uses, so the connector maps them family-by-family into graph primitives, with an explicit approximation flag whenever precision, length, or structure is lost:

  • NUMBER(38,0), int, bigint, serial, byteintInteger (a NUMBER with parameters drops its precision bound — approximated). Bare Snowflake NUMBER defaults to NUMBER(38,0) and maps to Integer; bare NUMERIC/DECIMAL (arbitrary precision) maps to Double.
  • numeric(38,2), decimal with nonzero scale, float, doubleDouble (exact decimal to binary double is recorded as a type approximation).
  • varchar(n), char(n)String, approximated because the declared length is dropped; bare text/string/clob map cleanly.
  • date, datetime, timestamp, timestamp_tz variants → DateTime, clean.
  • timeString, approximated (no time-of-day slot in the type system).
  • json, variant, array, geography, anything unrecognized → String, approximated. A missing data_type is also an approximated String — a guess, and flagged as one.

The crucial guarantee: the exact native string always survives in the dbt Metadata mixin. Approximation affects the governed primitive used for cross-format reasoning; it never destroys the warehouse truth, and generation prefers the recorded native type over any fallback when emitting contracts.

Tests become checks — with dbt's quirks handled

Generic tests attach to their dataset and column via attached_node + column_name, never via depends_on — a relationships test depends on two models but belongs to one, and using dependencies would attach it to both. Singular tests fall back to their first dataset dependency at dataset level. A column that exists only in tests (never documented under columns:) is materialized as a field with a lossiness record saying so.

relationships.to accepts ref('model'), ref('project','model'), or a bare name, resolved against the snapshot; an unresolvable target drops the reference but keeps the detail. One boundary is handled rather than papered over: the graph schema does not enforce required/nullable, so not_null becoming "Required" is stored as an element-facts mixin value — visible on the node, not enforced by the graph — and the check itself is preserved verbatim in the mixin's checks JSON, which generation reads back. Nothing is lost; it just lives in the bookkeeping channel instead of the schema channel.

Catalog merge: declared intent vs. warehouse reality

When you supply catalog.json, warehouse-real column types merge into the snapshot under three rules: a column with no declared type gets the warehouse type; a column whose declared and warehouse types disagree keeps both — the warehouse type is recorded as dbt.catalogType metadata so the audit can surface the mismatch; and a column the warehouse has but the manifest never documented is imported flagged dbt.catalogOnly. The import takes sides with neither dbt nor the warehouse — it records the disagreement and lets the audit report it.

Lineage and projections

parent_map (with a depends_on.nodes fallback) becomes Depends On relations between Types, each edge valued dbt.ref or dbt.source so you can distinguish model-to-model edges from source reads. dbt-Mesh cross-project refs that dangle outside the manifest are kept as external lineage in the snapshot and reported as semantic narrowing — they are never written as graph edges to nodes that don't exist.

Semantic models and exposures become Components: a view node per projection, with In-View membership edges to the projected model's Type and to the field Elements its entities, dimensions, and measures reference. An expr that is a plain identifier resolves to its column; anything more complex is recorded in the projection's metadata rather than mis-resolved. Metric names defined over a semantic model ride that projection's metadata (dbt.metrics); exposures carry their type, owner, and URL and project whole models. The audit reports projection coverage (projection-unmapped findings) alongside dataset coverage. Re-import is idempotent here too.

The connector's audit rules

Beyond the core coverage and drift checks, the dbt connector contributes four conformance rules, each with a stable code:

CodeSeverityFires when
contract-not-enforcedError for public models, Info otherwisea non-ephemeral model has no enforced contract — a public model is cross-team API surface, so its consumers have no schema guarantee
contract-column-missing-typeErrora contract-enforced model has a column with no data_type — dbt itself fails this at parse
key-column-untestedWarninga column named id or *_id has no tests and no reference
source-no-freshnessInfoa source declares no freshness policy — upstream drift will surface as broken models, not alerts

The severity split on contract-not-enforced follows the model's recorded access; ephemeral models are exempt because dbt contracts cannot apply to them.

Generation reads it all back

The generate verb inverts the mapping into models/coremodels_contracts.yml: enforced contracts with per-column data_type (the recorded native type when the model round-tripped from dbt, otherwise a warehouse-neutral type from the primitive), not_null constraints and unique/not_null tests from the checks JSON, accepted_values from governed taxonomies, and relationships from governed references. For the orders model this trace started with, the artifact is ordinary dbt schema YAML, shaped like this excerpt:

version: 2
models:
  - name: orders
    config:
      contract:
        enforced: true
    columns:
      - name: order_id
        data_type: number(38,0)
        constraints:
          - type: not_null
        tests:
          - unique
          - not_null
      - name: order_total
        data_type: number(38,2)
        constraints:
          - type: not_null
      - name: order_status
        data_type: varchar
        tests:
          - accepted_values:
              values: [placed, shipped, completed, returned]
      - name: customer_id
        data_type: number(38,0)
        tests:
          - relationships:
              to: ref('customers')
              field: customer_id
``` Two honesty rules are worth knowing. A `relationships` test needs a target field: the generator takes it from the original test's recorded detail, else falls back to the target model's unique-tested column — and if neither exists, it *omits* the test with a lossiness record rather than guessing a field name that would fail against a real warehouse. And models that cannot legally carry contracts — ephemeral ones, or types with no columns (dbt rejects a contract-enforced model without columns) — are skipped with declared lossiness, never emitted broken.

## Limits, stated plainly

- **No live connection, ever.** No warehouse, no dbt Cloud credential; the connector reads the build artifacts you send it and nothing else.
- **Snapshot cap.** The parsed estate snapshot stored at import (what powers artifact-free re-audits) is size-capped at roughly 1.5 MB encoded; beyond it, import reports `snapshotStored: false` with a lossiness record, and fresh-artifact audits remain fully functional.
- **Additive import.** Import never edits governed meaning — not even to "fix" drift it can see. Drift is the audit's finding to report and a human's decision to apply.
- **Metrics are metadata, not nodes.** Metric names ride their semantic model's projection metadata; modeling derived-metric graphs as first-class nodes is deliberately out of scope today.

Every approximation above is visible at runtime — in `lossiness` arrays on import and generate, in `dbt.catalogType`/`dbt.catalogOnly` metadata, in audit findings. The mapping's design rule is that nothing degrades silently. For the operational side of the same connector, start with the dbt quickstart that ships with CoreModels.