Inside the BigQuery Connector: How an Estate Becomes a Governed Graph

What actually happens between a flat JSON extract and a queryable governed model — identities, type mappings, metadata mixins, audit rules, and the approximations we admit to.

A flat JSON artifact goes in; a graph of Types, Elements, checks, and vendor metadata comes out. Everything the CoreModels (by ARAMAI) BigQuery integration does downstream — drift audits, CI gates, DDL generation, agent queries over MCP — depends on how faithfully that first translation is made, and on how honestly its limits are declared. This article is the map: exactly what the connector reads, exactly where each fact lands in the graph, and exactly what gets approximated on the way.

One artifact, deliberately flat

The connector consumes a single required artifact named information_schema: JSON rows of INFORMATION_SCHEMA.COLUMNS joined with TABLES, stitched with column descriptions from COLUMN_FIELD_PATHS and table descriptions from TABLE_OPTIONS by the documented extraction query (run per dataset; the quickstart has the SQL). Flat rows are a deliberate choice. They are trivially producible from the console or the bq CLI, they require no credentials to be shared with CoreModels — LiveSync is a declared-but-deferred capability precisely because we never hold your Google credentials — and they make the parse fully deterministic.

Parsing is open-world and order-aware. Rows are processed in ordinal_position order, so governed elements preserve column order. Rows without a table_name are skipped; duplicate column names within a table are ignored case-insensitively after the first. Only genuinely unusable payloads fail: an empty extract, invalid JSON, or rows with no table_name values at all, which earns the pointed error "not an INFORMATION_SCHEMA extract?".

Each distinct table_catalog.table_schema.table_name becomes one dataset in the neutral estate model with that dotted path as its vendor identity — my-gcp-project.analytics.orders — and its bare table name as the display name. table_type is interpreted twice: any variant containing VIEW marks the dataset as a view rather than a table, and the specific value is normalized into a materialization fact — view, materialized_view, external_table, or table. Table descriptions get one quirk handled at the door: TABLE_OPTIONS option values arrive wrapped in quotes, and the connector strips exactly one layer. Finally, is_nullable = "NO" becomes a normalized not_null check on the field — the one constraint BigQuery actually enforces.

The snapshot also records estate-level facts — table and view counts — and a fingerprint: the first 16 hex characters of a SHA-256 over the raw artifact text. That fingerprint follows the estate through audits and history, letting you prove two runs saw byte-identical extracts.

Where each fact lands in the graph

The import writes through two paths with one rule: schema shape goes through the same hardened writer every CoreModels import uses, and estate facts that are not schema shape ride the graph's extensibility instead of distorting it.

• Table/view → Type. A governed Type node whose label is the native name. Materialized views and external tables stay distinguishable via their materialization fact.
• Column → Element. One Element per column, in ordinal order, carrying the mapped primitive type (next section). Elements are per-dataset: the node id is the dataset id plus the PascalCased column name, so two tables with a status column stay two distinct governed Elements.
• Vendor identity → a mapsTo mixin value with standard bigquery and URI project.dataset.table. This is the join key of the whole system: it is how audits match estate objects to governed nodes, how re-imports know what already exists, and how cross-estate reconciliation later asserts that a BigQuery table and another vendor's model are the same physical relation via reciprocal sameAs values.
• Everything else vendor-specific → the Google BigQuery Metadata mixin. The exact native type string (NUMERIC(10,2), STRUCT<a INT64>, ...), the normalized checks, the materialization, the physical name, and the table and column descriptions ride this per-vendor metadata mixin — one value per node, deterministic ids. Descriptions live here because CoreModels has no schema-level description slot, and the mixin is also where generation reads them back from. Nothing vendor-flavored leaks into the governed schema itself, and nothing is thrown away either: the original data_type string survives verbatim even when the governed type is an approximation.
• Last-import state → a state node recording versions, timestamps, the fingerprint, counts, and the table/view facts. This is what the status route and the MCP status tool read.

Two things are conspicuously not written. BigQuery enforces no primary or foreign keys, so the connector reads none and — the important half — invents none: a key that exists only in a diagram is not governance. And the extract carries no lineage, so lineageEdgesAdded is always zero for this connector; lineage-bearing vendors populate the same graph seam, which is how a multi-vendor estate composes.

Import is additive on re-import: existing governed nodes are never mutated. The vendor-metadata mixin values are refreshed — that is estate bookkeeping, not governed meaning — but meaning changes are a human act, surfaced by the audit rather than applied by the importer.

Two structural rules are worth knowing before your first import. First, graph node ids must be camelCase alphanumerics, so vendor identities are sanitized on the way in — separators fold away and the original identity is preserved on the label, the mapsTo value, and the metadata mixin. The interesting consequence is that two distinct BigQuery identities can collapse onto the same sanitized id — shop.order_items and shop.orderItems, both legal side by side in BigQuery, both sanitize to shopOrderItems. Silently merging two tables would be the worst possible outcome, so the import refuses instead, naming the colliding identities so you can rename one in the estate. Second, CoreModels has no schema-level required/nullable: the not_null check travels into the schema mapping as a requiredness flag, is persisted as an element-facts mixin value (visible in the product, not enforced by the graph), and survives verbatim in the mixin's checks JSON — which is where DDL generation reads NOT NULL back from.

The type mapping, including the uncomfortable rows

Native types are matched on their base name — anything before a ( or < — so NUMERIC(10,2) maps as NUMERIC and ARRAY<STRING> as ARRAY:

BigQuery type	Governed as	Exact?
INT64, INT, INTEGER, SMALLINT, BIGINT, TINYINT, BYTEINT	Integer	yes
FLOAT64, FLOAT	Double	yes
NUMERIC, BIGNUMERIC, DECIMAL, BIGDECIMAL	Double	approximated — exact decimal to binary double
BOOL, BOOLEAN	Boolean	yes
DATE, DATETIME, TIMESTAMP	DateTime	yes
STRING	String	yes
everything else — BYTES, TIME, STRUCT<...>, ARRAY<...>, JSON, GEOGRAPHY, INTERVAL, RANGE	String	approximated

The "approximated" column is the connector's honesty made structural. Approximations are not warnings buried in a log; they are declared lossiness records on a successful import — lossiness is a success channel throughout CoreModels, distinct from errors, which mean could-not-proceed. And because the verbatim native type rides the metadata mixin, an approximation never destroys information: BIGNUMERIC is governed as Double for cross-format reasoning, while the string BIGNUMERIC remains queryable and is what DDL generation will emit back.

The audit rules BigQuery contributes

Every vendor audit shares the core engine — Coverage (dataset-unmapped, field-unmapped), Drift (dataset-removed, field-removed, field-type-drift, enum-constraint-removed, enum-narrowed, enum-widened, contract-drift) — and each connector adds conformance rules that encode vendor-specific good practice. BigQuery adds two, with stable codes:

• table-no-description (Info) — the table or view has no description. Undocumented datasets resist governance and give AI agents nothing to ground on; because the extraction query joins TABLE_OPTIONS, fixing this in BigQuery fixes it in the next audit automatically.
• semi-structured-column (Warning) — an aggregate finding per table: it counts the STRUCT/ARRAY/JSON columns and lists their names in the finding's detail. This is the type-mapping table's bottom row promoted to a governance signal: those columns carry inner schemas the flat extract cannot see, so their contents are invisible to consumers, contracts, and agents alike. The remedies are real modeling decisions — govern the exploded fields, or accept the opacity knowingly.

Neither rule produces an error, so neither fails a CI gate; errors are reserved for violated governed meaning, which comes from the drift rules.

Generation: meaning flows back as DDL

BigQuery is a full-loop connector. From the governed graph, generate emits one artifact, coremodels_bigquery_tables.sql, and its design choices mirror the import's honesty. A representative output:

-- Generated by CoreModels — governed BigQuery table definitions.
-- BigQuery enforces no keys: governed references and value sets ride the descriptions.
-- Meaning changes belong in CoreModels; regenerate this script rather than editing it.

CREATE TABLE IF NOT EXISTS `my-gcp-project.analytics.orders` (
    order_id INT64 NOT NULL OPTIONS(description='Order identifier. References Customer.'),
    status STRING NOT NULL OPTIONS(description='Order state. Allowed values: draft, active, closed.'),
    order_total FLOAT64,
    created_at TIMESTAMP NOT NULL OPTIONS(description='Creation time (UTC).')
)
OPTIONS(description='One row per customer order.');

Table names come from the physical name recorded at import when there is one, otherwise from a sanitized form of the governed label. Column types prefer the verbatim native type preserved in the metadata mixin; for governed-only columns the reverse map applies — Integer to INT64, Double to FLOAT64, Boolean to BOOL, DateTime to TIMESTAMP, everything else STRING, with taxonomy-governed columns as STRING and reference columns as INT64 carrying the key value. NOT NULL comes from governed checks. And because BigQuery enforces no keys or CHECK constraints, governed taxonomies ("Allowed values: ...") and references ("References ...") are written into the column descriptions — the place BigQuery users actually read — never faked as constraints that would not hold. Views are skipped with a declared lossiness record (they are derived objects), and string literals are escaped for BigQuery's backslash-and-quote rules.

The limits, plainly

Worth knowing before you rely on the loop. The extraction query runs per dataset, so a multi-dataset estate is several extracts — additive import makes that a non-issue, but plan the loop per dataset. Very large estates can exceed the snapshot storage cap (~1.5 MB encoded): import then reports snapshotStored: false with a lossiness record, fresh-artifact audits keep working, but the one-call reaudit verb has no stored snapshot to replay. NUMERIC precision, TIME, BYTES, GEOGRAPHY, and the semi-structured family are all governed as approximations — declared, preserved verbatim in metadata, and in the semi-structured case actively flagged by the audit. And there is no live GCP connection at all, by design: every verb in this article operates on artifacts you extracted yourself.

That is the whole translation, in both directions, with the losses on the record. For the extraction recipe and the first import-and-audit walkthrough, see the Google BigQuery quickstart in the CoreModels documentation.