Inside the Airbyte Connector: Identity, Types, Checks, and the Limits We Publish

Every vendor CoreModels integrates parses into one neutral shape: datasets with fields and normalized checks, plus lineage between them. Vendor detail rides in metadata bags, never as new top-level concepts, and connectors are pure — they parse, map types and contribute audit rules, never touching the graph.

That constraint is what makes each connector's decisions worth reading. Airbyte's answer one question — what does a JSON-Schema-shaped ingestion catalog mean in a governed model? — in neutral vocabulary. Here they are, and then the limits.

One parser, two catalog shapes

A source discovery result (AirbyteCatalog) and a connection export (ConfiguredAirbyteCatalog) differ in one structural way: the configured form wraps each stream in an object carrying the connection's own choices. The parser treats that wrapper as optional — if an entry has a stream member, that is the stream and the entry is the configuration; otherwise the entry is the stream.

A configured catalog therefore produces sharper governance, because these keys exist only there:

Configuration key	Used for
sync_mode	recorded as metadata
destination_sync_mode	recorded as metadata
primary_key	takes precedence over source_defined_primary_key
cursor_field	takes precedence over default_cursor_field

The parser requires {"streams": [...]} with at least one entry, and at least one named entry. Anything else is a structured error rather than an empty import: Expected {"streams": [...]} with at least one stream. or No named streams found in the catalog.

Identity: what a stream becomes

Each stream becomes one dataset. The identity choice is the connector's most consequential:

• Vendor id — namespace.name when the stream declares a namespace, otherwise the bare name. This is the join key for coverage, drift, re-audit and cross-estate reconciliation.
• Name — the stream name, which becomes the governed Type's label.
• Kind — Source. A stream is ingestion, not a derived model.
• Materialization — the literal string stream.
• Physical name — the same namespace-qualified string as the vendor id.

Sync configuration is recorded, not promoted into governed meaning:

Metadata key	Source
airbyte.syncModes	supported_sync_modes, comma-joined
airbyte.sourceDefinedCursor	source_defined_cursor
airbyte.syncMode	the configured sync_mode
airbyte.destinationSyncMode	the configured destination_sync_mode
airbyte.cursorField	the first segment of cursor_field, else of default_cursor_field

Properties: three rules and a precedence

Each property under the stream's json_schema.properties becomes one field. Three rules decide what it means.

Nullability comes from the type array. Airbyte expresses optionality as a union: ["null", "string"]. A null branch is the optionality signal; the first non-null entry is the base type; a property with no type at all counts as nullable.

Not-null has two independent sources. A property is marked not-null when it appears in the stream's required array or when its type array has no null branch. The second half is deliberate: "type": "string" promises a value on every record, whether or not anyone listed it in required.

Native type follows a precedence. airbyte_type beats format, which beats type:

{
  "created_at": {
    "type": ["null", "string"],
    "format": "date-time",
    "airbyte_type": "timestamp_with_timezone"
  }
}

The recorded native type here is timestamp_with_timezone, the most specific statement available. Drop the airbyte_type and it becomes date-time; drop the format too and it becomes string. A declared enum becomes the field's accepted-value set, and description carries through.

Primary keys, including composite ones

Key paths arrive as arrays of arrays — [["id"]] for a single key, [["order_id"], ["line_no"]] for a composite one. The parser takes each path's head segment, so nested paths still resolve to a top-level column, and configured primary_key wins over source_defined_primary_key. What happens next depends on the arity:

• Single-column key — the column gets a Unique check named primary_key and, if it is not already not-null, a NotNull check.
• Composite key — every member column gets NotNull (no per-column Unique, because no single column is unique), and the dataset gets one custom check named composite_primary_key whose detail is the comma-joined member list, e.g. order_id,line_no.

That distinction is not cosmetic: marking each column of a composite key Unique would assert something false about the data, while a dataset-level check keeps the fact without the falsehood.

Native types to governed primitives

The connector maps native type strings to governed primitives, flagging approximations so the mapper records them:

Native type	Governed primitive	Approximated
integer, big_integer	Integer	no
number	Double	no
boolean	Boolean	no
date, date-time, timestamp_with_timezone, timestamp_without_timezone	DateTime	no
time, time_with_timezone, time_without_timezone	String	yes
string	String	no
anything else (object, array, formats like email, undeclared)	String	yes

Every approximation becomes a lossiness record on the import — for example:

{
  "kind": "TypeApproximation",
  "path": "public.users.email",
  "explanation": "Native type 'email' was approximated as String; the exact native type is preserved in the vendor metadata mixin."
}

Note the second clause: the approximation concerns only the governed primitive, and the audit compares against the verbatim native string kept on the metadata mixin.

What lands in the graph

Schema-shaped facts flow through the neutral intermediate representation into the graph's writer; estate facts it cannot hold are written natively by the enricher.

Estate fact	In the governed graph
stream	a Type, labeled with the stream name
property	an Element on that Type, required when a not-null check exists
enum values	a Taxonomy plus a controlled-list relation; the Taxonomy is labeled <stream> <property> values
vendor identity	a mapsTo value (standard airbyte, uri = the vendor id) — queryable, round-trips
native type, checks, tags, meta, materialization, physical name, description	one value per node on the Airbyte metadata mixin
last-import state	a state node carrying timestamp, fingerprint, counts and parser facts
parsed estate	a snapshot node — the baseline the artifact-free re-audit runs against

Taxonomies carry their own vendor identity, suffixed #values (public.users.status#values) — which is why the canonical-coverage metric counts more identified nodes than streams plus properties.

What rides the vendor-metadata mixin

Each vendor gets one metadata mixin type — here Airbyte Metadata — whose property ids are the vendor key plus the property name. Values are written add-or-update under deterministic ids, so re-imports refresh rather than duplicate. On a stream's Type node:

Property id	Content
airbyteUniqueId	the namespace-qualified vendor id
airbyteResourceKind	source
airbyteMaterialization	stream
airbyteRelationName	the physical name
airbyteMeta	the airbyte.* metadata bag as JSON
airbyteChecks	JSON of dataset-level checks, e.g. the composite key
airbyteContract	always false — a catalog has no contract-enforcement concept
airbyteAccess, airbyteTags, airbyteDescription	present for shape; empty when a catalog has no such concept

On a property's Element node: airbyteDataType (verbatim native type), airbyteDescription, airbyteChecks, airbyteTags, airbyteMeta.

This is the boundary that keeps the model honest: mixin values mirror the estate and are refreshed on every import — bookkeeping — while Types, Elements and Taxonomies are governed meaning that import never mutates once they exist.

The connector's audit rules

Three conformance rules, evaluated per stream:

Code	Severity	Fires when
stream-no-primary-key	Warning	no field carries a Unique check and the dataset has no composite_primary_key check
no-cursor-field	Info	supported_sync_modes contains incremental, no airbyte.cursorField was recorded, and source_defined_cursor is not true
untyped-fields	Info	one or more fields have a native type that is null, object or array; the detail lists their names

They sit alongside the neutral sections: Coverage (dataset-unmapped, field-unmapped) and Drift (dataset-removed, field-removed, field-type-drift, enum-narrowed, enum-widened, enum-constraint-removed, contract-drift). No Airbyte rule is Error severity — the connector reports hygiene, the drift engine reports violations.

How drift is actually compared

Three details of the comparison engine matter when reading a report.

Fields are matched by vendor identity, never by label. Renaming a governed element is normal stewardship, so removal is detected through the mapsTo identity instead.

Comparisons are normalized. Type strings fold whitespace and compare case-insensitively; enum sets compare case-insensitively too, reporting removed and added values separately (enum-narrowed at Error, enum-widened at Warning).

Dataset removal is namespace-scoped. One project may govern several estates of the same vendor, so dataset-removed fires only for governed datasets whose identity namespace still appears in the catalog being compared — the namespace being everything before the last dot, or the whole string when there is no dot. The consequence for Airbyte deserves stating outright: a stream with no namespace that disappears is not reported as removed, because its namespace was its own name and that name is gone. Namespaced streams compare normally. If removal detection matters to you, namespace your streams.

Limits, stated plainly

No lineage, no references. A catalog describes streams independently, with nowhere to declare that one stream's field points at another's. An Airbyte import reports lineageEdgesAdded: 0, and no reference is ever invented.

Additive re-import has a stated cost. A re-import adds new properties to an already-governed stream but does not auto-create accepted-value semantics on them — changing governed meaning is a human act. It is reported as a ConstraintRelaxation record naming the field, with the facts on the mixin where the audit surfaces them.

Snapshots are capped. The parsed estate is persisted so re-audit can run without artifacts, under a size cap. Over it, import returns snapshotStored: false plus a lossiness record: fresh-artifact audits still work, stored-snapshot re-audit does not.

Identity collisions are refused, not merged. Node ids are sanitized alphanumeric strings, so two distinct vendor identities could in principle collapse onto one. Rather than silently merging two streams, the import fails with an error naming both sources and asking for a rename on the estate side.

No generation, no live connection. The connector declares Import | Audit; generate refuses explicitly rather than asserting authorship over a source we do not control. CoreModels never holds Airbyte credentials — every verb works from an artifact you exported.

Reconciliation is exact. Linking a stream to the warehouse table it lands in matches the recorded physical relation name, normalized only by stripping quotes, trimming and upper-casing. An Airbyte stream's physical name is namespace.name; if the counterpart records a three-part name, the pair is reported as unmatched rather than guessed.

Read that as design, not disclaimer: a layer that reports what it could not carry is one you can build a gate on.

For the extraction recipe and the calls that exercise it, see the Airbyte quickstart in the CoreModels integration docs.