Anatomy of a Governed Streaming Estate: How the Confluent Connector Maps Subjects into the Graph

A schema registry is three naming systems wearing one trench coat. There is the subject (orders-value), the registry's versioned unit. There is the record (com.acme.Order), the Avro type the schema declares. And there is the topic (orders), the physical stream the subject-name strategy encodes. Most tooling collapses these into one string and loses information doing it. This deep dive walks exactly how CoreModels maps a Confluent Schema Registry export into the governed graph — what becomes a Type, an Element, a Taxonomy or a reference; what rides metadata; which audit rules are this vendor's; and where the mapping is lossy, because a governance product that hides its own approximations cannot be trusted about anyone else's.

The artifact and the first sorting pass

The connector consumes one artifact, subjects: a JSON array of the registry's GET /subjects/{subject}/versions/latest responses, rows of {subject, version, id, schemaType, schema}. The parser makes a first pass to sort rows into four buckets:

• Avro record schemas (schemaType absent or AVRO, schema of type record) — parsed fully. The registry's default, and the connector's best case.
• JSON Schema subjects (schemaType: "JSON", an object schema) — parsed at depth 1: properties, required, enum, format, description.
• Protobuf subjects — counted, never parsed. Each produces a declared lossiness record, and the count is stored as an estate fact.
• Non-record schemas — a bare "string" or primitive union, typical for key subjects. Nothing structural exists to import; each is counted and reported as lossiness.

Beyond an unusable or empty payload, the parse fails outright only when no subject in the file yields any structure at all — No record-shaped subjects could be parsed. Everything else proceeds, oddities in the lossiness ledger rather than swallowed. The export is fingerprinted (a 16-character content hash) so audits can say "same input" with certainty.

Subjects become Types — with all three names kept

Each parsed subject becomes a governed Type, and the trench coat comes off:

Registry fact	Where it lands in the graph
subject name (orders-value)	the Type's vendor identity — a queryable mapsTo mixin value with standard confluent and the subject as URI
Avro record name (Order)	the Type's label
topic (orders, from stripping -key/-value)	the Type's physical name
key-or-value role, subject version, schema id, Avro namespace, schema type	the vendor-metadata mixin (see below)

The -key/-value suffix does real work: it derives both the physical topic name and a role of key or value, which the audit rules use to hold value subjects — where the payload contract lives — to documentation hygiene.

Fields become Elements, and optionality is Avro's own

Every Avro field becomes an Element on its Type. The subtlety is where required-ness comes from: the union. A field whose type has no null branch always carries a value, so the parser derives a NotNull check from the absence of a union-null branch rather than inventing a convention. ["null", "string"] is optional; "string" is required. Field doc strings become descriptions, and field defaults are preserved verbatim as metadata.

Three more Avro constructs map structurally:

• Enums become Taxonomies. An Avro enum's symbols become the accepted values of a governed Taxonomy, linked to the Element through a controlled-list relation. This is what later powers the enum drift family in audits — symbol changes are compared against governed terms, not string-matched schemas.
• Named-record references resolve across subjects. The parser indexes every record name in the export — both bare (Order) and namespace-qualified (com.acme.Order) — and when another subject's field is typed by that record name, the field gets a governed reference to the target subject's Type — your event model's shape, reconstructed from the registry alone.
• Logical types are honored. date, timestamp-millis, timestamp-micros, and their local- variants map to the governed DateTime primitive exactly.

For JSON Schema subjects the same machinery runs one level deep: properties become Elements, membership in required becomes the NotNull check, enum arrays become accepted values, and format (falling back to type) supplies the native type.

The type map, with its approximations on the label

Native registry types map to governed primitives like this:

Native type	Governed primitive	Exact?
int, long, integer	Integer	exact
double, number	Double	exact
float	Double	approximation, recorded
boolean	Boolean	exact
string	String	exact
date, timestamp-millis, timestamp-micros, local-timestamp-millis, local-timestamp-micros, date-time	DateTime	exact
time-millis, time-micros	String	approximation — time-of-day has no governed slot
bytes, fixed, decimal, uuid	String	approximation, recorded
array<T>, map	String	approximation — the collection shape survives in the native-type metadata

Every approximation is flagged as such, and the original native string (array<string>, enum(OrderStatus), record(Customer)) survives in metadata: the governed primitive is a working approximation, the vendor truth is never destroyed.

What rides the vendor-metadata mixin

Governed meaning — types, elements, taxonomies, references — goes through the same hardened writing path every CoreModels import uses. Estate facts that are bookkeeping rather than meaning ride a per-vendor metadata mixin, one value per node, refreshed on re-import:

• the native type string per field, and the checks (the NotNull derivations) as structured JSON;
• per-subject: confluent.schemaType, confluent.role (key/value), confluent.version, confluent.schemaId, and confluent.namespace for namespaced Avro records;
• per-field: confluent.default, the verbatim Avro default value;
• descriptions from doc strings, the topic as physical name, and the materialization kind (subject);
• a state node recording the last import — timestamps, fingerprint, and the parser's estate facts: total subjects, imported, Protobuf count, unparsed count.

This split is a posture, not an implementation detail. Import is additive: re-importing never mutates or deletes governed nodes; only the metadata mixin values refresh, because estate bookkeeping is not governed meaning. One consequence worth knowing: the connector emits no lineage edges — a registry export declares contracts, not dataflow — so lineageEdgesAdded is honestly 0, and streaming lineage comes from pairing this connector with an orchestration or pipeline connector in the same project.

The audit rules, vendor and core

Audits compare a fresh export (or the stored snapshot) against the governed model in three sections. The core engine contributes coverage (dataset-unmapped, field-unmapped) and drift (dataset-removed, field-removed, field-type-drift, contract-drift, enum-constraint-removed, enum-narrowed, enum-widened) — the enum family doing real work for Avro estates, where symbol evolution is routine and mostly unreviewed.

The connector adds two rules of its own:

Code	Section	Severity	What it means
fields-no-doc	Conformance	Info	Aggregated per value subject: fields carrying no doc. The message counts them; the detail names them. Key subjects are exempt — the payload contract is where documentation debt hurts.
protobuf-unparsed	Coverage	Info	The export contained Protobuf subjects; they were counted, not parsed — that part of the estate is ungoverned here, and the audit says so every time rather than letting the coverage gap fade from memory.

Both are Info by design: they never fail a CI gate, and never stop being visible.

Generation: the loop closes, with conditions

The connector also runs in reverse, emitting one registry-ready schemas/{Record}.avsc per governed Type. The generation rules mirror the parsing rules deliberately:

• a NotNull check makes the field a bare Avro type; its absence makes it ["null", T] with default: null — the union-null convention, round-tripped;
• governed Taxonomies become Avro enums with sanitized, uppercased symbols (Avro names must match [A-Za-z_][A-Za-z0-9_]*);
• governed DateTime becomes long with logicalType: timestamp-millis;
• a governed reference becomes a named record type only when the target record is generated in the same call — otherwise the field degrades to a string key with a recorded semantic-narrowing note, because a dangling record name would produce a schema the registry rejects;
• vendor descriptions become doc strings; absent one, the record carries a stamp telling a human to regenerate rather than hand-edit.

Types with no elements are skipped with declared lossiness — an Avro record requires at least one field.

One subject, both directions

One row of a subjects.json export — an orders-value subject whose record carries a documented required string, a logical timestamp, an enum, and an optional cross-subject reference:

{
  "subject": "orders-value",
  "version": 3,
  "id": 42,
  "schemaType": "AVRO",
  "schema": "{\"type\":\"record\",\"name\":\"Order\",\"namespace\":\"com.example.events\",\"fields\":[{\"name\":\"order_id\",\"type\":\"string\",\"doc\":\"Business key\"},{\"name\":\"placed_at\",\"type\":{\"type\":\"long\",\"logicalType\":\"timestamp-millis\"}},{\"name\":\"status\",\"type\":{\"type\":\"enum\",\"name\":\"OrderStatus\",\"symbols\":[\"DRAFT\",\"PAID\",\"SHIPPED\"]}},{\"name\":\"customer\",\"type\":[\"null\",\"Customer\"],\"default\":null}]}"
}

Imported, that becomes a Type labeled Order (vendor identity orders-value, physical name orders) with Elements order_id (String, required, described "Business key"), placed_at (DateTime, required), status (required, governed by a Taxonomy of DRAFT, PAID, SHIPPED) and customer (optional, a governed reference to the subject declaring the Customer record). Generated back out — with the Customer Type generated in the same call, so the reference can stay a named record — the same Type emits schemas/Order.avsc:

{
  "type": "record",
  "name": "Order",
  "namespace": "coremodels.generated",
  "doc": "Governed by CoreModels — regenerate rather than hand-edit.",
  "fields": [
    { "name": "order_id", "doc": "Business key", "type": "string" },
    { "name": "placed_at", "type": { "type": "long", "logicalType": "timestamp-millis" } },
    { "name": "status", "type": { "type": "enum", "name": "statusValues", "symbols": ["DRAFT", "PAID", "SHIPPED"] } },
    { "name": "customer", "type": ["null", "Customer"], "default": null }
  ]
}

Two round-trip facts read off that output: the generated record sits in the coremodels.generated namespace — the subject's registry namespace rides the metadata as confluent.namespace — and the enum type is named from the governed element (statusValues), not the registry's OrderStatus, which survives as the field's native type string. Generation produces schemas the registry accepts, not byte-for-byte copies of your files.

The honest-limits ledger

Every mapping above has a boundary, and these are this connector's:

1. No live registry connection. You export; we parse. CoreModels never holds registry credentials — live sync is declared but deferred.
2. Protobuf subjects are inventoried, not parsed. Counted at import, reported by protobuf-unparsed on every audit.
3. JSON Schema parses at depth 1, and bare-primitive key schemas carry no structure at all — counted, never decomposed.
4. Time-of-day, decimals, UUIDs, bytes, arrays, and maps approximate to String, with the native type preserved in metadata and the approximation recorded.
5. Very large registries can exceed the stored-snapshot cap (roughly 1.5 MB encoded); import then reports snapshotStored: false with a lossiness record — fresh-artifact audits are unaffected, only snapshot-based re-audit is unavailable.

None of these limits is discovered by surprise: each announces itself in the lossiness channel or an audit finding at the moment it applies. That is the principle underneath the whole mapping — the graph holds what the registry means, the metadata holds what the registry said, and the ledger holds the difference. For the export recipe and the calls end to end, see the Confluent Schema Registry quickstart in the CoreModels docs.