Confluent Schema Registry to CoreModels: Zero to First Audit

Your Schema Registry already knows every event contract in your streaming platform — every subject, every version, every Avro record your producers have ever registered. What it does not know is what those contracts mean, whether the meaning is still what your consumers agreed to, and who is guarding it. In this tutorial we take a real registry from nothing to a completed schema audit in CoreModels: export the subjects with one shell loop, import the estate into a governed project, run the first audit, and read the result line by line.

The vendor key is confluent, and the connector supports all three capabilities: Import, Audit, and Generate. Everything below works from a single JSON file you export yourself — CoreModels never connects to your registry and never holds a registry credential.

What you need

• A Confluent Schema Registry you can reach with curl (self-managed or Confluent Cloud — any registry that serves the standard /subjects REST API).
• A CoreModels project to govern the estate, identified by its 32-character hex project id — $PROJECT_ID below.
• A CoreModels login token ($TOKEN). Import requires the Admin role on the project; the audit only needs Viewer.
• curl and jq.

The API base URL is written as https://coremodels.example.com throughout — substitute your deployment's host.

Step 1 — Export the subjects

The connector takes exactly one artifact, named subjects: a JSON array of the registry's own GET /subjects/{subject}/versions/latest responses. Each row is the shape the registry already returns — {subject, version, id, schemaType, schema}. One loop produces it:

# Export every subject's latest schema from the Schema Registry:
for S in $(curl -s $REGISTRY/subjects | jq -r '.[]'); do
  curl -s "$REGISTRY/subjects/$S/versions/latest"
done | jq -s '.' > subjects.json

If your registry requires authentication, add your usual auth flags to the two curl calls — those credentials stay entirely on your side of the fence. That is the design, not an accident: you upload a file the registry API already produces, and nothing else.

Take a moment to look at subjects.json. Avro subjects (the registry default) carry their record schema as an escaped string in the schema field; JSON Schema and Protobuf subjects declare themselves in schemaType. All of them go in the same file — the connector sorts out what it can parse and tells you honestly about the rest.

Step 2 — Import the estate

The import call sends the artifact content as a string inside a JSON body. Since subjects.json is itself JSON, it must be escaped as a string value — jq --rawfile does that correctly:

jq -n --rawfile subjects subjects.json \
  '{artifacts: {subjects: $subjects}}' > import-request.json

curl -sS -X POST \
  "https://coremodels.example.com/graph/integrations/confluent/import/$PROJECT_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  --data-binary @import-request.json

A successful import answers with counts plus two honesty channels:

{ "success": true, "vendor": "confluent", "projectName": "schema-registry",
  "datasetsAdded": 7, "datasetsSkippedExisting": 0, "fieldsAdded": 55,
  "lineageEdgesAdded": 0, "lineageEdgesSkipped": 0, "nodesEnriched": 62,
  "snapshotStored": true, "lossiness": [], "errors": [] }

How to read it:

• datasetsAdded — record-shaped subjects that became governed Types. For an Avro subject the Type takes the record's name as its label, the topic as its physical name, and the key-or-value role, subject version, and schema id as metadata. Fields became Elements (fieldsAdded) with Avro's own optionality — a field with no union-null branch is required. Avro enums became governed Taxonomies, and named-record references were resolved across subjects into governed references.
• datasetsSkippedExisting — import is additive. Anything already governed is left exactly as it was; re-importing never mutates or deletes governed nodes. Changes surface through the audit instead, because applying a meaning change is a human act.
• snapshotStored: true — the parsed registry snapshot was persisted, which is what enables one-call re-audits later without exporting fresh artifacts.
• lossiness — everything the import approximated or dropped, stated explicitly. Protobuf subjects and bare-primitive key schemas (a plain "string" key, say) carry no record structure to import; they are counted and reported here, never silently discarded. errors means could-not-proceed.

Step 3 — Run the first audit

The audit compares a fresh export against the governed model. It is strictly read-only and runs at Viewer role; setting recordHistory: true additionally appends the run to the project's rolling audit trail, which is opt-in bookkeeping rather than governed meaning:

jq -n --rawfile subjects subjects.json \
  '{artifacts: {subjects: $subjects}, recordHistory: true}' > audit-request.json

curl -sS -X POST \
  "https://coremodels.example.com/graph/integrations/confluent/audit/$PROJECT_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  --data-binary @audit-request.json > audit-response.json

Auditing the same file you just imported is not a pointless exercise — it is the baseline. Coverage findings show what remains ungoverned, conformance findings show registry hygiene issues that existed before governance did, and the run's zero-drift result becomes the reference point every future export is compared against.

Step 4 — Read the result

The response is machine-readable counts plus findings plus a PR-ready markdown report:

jq '{errorCount: .errorCount, warningCount: .warningCount, infoCount: .infoCount,
     codes: .codes}' audit-response.json
jq -r '.markdown' audit-response.json

Findings arrive in three sections. Coverage flags parts of the export not yet governed (dataset-unmapped, field-unmapped). Drift flags governed facts the registry no longer honors — dataset-removed, field-removed, field-type-drift, contract-drift, and the enum family enum-constraint-removed, enum-narrowed, enum-widened, which is where Avro enum symbol changes land. Conformance carries the connector's streaming-hygiene rules. A typical first-run finding:

{ "section": "Conformance", "severity": "Info", "code": "fields-no-doc",
  "subject": "orders-value",
  "message": "3 field(s) carry no doc — consumers and agents are guessing their meaning.",
  "detail": "discount_code, channel, region" }

fields-no-doc aggregates undocumented fields per value subject. It is Info severity — it will never fail a build — but it is usually the most immediately useful output of a first audit: a precise, per-topic list of the fields whose meaning lives only in someone's head. If your export contained Protobuf subjects you will also see protobuf-unparsed, an honest coverage statement that those subjects were counted but not parsed.

The number that matters going forward is errorCount. Zero means the registry and the governed model agree. Greater than zero means real drift against governed meaning — and that same field, served on the machine-to-machine API surface, is what a CI gate checks before a schema change reaches production consumers.

Where you are now

In four commands you have a governed model of your streaming estate: subjects as Types, fields as Elements with real optionality, enums as Taxonomies, cross-subject references as governed relationships — and a baseline audit recorded in the project's history. You can confirm the import state at any time:

curl -sS "https://coremodels.example.com/graph/integrations/confluent/status/$PROJECT_ID" \
  -H "Authorization: Bearer $TOKEN"

From here, the loop closes in both directions: re-run the audit whenever the registry changes, re-audit the stored snapshot whenever the governed model changes, and generate registry-ready .avsc schemas back out of governance. The full HTTP surface, the CI gate recipe, and the agent-facing MCP tools are each covered in the Confluent Schema Registry quickstart in the CoreModels docs.