The Meaning Gap Under Every Kafka Topic

Ask your Schema Registry for its subject list and you get an inventory: a name per subject, each with a version, a schema id, a schema type, and a schema document. It is a complete answer to the question what shape is this?

Now ask a different question — which of these fields may a consumer depend on, and who agreed to that? — and the inventory goes quiet. Not because the registry is failing at its job, but because that fact was never stored anywhere.

That silence is the meaning gap — the reason we built a Confluent Schema Registry connector for CoreModels. It is worth being precise about the pain first, because it looks like a documentation problem when it is really a governance one.

What a compatibility check actually promises

Schema Registry is good infrastructure, and teams that run one are ahead of teams that don't. It gives every topic's schemas an identity and a version, lets binary Avro travel as a schema id instead of a full document, and stops producers from registering a version existing readers cannot decode.

Read that last promise carefully: cannot decode. A compatibility check is a statement about bytes and readers. It is not a statement about whether a change is a good idea, whether anyone downstream agreed to it, or what a field is for. A schema change can be perfectly compatible and completely wrong.

Four ways the gap bites

A field becomes optional. Someone wraps a field in a union with null because a new producer path can't always populate it. The registry accepts this — it is textbook schema evolution. Downstream, a service that has always treated that field as guaranteed starts receiving nulls. Nothing malfunctioned. A guarantee that only ever existed in people's heads simply stopped being true.

An enum grows. A new symbol appears in a status enum. Wire compatibility is untroubled. But three consumers branch on that enum, and every one of them routes the unfamiliar value into a default branch written for "impossible." Nothing throws. Records go the wrong way, quietly, at whatever rate the topic runs. The inverse hurts differently: when a symbol disappears while downstream logic still treats it as valid, the estate and the agreed vocabulary have forked without anyone noticing.

A field leaves. A record drops a field its producer no longer populates. The consumer that read it — perhaps a low-traffic reconciliation job nobody remembers owning — breaks on its next deploy, or keeps running against a default it should never have had.

Nobody wrote anything down. Value subjects carry dozens of fields with no doc strings, so meaning lives in two engineers' heads, an old chat thread, and the defensive parsing each consumer team maintains privately. A team onboarding to a topic reads another team's source code, because that is the only surviving record of what the events mean.

None of this is exotic; it is the ordinary weather of a streaming estate.

Streaming makes the gap expensive

In a warehouse, a drifted column produces a wrong number in tomorrow's report and an argument in a meeting. In a stream, a drifted schema produces failed deserialization or corrupted downstream state now, in production, across every consumer of that topic at once. The blast radius is the topic's fan-out; the detection latency is however long it takes someone to notice a dashboard.

The estate isn't uniform, either: Avro subjects sit beside JSON Schema subjects, sometimes Protobuf ones, plus key subjects that are often a bare primitive with no structure at all — each dialect with a different level of team discipline, and no single view spanning them.

Why the usual fixes don't hold

Teams tighten compatibility settings, which makes the wire-level rules stricter without making them semantic — and maximum strictness mostly blocks legitimate evolution. Teams adopt doc-string conventions for their .avsc files, which hold until the first deadline. Teams write consumer-side contract tests, each protecting exactly one consumer, so N consumers encode the same undocumented assumption N times with no way to notice when two copies disagree. Teams write a wiki, which is accurate on the day it is written.

All of it treats the symptom. What's missing is a single governed statement of what each event means — held outside the registry, owned by people rather than by a serializer, and checkable before a change ships.

What closing the gap requires

Three properties, in our view. It has to be fed by artifacts you already have, because governance that demands new infrastructure never gets adopted. It has to run when a change is proposed, not after deploy. And it has to be honest about what it cannot see, because a tool that overstates its coverage is worse than none at all.

That is the shape our Confluent connector implements. The input is a file you produce yourself: a short loop against your own registry's REST API collects each subject's latest version — subject, version, schema id, schema type, schema — into one JSON array. Your credentials never leave your side; CoreModels never connects to your registry.

Import it once and the estate becomes a governed model: each subject a Type named for its Avro record, with the topic as its physical name and the key-or-value role, version, and schema id as metadata; each field an Element carrying Avro's own optionality, so the fact that broke the consumer in our first scenario is recorded rather than assumed; enums as governed taxonomies; named-record references resolved across subjects into explicit references.

From then on, a proposed change can be audited against that model before it reaches the registry — coded findings for a drifted field type, a narrowed or widened enum, a governed field that disappeared, a subject nobody governs — with an error count a CI job can fail a build on and a readable report the author sees while the change is still in their head.

The cost was never one incident

One paged consumer team is survivable. What compounds is the ambient tax: every consumer maintaining private assumptions, every schema review approving a change nobody can fully evaluate, and meaning drifting one compatible change at a time until no one can state what the estate is supposed to look like.

Your registry will go on doing its job exactly as designed. The gap it leaves is not its fault — and it is not going to close itself.

The extraction recipe and the exact calls are in the CoreModels Confluent Schema Registry quickstart in our docs.