The Four Places Your Avro Schema Isn't

Someone posts a question in a team channel: what are the legal values of status on the orders stream?

There are four places to look, and all four answer.

The .avsc in the producer's repository says status is an enum with five symbols. The warehouse table that mirrors the topic says VARCHAR, and a quick SELECT DISTINCT returns six values. The API that exposes the same orders to partners describes status as a string with a helpful comment. And the data dictionary a governance team maintains — a spreadsheet, updated by hand — lists four.

Nobody lied. Each artifact was written by a careful person doing their job. They disagree because they are four independent transcriptions of one idea, and transcriptions drift. The question gets answered by whoever replies first with the most confidence, which is not the same thing as being answered correctly.

That is the problem Avro support in CoreModels exists to solve — and Avro is not the villain in it. The format does its job well. This is simply not the job it does.

The schema is complete, and it is stuck

Avro's defining design decision is that the schema travels with the data, and the consequence is that Avro schemas tend to be unusually complete. Because a producer cannot write a record without one, the .avsc file carries details that live nowhere else: whether a number is int or long, whether a timestamp is timestamp-millis or timestamp-micros, whether a field is genuinely optional, the namespace, the field defaults, and — in the good teams — a doc string per field. For many streams, that file is the most complete description of a business entity anywhere in the organization.

It is also the least portable, because streaming tooling reads it and nothing else does. The warehouse modeler re-deriving the same entity in SQL cannot see it. The analyst assembling a semantic layer cannot see it. The compliance officer maintaining the fourth transcription has never opened one. The knowledge exists; it has no way to leave the topic.

Every hop is a re-typing, and re-typings disagree

Watch one entity travel. It leaves as Avro. It lands in a warehouse as DDL, written by hand from the .avsc. Someone exposes it through an API and writes JSON Schema from the DDL, because that was closer to hand. A partner asks for a data contract and receives a YAML file typed from the API docs. Three hand-conversions, each a fresh opportunity to make the same three mistakes.

The enum evaporates. Avro's enum is a closed list of symbols; SQL's usual answer is a VARCHAR that accepts anything. The constraint doesn't fail loudly — it stops existing.

Optionality flips. ["null", T] is Avro's only way to say "this field may be absent"; downstream it becomes "this column is nullable," a different claim about a different thing.

The timestamp loses its scale. A timestamp-millis is a long underneath; re-typed by hand it becomes a BIGINT, or a string, or a timestamp in whatever precision the target defaults to.

None of these produce an error. They produce a slow disagreement that surfaces two quarters later in a number nobody can reconcile.

Avro's escape hatches encode indecision permanently

A field typed as a union of null, string, and long is a decision that nobody made, frozen into the wire format. Months later nobody can say whether the long branch is legacy, a mistake, or load-bearing — and no tool can tell them, because the schema is syntactically perfect and semantically mute. The same goes for a map standing in where somebody didn't want to model a structure, for a fixed whose meaning lives in a producer's head, and for a decimal whose precision matters to finance and is invisible to consumers that treat it as a number.

These are not Avro's failures. They are the residue of real engineering decisions. The failure is that there is nowhere to record what they mean.

What the pain is actually asking for

Not another linter, and not a stricter compatibility mode — compatibility checking answers "can old readers decode new writes," which is a question about bytes, while every question above is a question about meaning.

What it asks for is a change in status: the .avsc should stop being the truth and become one dialect of it, alongside the DDL, the JSON Schema, and the contract, with all of them generated from a single governed definition.

That is how CoreModels treats the avro format. Import a record schema and the record becomes a Type; each field an Element; each enum a Taxonomy — a named, governed list of allowed values rather than an array of strings inside a JSON file. An array becomes a collection. A ["null", T] union becomes an optional element, which is what it always meant. A nested or named record becomes a reference to another Type, so shared records stay shared. The date, time-millis, timestamp-millis, and timestamp-micros logical types become genuine date-time semantics rather than anonymous integers.

Fidelity travels with the structure. The exact Avro type token is preserved — int stays int rather than widening to long — along with the namespace, the doc strings, and the field defaults, so exporting back to Avro re-emits them. A round trip is structurally stable, held to that by a test that decodes, re-encodes, decodes again, and compares.

Avro carries no IRIs, so it has no native way to say "this field means the same thing as that public vocabulary term." It does tolerate custom attributes, so we use that headroom instead of a sidecar file: semantic annotations ride inside the .avsc, ignored by anything that doesn't care and lifted by us on import.

And where it doesn't fit, we say so

The compromises are specific and written down: a union with more than one non-null branch narrows to its first branch, a map or a fixed is carried as a string approximation, a decimal gives up its precision and scale. Each lands in the ledger with a kind, a location, and a plain-English explanation, on every import and export.

Silence is what produced the four contradictory answers. An explicit ledger is the smallest honest alternative.

The question at the top of this article deserves one answer, with the other three artifacts generated from it and every difference written down. To try that against a schema of your own, start with the transform quickstart in the CoreModels docs.