Avro Won a Layer, Not the Estate

Every durable format is built around one decision, and Avro's is easy to name: the schema travels with the data. A reader does not guess; it is handed the writer's schema and reconciles it with its own. That single choice produced the compact encoding, the evolution rules, the registries, and the reason Avro is still a default schema language of event streaming.

It also produced a second effect that gets less attention. Because you cannot write an Avro record without a schema, Avro schemas are unusually complete: numeric width, optionality, logical types, namespaces, defaults, per-field documentation, all in the file, because the format made it mandatory to decide. In many organizations, the most rigorous description of a business entity sits in an .avsc that only stream tooling ever opens.

Avro won its layer decisively and cannot leave it. That is the key to where we put it.

The neighborhood

Avro's immediate surroundings have three landmarks.

The format itself is an Apache project with an open specification. Its schemas are ordinary JSON documents, which is quietly important: anything that reads JSON can read an .avsc. Its type system is small, and its logical types — dates, timestamps at millisecond and microsecond precision, decimals — sit on top of primitive encodings rather than replacing them.

Then there are the registries, which store schemas per subject and per version and enforce compatibility between versions. A registry is the runtime authority for whether a change is safe to deploy on the wire — a real and valuable job, and a different one from the job in this article.

Then there are the neighbors: Protocol Buffers from the RPC world, JSON Schema from the API world. Modern registries carry all three, and most organizations of any size run more than one.

That last fact is the structural one. Avro rarely loses its niche, because it is genuinely excellent at schema-governed serialization for streams — and it rarely gets to own the estate. The entity moving through your topics as Avro lands in a warehouse as SQL, appears in an API as JSON Schema, and gets promised to a partner in a contract. Every boundary between those layers is where meaning gets re-typed by hand.

Where CoreModels puts Avro

We implement the Apache Avro schema format as one of twelve formats our transform surface can decode, and one of twelve it can encode. The others are JSON Schema, ShEx, JSON-LD, SQL DDL, OWL, LinkML, Protocol Buffers, Apache Ossie semantic models in YAML or JSON, the Bitol Open Data Contract Standard, the MACH Alliance Open Data Model on the decode side, and the Sage Bionetworks Synapse draft-07 profile on the encode side. All of them translate through the same neutral model, which is what makes the combinations work without a converter per pair.

Avro works in both directions. Import a record schema and its records become Types, its fields Elements, its enums Taxonomies, its arrays collections, its ["null", T] unions optional elements, its named and nested records references between Types. Export and a well-formed record schema comes back, with optionality as unions, taxonomies as enums, date-times re-emitting their logical types, and preserved tokens keeping an int an int. Records travel too: Avro's JSON encoding sits alongside JSON, CSV, JSON-LD and SQL inserts on the data surface.

What it composes with

Because everything meets at one model, Avro composes with formats that have never heard of it. An entity that entered as an .avsc can leave as Postgres DDL for the warehouse, as LinkML for a modeling group, as a data contract for a partner, as OWL for an ontology team, or as Protobuf for a service team that prefers it — each export arriving with its own account of what that target could not hold.

The registry side composes too, without us standing in the data path. A subjects export — the artifact registry tooling already produces — can be imported to bring a streaming estate under governance, or audited read-only for findings about coverage, drift, and hygiene. In the other direction, governed Types generate registry-ready .avsc files that you register through your own process. No broker connection, no held credentials.

And because Avro tolerates custom attributes, semantic annotations can ride inside the .avsc itself rather than in a sidecar file that gets lost in the third copy. We use Avro's own extension headroom, conservatively; tooling that ignores those attributes keeps working as before.

Why neutrality is the point, not a pose

Building Avro-shaped tooling would have been easier. We chose neutral ground for structural reasons.

Serialization choices are provisional; meaning is not. Teams move from Avro to Protobuf, or run both for a decade, or adopt JSON Schema for a new surface. If your definitions live inside one format, each of those is a re-modeling project; if they live in a neutral model, each is an export. What your organization means by "customer" should not be hostage to a serialization decision made by people who have since left.

Boundaries are where drift breeds, and no format sees its own edge. A registry can guarantee that version 7 of a subject is readable by consumers of version 6. Nothing inside it can notice that the warehouse column and the topic's enum stopped agreeing last spring. Only a vantage point outside both formats can hold two projections accountable to one definition.

A tool that belongs to one ecosystem has an incentive to flatter it. We publish the opposite: a ledger, on every translation, naming what the target could not express — the union we narrowed, the decimal precision we could not carry, the constraint the target could not enforce. That candour is only credible from a position with no format to defend.

Open specifications, implemented as published. We implement Avro's specification, and the other open standards on the surface, as written. Where a specification says something we do not support exactly, the honest move is a lossiness record, not a private extension only our tooling understands.

What we are not

We are not a registry, not a broker, and not a reason to reconsider Avro — for schema-governed streams it remains an excellent choice. What we do ask you to reconsider is the idea that any single format, Avro included, should be the definition rather than a projection. Where streams, warehouses, APIs, and contracts each speak their own language, the definition belongs on ground none of them owns, with every dialect generated from it and every translation's cost written down.

To see the full format list and try a translation, start with the transform quickstart in the CoreModels docs.