The Day After Import: What a Team Can Do Once Avro Is a Projection

Four requests land on a platform team in one week. None is unusual, and each is, underneath, the same request wearing a different hat.

Analytics wants the orders topic as a warehouse table. A service team building an internal API would rather work in Protobuf. A partner's procurement process requires a data contract in a standard format before the integration can be signed. A modeling group wants the entity in LinkML, to reconcile it with a published vocabulary.

Usually that means four people reading the same .avsc and typing four new artifacts, each in a format they know well and the others don't. Four transcriptions, four judgement calls about the same union, four chances for the enum to become free text — and in six months, four things to keep in sync by hand.

This article is about the other arrangement — the day after those schemas became a governed model.

The entity gets imported once

The starting move is one call to the schema import endpoint: format avro, the .avsc as the payload. The record becomes a Type; its fields become Elements; its enums become Taxonomies; arrays become collections; ["null", T] fields become optional elements; nested and named records become references between Types. Requiredness, cardinality, and the x-maps-to annotations ride along as governed facts; the namespace, the doc strings, the field defaults, and the exact Avro type tokens travel with every stateless translation.

The response tells you two things: that it ran, and what it could not represent exactly. A three-branch union, a map, a fixed, or a decimal each appears in the lossiness list with a path and an explanation. That list is the first useful artifact of the exercise: an inventory of the places where your schema was never actually decisive.

Request one: the warehouse table

The analytics request is now an export, not a transcription. Ask the project for sql with a vendor — postgres, mysql, or sqlserver — and you get DDL generated from the definition the stream is generated from.

The interesting part is the ledger attached to it. Where a target genuinely cannot enforce something, the export says so instead of pretending: a governed allowed-value list has no inline equivalent in Postgres, so the column comes out as a character type and the relaxation is recorded against that element by name. The DDL is not smarter than a careful engineer's; it is more accountable, because the compromises arrive as a checklist rather than as folklore.

Request two: Protobuf for the service team

Same model, different target. The service team gets a proto3 file with messages, fields, and enums, plus its own account of the differences — proto3 has no date-time scalar, so a date-time degrades to a string, and identifiers illegal in proto are sanitized. Both facts come back as records attached to paths. Nobody has to carry "Avro says this, proto says that" in their head; the divergence between two dialects is written down at the moment it is created.

Request three: the partner contract

Same shape again: export the model as a Bitol Open Data Contract Standard v3 YAML contract, with its contract head, schema block, and per-property logical types. Anything the contract format cannot carry — a controlled list, for instance — is reported rather than smuggled into the wire format under a name the specification does not define.

Request four: the modeling group

LinkML out, with classes, attributes, enums, and required-ness intact. The modeling group can then align terms against published vocabularies, and that work is not stranded in a side document, because the annotation carrier they write to is the one every other export reads.

And the stream still gets its .avsc

This is the part that decides whether a team adopts the arrangement or abandons it. Governance that costs you your streaming workflow is not governance; it is a tax.

Export back to avro and you get a record schema again: required fields bare, optional fields as ["null", T] unions, taxonomies as Avro enums, date-times as a timestamp-millis logical type, named types defined on first use and referenced by name after that. On the translation surface, preserved tokens mean an int field comes back as int, not widened to long because that suited the model better. Decode, encode, decode: structurally stable, and held to that by test.

Teams running a schema registry have a second path in the same spirit. A subjects export — the artifact your registry tooling already produces — brings an existing streaming estate under governance in one direction; in the other, governed Types generate registry-ready .avsc files, one per Type, with optional fields as ["null", T] carrying default: null, taxonomies as Avro enums, and date-times as timestamp-millis. Those files come back as artifacts: you review them and register them through your own process. No broker connection is opened, and no registry credentials are held on our side.

Records move too, not just shapes

Schemas are half the job. The data surface reads and writes records in Avro's JSON encoding alongside JSON, CSV, JSON-LD, and SQL inserts — which matters, because Avro's JSON encoding is not plain JSON and hand-rolling it is where integrations break. An optional field is null or a one-key union object naming its branch. A date-time is the underlying epoch-milliseconds integer, not an ISO string. Collections are arrays; enum values are bare symbols. We handle that faithfully both ways, and are equally plain about Avro's two record-level limits: records carry no identity, so a record id is not written — recorded in the ledger on every run — and Avro links by embedding rather than by id, a documented divergence stated up front rather than discovered during an incident.

That turns cross-schema record migration into an ordinary operation: read records shaped by one schema, execute a reviewed plan, write them shaped by another, losses aggregated across the whole path.

What actually changed

Before: four artifacts, four authors, four private judgement calls, and a standing meeting to reconcile them.

After: one governed definition, four generated projections, and — in place of the meeting — a written list of exactly where each projection had to compromise. Disagreements that used to surface in a quarterly reconciliation now surface at the moment of translation, attached to a field name, in front of the person who can decide.

The team's work stops being transcription and becomes review. That is the job those people were hired to do.

To see the full set of target formats and try an export against a schema of your own, start with the transform quickstart in the CoreModels docs.