Inside the proto3 Coder: The IR Mapping, the Extras, and Every Lossiness Record

Proto3 has fifteen scalar types. A neutral intermediate representation that also hosts JSON Schema, SQL, Avro, LinkML, OWL, and half a dozen other formats cannot afford fifteen distinct numeric primitives, and any tool that claims otherwise is hiding something. The arithmetic settles the question up front: some proto3 facts map structurally, some ride alongside as preserved detail, and some are declared as loss. This is the complete account, in that order.

Import is decode (proto3 text to IR), export is encode (IR to proto3 text), and the scope is a single .proto file — we do not resolve an include graph. The format key is protobuf, alias proto, valid in both directions.

What maps structurally

proto3 construct	IR construct	Notes
message	Type	id from the full name, label = the simple name
field	Element	id = type id + capitalized field name; label = the wire name
nested message	its own Type	enclosing type recorded, re-nested on encode
enum	Taxonomy	values become terms in order, with their numbers
enum value	term	label = the wire spelling
repeated	collection cardinality	item type becomes the element's value type
explicit optional	non-required element	the label is remembered and re-emitted
plain scalar / enum field	required element	proto3 implicit presence: it always carries a value
message-typed field	non-required element	message fields track presence
google.protobuf.Timestamp	DateTime primitive	the one well-known type treated specially
message/enum reference	type / taxonomy reference	resolved by proto3 scoping rules
oneof	optional elements + a group marker	exclusivity declared as loss, group reassembled on encode
map<K,V>	String element + key/value kept	declared as loss
package	schema id and label	acme.orders becomes id acmeOrders, label acme.orders
service, extend, extensions, group	—	skipped, declared as loss

Reference resolution follows the specification rather than approximating it. Every message and enum full name is registered before any field type is resolved, so forward references and nested references bind correctly. A leading dot means fully qualified; otherwise the enclosing scopes are walked innermost-out. A name that resolves to nothing in this file — an imported or unknown type — becomes a String with its verbatim token preserved and a record filed.

The scalars

proto3	IR primitive	Recorded as an approximation?
int32, int64	Integer	no
uint32, uint64, sint32, sint64	Integer	yes — unsignedness and zigzag encoding are not modeled
fixed32, fixed64, sfixed32, sfixed64	Integer	yes — fixed-width encoding is not modeled
float, double	Double	no
bool	Boolean	no
string	String	no
bytes	String	yes — proto3 bytes has no IR equivalent

Encoding IR that never saw a .proto, the defaults invert: Integer becomes int64, Double double, Boolean bool, and everything else — String, DateTime, rich text — becomes string. The last two are recorded: "proto3 has no date-time scalar; DateTime degrades to string" and "proto3 has no rich-text scalar; RichText degrades to string." A definition that did arrive from proto3 skips that degradation entirely, because its original token was preserved. That is the whole purpose of the next section.

The extras channel

Every IR node carries an annotation bag. The coder writes each proto3 fact the IR has no slot for into it under a dotted protobuf.* key, and encode reads them back:

Key	On	Carries
protobuf.package	schema	the package statement
protobuf.imports	schema	the import clauses verbatim, as a JSON array
protobuf.option.<name>	schema, type, taxonomy	the option constant, verbatim
protobuf.parent	type, taxonomy	the enclosing message's type id
protobuf.reserved	type, taxonomy	reserved statement bodies, as a JSON array
protobuf.type	element	the verbatim wire type token
protobuf.fieldNumber	element	the field number
protobuf.optional	element	"true" when the optional label was explicit
protobuf.oneof	element	the enclosing oneof's name
protobuf.mapKey / protobuf.mapValue	element	the map<K,V> key and value types, verbatim
protobuf.fieldOptions	element	bracketed field options, verbatim
protobuf.enumNumber	term	the enum value's number
protobuf.valueOptions	term	bracketed enum-value options, verbatim

Decoding an order schema, sequence becomes an Integer element carrying protobuf.fieldNumber = "6" and protobuf.type = "uint64"; total carries protobuf.fieldOptions = json_name = "orderTotal"; the Order type carries protobuf.option.deprecated = true and protobuf.reserved = ["100 to 199","\"legacy_id\""]. Encode re-emits every one of them exactly.

This is also the boundary worth knowing: the extras channel lives in the transform, and a project's schema store does not persist it. Round trips through the stateless routes keep the wire facts; round trips through a governed project keep the meaning and mint fresh field numbers.

Ids, labels, and names

CoreModels node ids must be alphanumeric, so ids are sanitized while the original spelling is kept on the label. The rules, and their visible consequences:

• A package acme.orders yields schema id acmeOrders, label acme.orders.
• A nested message Order.LineItem yields type id OrderLineItem, label LineItem, plus protobuf.parent = "Order".
• A field id is its type id plus the capitalized, sanitized field name: order_id in Order becomes OrderOrderId, label order_id.
• Ids are unique across types, elements, and taxonomies, because relation endpoints resolve by bare id. A collision gets a numeric suffix — in the order schema the nested enum Status claims OrderStatus first, so the status field becomes OrderStatus2. The label is still status.
• Enum term ids drop non-alphanumerics: SHIPMENT_STATUS_UNSPECIFIED becomes id SHIPMENTSTATUSUNSPECIFIED with the label holding the exact spelling. Targets that emit ids — JSON Schema enum arrays, for instance — show the stripped form; the proto3 encoder emits labels, so a proto-to-proto trip restores the underscores. If exact symbols matter downstream, check this line.

On the encode side, names come from labels, disambiguated per nesting scope, and forced into valid proto3 identifiers. A label that is not a legal identifier is sanitized and declared: body ref becomes body_ref. A label that would be mis-read in type position — a statement keyword, a field label, a range word, or one of the fifteen scalar names — gains a trailing underscore: a type labeled message emits as message_, and a reference to it emits message_ body_ref = 1;, which re-decodes to the same binding.

One consequence to internalize when authoring plans: the emitted field name comes from the element's label, not its id. Renaming a target element id in a plan does not rename the wire field.

Field numbers

Preserved numbers re-emit verbatim. Elements without one are minted the next free number in declaration order, skipping numbers already taken and the implementation-reserved 19000–19999 range. So a message whose fields are 1 and 4, plus one freshly added element, emits:

syntax = "proto3";

message Sparse {
  string a = 1;
  string b = 4;
  optional int64 c = 2;
}

c takes 2 — the lowest free number — rather than 5, and it carries optional because the element was not required. Growth is wire-safe by construction, not by remembering to be careful.

The complete lossiness inventory

Decode side. Every one of these is a record, not a failure:

Situation	Kind	Path
an import clause	SemanticNarrowing	imports
a oneof group	SemanticNarrowing	<Message>.<oneofName>
an option inside a oneof	StructuralDrop	<Message>.<oneofName>
map<K,V>	SemanticNarrowing	<Message>.<field>
uint*, sint*, fixed*, sfixed*	TypeApproximation	<Message>.<field>
bytes	TypeApproximation	<Message>.<field>
a type this file does not define	TypeApproximation	<Message>.<field>
service / extend at file level	StructuralDrop	service <Name>
extensions / extend / group in a message	StructuralDrop	the message's full name
no syntax statement	SemanticNarrowing	$

The last one is worth quoting in full, because it states an assumption most tools make silently: "No syntax statement: the file defaults to proto2 per the spec; it was decoded with proto3 semantics."

Encode side, for IR constructs proto3 cannot express:

Situation	Kind
an IR component (a curated view)	StructuralDrop
an IR relation instance	SemanticNarrowing
type inheritance	SemanticNarrowing — no inheritance; inherited fields are not repeated
a term hierarchy	SemanticNarrowing — enum values are flat
an element belonging to no type	StructuralDrop — proto3 has no top-level fields
a nullable element	ConstraintRelaxation — proto3 does not distinguish null from absent
collection min/max bounds	ConstraintRelaxation — repeated carries no item bounds
DateTime or rich text without a preserved token	TypeApproximation
a required message-typed field	ConstraintRelaxation — message fields always track presence
a name that had to be escaped or sanitized	SemanticNarrowing
a taxonomy with no terms	SemanticNarrowing — a UNSPECIFIED = 0; placeholder is minted

That last case exists because proto3 enums require at least one value; the placeholder keeps the output compilable and the record keeps it honest.

Round-trip fidelity

The coder's own test suite decodes a schema exercising nested messages, an enum with a gap in its numbering, a oneof, a map, a bracketed field option, message options, two reserved forms, and a service block — then encodes it, decodes the result again, and asserts that the structural signature of the two IR schemas is identical and that both validate. The emitted text carries reserved 100 to 199;, optional string note = 2;, repeated Order.LineItem items = 3;, google.protobuf.Timestamp placed_at = 8;, map<string, string> labels = 9;, the reassembled oneof payment {, double total = 12 [json_name = "orderTotal"];, and CANCELLED = 5; — the gap in the enum numbering preserved rather than renumbered.

Three details of emitted layout, all deliberate:

• References to nested definitions are emitted as dotted full names — Order.LineItem, Order.Status — so a re-decode binds to the same definition regardless of scope.
• Inside a message the order is options, reserved, fields (with oneof groups reassembled in place), nested enums, then nested messages.
• Encoding is a fixed point: encode, decode, encode again, and the text is unchanged. For a comment-free file already in the encoder's canonical layout, the first encode is byte-identical to the input. Comments are consumed by the tokenizer and never come back.

Edge cases worth knowing

Where it refuses. Only genuinely malformed or non-proto3 input fails: Only proto3 is supported; the file declares syntax "proto2"., 'required' fields are proto2; only proto3 is supported., The protobuf schema is empty., The text is not a protobuf schema (no syntax, package, message, or enum statement was found)., and parse errors that name the token, such as Expected ';' but found '}'. Everything else decodes as far as it can.

Nesting cycles. Encode rebuilds nesting from protobuf.parent. If a chain of parents forms a cycle — possible only in hand-assembled IR — the type is emitted at the top level instead of recursing forever.

oneof round trips. Members are flattened into optional elements carrying protobuf.oneof, and encode regroups them under the original name at the position of the first member. The exclusivity constraint is not modeled anywhere, which is exactly what the SemanticNarrowing record says.

Enum value options and negative numbers. Both survive: value options ride in protobuf.valueOptions, and the parser accepts a leading minus on a value number.

Duplicate term names. If two enum values sanitize to the same id, the second gets a numeric suffix; the labels stay distinct, so the emitted file is unchanged.

The transform section of the CoreModels documentation covers the routes, roles, and mapping vocabulary that surround this coder.