Inside the ShEx Coder: The IR Map, the Extras Ledger, and What Round-Trips

Most schema formats name things locally: a SQL column is full_name, an Avro field is full_name, and any connection to the wider world has to be bolted on with annotations. ShEx arrives different. Its predicates and shape names are IRIs — schema:name is https://schema.org/name — which means a ShEx document carries its cross-standard identity in its bones. Our ShEx coder in CoreModels, by ARAMAI, is built around that fact, and this article is the full technical account of it: exactly what maps into our intermediate representation (IR), what rides in the preserved-extras ledger, every lossiness record the coder can emit, and precisely what survives a round trip.

The coder is bidirectional under the format key shex. Its external representation is the ShExC (compact syntax) document as a string, and the subset it models is: shapes, triple constraints, xsd: datatypes, shape references (@Shape), inline value sets ([ ... ]), the cardinality operators (?, *, +, {n,m}), EXTENDS inheritance, and PREFIX declarations.

That entire subset fits in one small document:

PREFIX schema: <https://schema.org/>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>

schema:PersonShape {
  schema:name xsd:string ;
  schema:email xsd:string * ;
  schema:birthDate xsd:date ?
}

schema:EmployeeShape EXTENDS @schema:PersonShape {
  schema:jobTitle xsd:string ;
  schema:workStatus [ "active" "leave" "terminated" ]
}

Decoded, that is two Types (Person and Employee, suffixes stripped), five Elements, one Taxonomy from the value set, a single parent link from EXTENDS, and a prefix map riding in extras — the mechanics the rest of this article specifies table by table.

The structural map

The IR describes schemas as Types (record-like structures) containing Elements (fields), plus Taxonomies (controlled term lists), with every node carrying an annotation bag: mapsTo links to external standards, free-form Extras, and provenance (ShEx-decoded nodes are stamped with source format ShEx and the originating shape or predicate).

ShExC construct	IR construct
Shape (schema:PersonShape { ... })	Type; id is the shape name, label is the local name with the Shape suffix stripped (Person)
Triple constraint (schema:name xsd:string ;)	Element; id is <shapeName>|<predicate>, label is the predicate's local name (name)
Datatype constraint (xsd:integer)	Primitive value type (see the datatype tables)
Shape reference (@schema:PersonShape)	Type reference — a real link to the target Type, not a string
Value set ([ "draft" "active" ])	Taxonomy with one term per entry; id is <elementId>|enum
EXTENDS @Parent	The Type's parent — single inheritance
Cardinality symbols	Required flag + single/collection cardinality (next table)
PREFIX p: <iri>	Schema-level prefix map, preserved in extras

Cardinality is a two-axis decode — is the element required, and is it collection-valued:

ShExC	Required	Cardinality
(none)	yes	single (ShEx's default is exactly one)
?	no	single
+	yes	collection
*	no	collection
{n,m}	yes when n ≥ 1	collection when m > 1, with min/max kept in the IR

The numeric form is the one asymmetry: {1,4} decodes faithfully (required, collection, min 1, max 4), but the encoder writes only the four symbols, so it re-emits as +. The bounds live on in the IR — formats that can express them receive them — but ShEx output rounds to the nearest symbol. {0,1} is recognized as exactly ?.

Datatypes, both directions

Decoding xsd: datatypes to IR primitive kinds:

ShEx datatype	IR kind
xsd:integer, xsd:int, xsd:long, xsd:short	Integer
xsd:decimal, xsd:double, xsd:float	Double
xsd:boolean	Boolean
xsd:dateTime, xsd:date, xsd:time	DateTime
xsd:string, xsd:anyURI, anything unknown	String

Encoding fresh IR back to ShEx:

IR kind	ShEx datatype
Integer	xsd:integer
Double	xsd:decimal
Boolean	xsd:boolean
DateTime	xsd:dateTime
String, RichText	xsd:string

Read those two tables together and you see a many-to-one funnel: xsd:short and xsd:long both become Integer. That would be lossy — except it isn't on a round trip, because of the extras ledger below.

A non-xsd node constraint — ., LITERAL, NONLITERAL, a bare IRI, or a class IRI used as a datatype — has no IR primitive. The decoder approximates it as String and records a TypeApproximation: "ShEx value constraint '.' has no IR primitive; approximated as String." The original token is still preserved, so ShEx-to-ShEx re-emits it exactly; the approximation is real only for other target formats, and the ledger says so.

The extras ledger

Everything the IR's type system cannot hold natively, but the original document needs back, rides in dotted, coder-prefixed Extras keys — namespaced so no two coders ever collide:

Key	On	Holds
shex.prefix.<p>	the schema	each prefix's namespace IRI (the full prefix map)
shex.shapeName	a type	the original prefixed shape name (schema:PersonShape)
shex.predicate	an element	the original prefixed predicate (schema:email)
shex.datatype	an element	the verbatim datatype token (xsd:short, xsd:anyURI, even .)

On encode, preserved values win over synthesis: a ShEx-born element re-emits its exact predicate and its exact datatype, whatever the IR kind funneled to. This is how xsd:date survives being represented as DateTime, and xsd:anyURI survives being represented as String. The prefix map is re-emitted sorted, filtered to the prefixes actually used (with xsd always retained). Five prefixes are built in as defaults on both sides: xsd, rdf, rdfs, schema (https://schema.org/), and our own cm (written here as https://coremodels.example.com/ns/).

The mapsTo lift — no annotation syntax required

Because predicates are IRIs, the decoder lifts every one into the element's mapsTo annotation automatically: standard = the prefix (schema), URI = the expansion (https://schema.org/name). Shapes get the same treatment at type level with one refinement: schema:PersonShape conventionally validates instances of schema:Person, so the coder strips the Shape suffix and maps the type to the class IRI https://schema.org/Person. Two deliberate exclusions: the a / rdf:type predicate is never lifted (it states rdf-typing, not a property meaning), and a prefix the document never declared cannot be expanded, so it produces no mapping.

This is the coder's biggest dividend. mapsTo is the cross-standard backbone our whole transform stack reads: a ShEx schema over schema.org predicates aligns — automatically, by URI — with any schema in any other format that maps to the same URIs, and mapping guides can set autoMatchByMapsTo instead of matching fragile labels.

Encoding runs the same wire in reverse for IR that was not born in ShEx: no preserved predicate means the encoder tries to reverse a mapsTo URI back into a prefixed name (longest-namespace match wins), and only then falls back to synthesizing cm:<label>. Shape names synthesize as cm:<PascalCaseLabel>Shape. So a SQL-born column with a mapsTo of https://schema.org/name exports as the real schema:name, while its unmapped siblings export under cm:.

The lossiness inventory

The decoder emits exactly two kinds of ledger records:

• SemanticNarrowing — a shape with multiple EXTENDS clauses. The IR holds single inheritance, so the first parent wins: "ShEx multiple EXTENDS (2) narrowed to a single IR parent 'schema:PersonShape'."
• TypeApproximation — the non-xsd node constraints described above, one record per occurrence, with the original token named.

Equally important is what is skipped without a ledger entry, because the coder does not model it at all: BASE, IMPORT, and start statements; the CLOSED and EXTRA shape modifiers; anything between a constraint's value expression and its terminating ; — inline // annotations and facet-style qualifiers alike; comments; and stray literals between shapes. Parsing never hard-fails on these — they are consumed and dropped. If your governance depends on CLOSED semantics or annotation content, know that they do not enter the IR. And one honest failure mode: an empty or whitespace-only document is rejected outright with the error "The ShExC document is empty." at path $.

What round-trips, exactly

Our test suite pins the invariant decode → encode → decode: the second IR is structurally identical to the first — same types, parents, element labels, value types, required flags, and cardinalities. Layered on top of that:

• Exact re-emission for ShEx-born schemas. Prefixes, shape names, predicates, and verbatim datatypes come back from the extras ledger character-for-character.
• Cross-format meaning survives. The suite also proves ShEx → IR → JSON Schema lands the predicate as "x-maps-to": { "schema": "https://schema.org/name" } on the corresponding property.
• Numeric cardinalities round to symbols on ShEx output, as described above.
• Value-set terms are opaque labels. Terms decode as term id/label pairs and re-encode as quoted string literals — [ "male" "female" ] is perfectly stable, but a value set of IRIs would come back quoted as literals rather than IRIs. Use datatype or shape constraints for IRI-valued properties you need to round-trip.

That is the whole machine: a one-to-one structural map, a dotted extras ledger for exact re-emission, an automatic mapsTo lift that turns RDF's global naming into cross-format alignment, and a lossiness inventory short enough to memorize. The same anatomy — map, extras, ledger — repeats for every format we support; the transform section of the CoreModels docs covers the rest of the lineup.