Every Key, Every Record: How the Ossie Coder Maps to the IR

An Apache Ossie semantic model and a typed intermediate representation disagree about the world in one fundamental way: an Ossie field has no type. It is a name, an optional expression, an optional description, and optional dimension metadata. The IR that CoreModels (by ARAMAI) moves schemas through is typed, and so is every neighboring format — SQL, Avro, JSON Schema, LinkML.

Everything interesting about this coder follows from that mismatch, and from one rule: anything the target cannot hold goes into the ledger — never silently discarded, never smuggled into the wire format. Here is the complete map.

The structural map

Ossie construct	IR construct	Notes
the document	one IR schema	id is the first model's sanitized name; label is the name as written
first semantic_model entry	the schema node itself	its description and ai_context ride on the schema's annotations
further semantic_model entries	flattened into the same schema	ids prefixed with the model name; recorded in the ledger
dataset	Type	label = name as written; id = sanitized name
field	Element	String — or DateTime when dimension.is_time
primary_key membership	Required on the Element	plus a marker annotation so the key can be rebuilt on encode
relationship	Relation in the osiRelationship group	one custom group, labeled OSI Relationship, Type → Type endpoints
metric	Component	members are the datasets its expression names
custom_extensions	schema annotations	read from the semantic model and, tolerantly, from the document root

Ids are sanitized because CoreModels node ids must be alphanumeric — the id index tokenizes on separators, so a raw web_sales could not be matched by an exact-id query. web_sales becomes webSales, ws_item_sk under it becomes webSalesWsItemSk, and the original names survive untouched as labels. Collisions get a numeric suffix: two datasets named web_sales and web sales decode to webSales and webSales2.

Metric membership is a best-effort scan: each dataset's name or source is matched at word boundaries inside the metric's expression (preferring ANSI_SQL), so SUM(web_sales.ws_sales_price) attaches the metric to webSales. When nothing matches, the component is still created — with a ledger entry saying it references no types.

The annotation contract

Everything the IR has no first-class slot for rides in dotted osi.* annotations, which is what makes an exact re-emit possible:

Key	Carries
osi.version	the wire version string
osi.description	description text (schema, dataset, field, metric)
osi.aiContext.instructions / .synonyms / .examples	the ai_context block (arrays as JSON)
osi.source	a dataset's source
osi.uniqueKeys	unique_keys, verbatim as JSON
osi.primaryKey	"true" when the field is in the primary key
osi.expression.<dialect>	one entry per dialect, expression text verbatim
osi.fromColumns / osi.toColumns	a relationship's join columns
osi.name	a relationship's original name
osi.metric	"true" — marks a component as an Ossie metric
osi.ext.<vendor_name>	a custom extension's data
osi.raw.<key>	any unknown wire key, preserved verbatim as JSON

Decoding the two-dataset retail model we test against produces exactly this on the schema node:

{
  "osi.version": "0.2.0.dev0",
  "osi.description": "Retail sales semantic model.",
  "osi.aiContext.instructions": "Prefer web_sales for revenue questions.",
  "osi.aiContext.examples": "[\"What was total revenue last month?\"]",
  "osi.ext.ARAMAI": "{\"owner\":\"analytics-team\",\"refresh\":\"daily\"}"
}

…and on a primary-key field with an expression:

{
  "osi.expression.ANSI_SQL": "ws_item_sk",
  "osi.description": "Item surrogate key.",
  "osi.primaryKey": "true"
}

The relationship keeps its own: osi.name, osi.fromColumns, osi.toColumns. The metric keeps osi.metric, osi.expression.ANSI_SQL, osi.description.

Note the version: 0.2.0.dev0 is preserved as written. We emit 0.1.1 when a model carries no version — pinned against the spec's JSON schema at the osi-0.1.1-rc1 tag of the apache/ossie repository — but we never rewrite a version somebody chose. The one exception is the legacy 1.0, a release that never existed; it normalizes to 0.1.1 with a SemanticNarrowing record, in both directions.

Parsing posture

The document is parsed once into a generic object graph and walked defensively — never bound to rigid classes, so an unfamiliar key cannot break a decode. JSON is a YAML subset, so the same parse reads the osi-json serialization; the two format keys differ only in what they emit. Consequences:

• An unknown key on a dataset, field, relationship, or metric is preserved as osi.raw.<key> and re-emitted verbatim. Give a dataset refresh_cron: "0 3 * * *" and it comes back out as refresh_cron: "0 3 * * *".
• An unknown key on the semantic model itself is reported (Unrecognized semantic-model keys not modelled: owner_team.) rather than carried.
• Unknown keys inside ai_context, expression, and dimension are reported the same way.
• primary_key accepts a scalar or a list. expression accepts the spec's dialects structure or a bare string, which is treated as ANSI_SQL.
• semantic_model accepts a single mapping as well as a list.

Three inputs fail outright: an empty document (The OSI YAML document is empty.), invalid YAML (The document is not valid YAML: ...), and a valid document with no model (The document contains no semantic_model entry.).

The lossiness inventory

Decoding can emit five records:

Kind	When
TypeApproximation at $	always, when the model has fields: OSI carries no field type system; every field was decoded as String (or DateTime when dimension.is_time). One summarized record for the whole schema, not one per field.
SemanticNarrowing at version	the legacy 1.0 was normalized to 0.1.1
SemanticNarrowing at semantic_model	several models flattened into one schema with prefixed ids
StructuralDrop at ...primary_key	a key column matching no field: they are dropped and a re-encode will emit the key without them
StructuralDrop at ...relationships[i]	endpoints that resolve to no dataset: the relation was dropped

Plus SemanticNarrowing for a metric whose expression names no known dataset, for unrecognized keys in the places listed above, and for a custom extension with no vendor_name.

Encoding reports what Ossie cannot express:

Kind	Path	Reason
ConstraintRelaxation	Taxonomy[...]	OSI has no controlled-list construct; the taxonomy (and any enum constraint) is not represented
SemanticNarrowing	Type[...]	OSI has no inheritance; the parent link is not represented
SemanticNarrowing	Relation[...]	only dataset-to-dataset OSI relationships are expressible; other relation groups are not
StructuralDrop	Component[...]	only components marked osi.metric are exported

That last table is why a JSON Schema enum converts to Ossie with a ConstraintRelaxation: the values have nowhere to live.

Two encode-side losses carry no record, because the wire format has no slot to lose them from. Field types are one: an Ossie field has none, and the decode-side approximation already states that rule for the format. Requiredness that is not key membership is the other: primary_key is rebuilt only from elements marked as key members, so a required flag arriving from JSON Schema or SQL does not invent a key. Both are worth remembering when you convert into Ossie.

What round-trips

Decode → encode → decode is stable. For the retail model, the structural signature — types, their element lists, each element's label, value type and requiredness, relations with their group and endpoints, components with their members — is identical before and after. So is the signature after a round trip through the JSON serialization, and the re-emitted YAML carries ai_context, unique_keys, the composite primary_key, per-dialect expressions, the time dimension, and the custom extension (abridged below to one dataset and two of its fields; the item dataset the relationship points at and the second metric are omitted for length):

version: "0.2.0.dev0"
semantic_model:
  - name: "retail_analytics"
    description: "Retail sales semantic model."
    ai_context:
      instructions: "Prefer web_sales for revenue questions."
      examples: ["What was total revenue last month?"]
    datasets:
      - name: "web_sales"
        source: "tpcds.web_sales"
        description: "Web channel sales facts."
        primary_key: ["ws_item_sk","ws_order_number"]
        unique_keys: [["ws_item_sk","ws_order_number"]]
        ai_context:
          synonyms: ["online sales","web orders"]
        fields:
          - name: "ws_item_sk"
            description: "Item surrogate key."
            expression:
              dialects:
                - dialect: "ANSI_SQL"
                  expression: "ws_item_sk"
          - name: "ws_sold_date"
            description: "Date of sale."
            expression:
              dialects:
                - dialect: "ANSI_SQL"
                  expression: "ws_sold_date"
            dimension:
              is_time: true
    relationships:
      - name: "web_sales_item"
        from: "web_sales"
        to: "item"
        from_columns: ["ws_item_sk"]
        to_columns: ["i_item_sk"]
    metrics:
      - name: "total_revenue"
        description: "Total revenue across web sales."
        expression:
          dialects:
            - dialect: "ANSI_SQL"
              expression: "SUM(web_sales.ws_sales_price)"
        ai_context:
          synonyms: ["revenue"]
    custom_extensions:
      - vendor_name: "ARAMAI"
        data: "{\"owner\":\"analytics-team\",\"refresh\":\"daily\"}"

Four details in that document are decisions, not accidents:

Emission is hand-written. Fixed key order, two-space indent, quoted scalars, JSON flow style for lists — output is byte-stable and exactly re-readable. The JSON serialization derives from the same emission, and scalar style disambiguates typing there: quoted scalars stay strings, plain ones re-type, so is_time becomes a real JSON true.

Extensions are emitted under the semantic model, their home in the spec, even when the input placed them at the document root; data is written as a JSON string literal, because the spec types that field as a string.

Every field gets an expression. The spec requires one, so a field that arrived without any dialect is re-emitted with its own name as the ANSI_SQL expression. Likewise a dataset with no source is emitted with its sanitized name — web_sales gets source: "webSales" — rather than producing a document that fails spec validation.

primary_key is rebuilt from the per-field markers. That is why a key column matching no field is a StructuralDrop at decode time: it would quietly change the key's composition on the way back out.

Fidelity beyond the coder

Two boundaries beyond the coder change what survives — both visible in the ledger.

Through the mapping engine (the schema/map route and the transform_schema tool), an inferred plan carries types, elements, and taxonomies. Relationships come back as StructuralDrop at Relation[...] — source relation not carried by any relation-mapping operation — until the plan carries a RelationMapping operation naming the osiRelationship group; metrics need a ComponentMapping. Add those two operations to the returned plan, replay it through plan/execute, and the relationships and metrics blocks come back.

Through a CoreModels project, the graph stores types, elements, value types and requiredness — not the format-specific osi.* annotations, and not relation instances or components (both reported as StructuralDrop on import). An export from a project is therefore a clean 0.1.1 document: real dataset and field names, sources and expressions derived from those names, time dimensions intact, no ai_context. If you need an Ossie document to survive a conversion with its annotations whole, keep it in the stateless routes, where the document never leaves the request.

The endpoint reference and the per-format quickstarts are in the CoreModels transform docs.