Ossie Conversions That Belong in CI: Plans as Artifacts, Ledgers as Gates

Run an Ossie conversion twice and diff the two outputs. If the bytes differ, the conversion cannot live in a pipeline — you would be re-reviewing generated files every build. CoreModels (by ARAMAI) gives you the other answer, and it comes from three deliberate design choices: the encoder is hand-written with fixed key order, the mapping plan is returned as an artifact you can store, and replaying a stored plan runs the same validation gate as the call that produced it.

This article is about turning those properties into automation for Apache Ossie (formerly OSI) semantic models: batch conversion, plans under version control, and a lossiness ledger used as a build gate.

Determinism, concretely

The Ossie encoder does not hand your model to a general-purpose YAML emitter and hope for stable key order. It writes the document itself: fixed key order, two-space indent, double-quoted scalars, JSON flow style for lists. Preserved detail — per-dialect expressions, custom extensions, unknown keys carried through for re-emit — is written in sorted order. The osi-json serialization is derived from that same emission, so the two wire forms carry identical content.

On top of that, the engine is a pure function of (source, plan): we ran the same stored plan against the same source in two independent executions and compared the encoded output — identical, for both the SQL and the osi-json targets. That is the property that makes generated schemas reviewable as code: a diff in the output means a change in the input or the plan, never a change in the weather.

The plan is the artifact

Take models/logistics.yaml — two datasets, a relationship, and a metric:

version: "0.1.1"
semantic_model:
  - name: logistics
    datasets:
      - name: shipments
        source: ops.shipments
        primary_key: shipment_id
        fields:
          - name: shipment_id
          - name: shipped_at
            dimension:
              is_time: true
          - name: carrier_code
      - name: carriers
        source: ops.carriers
        primary_key: carrier_id
        fields:
          - name: carrier_id
          - name: carrier_name
    relationships:
      - name: shipment_carrier
        from: shipments
        to: carriers
        from_columns: [carrier_code]
        to_columns: [carrier_id]
    metrics:
      - name: on_time_rate
        description: Share of shipments delivered on time.
        expression:
          dialects:
            - dialect: ANSI_SQL
              expression: AVG(shipments.on_time)

Every mapping response carries the executed plan. For this model the inferred plan holds seven operations; the first three:

{
  "operations": [
    {
      "kind": "TypeMapping",
      "origin": "Inferred",
      "sourceTypeId": "shipments",
      "targetTypeId": "shipments",
      "targetLabel": "shipments"
    },
    {
      "kind": "TypeMapping",
      "origin": "Inferred",
      "sourceTypeId": "carriers",
      "targetTypeId": "carriers",
      "targetLabel": "carriers"
    },
    {
      "kind": "ElementMapping",
      "origin": "Inferred",
      "sourceElementIds": ["shipmentsShipmentId"],
      "targetElementIds": ["shipmentsShipmentId"]
    }
  ]
}

Two things make this worth committing next to the source model. Every operation is stamped with its origin — Inferred here, meaning a heuristic label match; Explicit when you authored it — so a reviewer can see which alignments were guessed. And the ids are the decoder's sanitized ids (shipments, shipmentsShipmentId), which means a renamed dataset or field shows up as a plan diff in the pull request rather than as a surprise in the output.

Replaying it is one call, with the plan passed as a string:

jq -n --rawfile schema logistics.yaml --rawfile plan plans/logistics.plan.json '{
  sourceFormat: "osi",
  sourceSchema: $schema,
  plan: $plan,
  targetFormat: "osi-json"
}' | curl -s "https://coremodels.example.com/graph/transform/plan/execute/$PROJECT_ID" \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -d @-

A stored plan earns no shortcut. It is parsed, then validated by the same gate every strategy passes — an operation naming a construct that no longer exists is rejected with a path-carrying error, not executed against a stale assumption.

Editing the plan to carry what inference won't

Inference aligns types, elements, and taxonomies by label. It emits no relation or component operations, so for the logistics model the first conversion reports:

[
  {
    "kind": "TypeApproximation",
    "path": "$",
    "explanation": "OSI carries no field type system; every field was decoded as String (or DateTime when dimension.is_time). One summarized approximation for the whole schema."
  },
  {
    "kind": "TypeApproximation",
    "path": "$",
    "explanation": "OSI carries no field type system; every field was decoded as String (or DateTime when dimension.is_time). One summarized approximation for the whole schema."
  },
  {
    "kind": "StructuralDrop",
    "path": "Relation[shipmentCarrier]",
    "explanation": "Source relation not carried by any relation-mapping operation."
  }
]

(The type approximation appears twice on this route because the source and its identical hint each decode once.) The shipment_carrier relationship and the on_time_rate metric are not in the output. The fix is to append two operations to the plan and replay it:

{
  "kind": "RelationMapping",
  "origin": "Explicit",
  "sourceRelationId": "shipmentCarrier",
  "targetRelationGroupId": "osiRelationship"
}

{
  "kind": "ComponentMapping",
  "origin": "Explicit",
  "sourceComponentId": "onTimeRate",
  "targetComponentId": "onTimeRate"
}

All Ossie dataset relationships decode into one relation group, osiRelationship, which is why the target group id is the same for every relationship you carry. Metrics decode into components, keyed by their sanitized name. After the replay, the output regains its relationships and metrics blocks:

{
  "relationships": [
    {
      "name": "shipment_carrier",
      "from": "shipments",
      "to": "carriers",
      "from_columns": ["carrier_code"],
      "to_columns": ["carrier_id"]
    }
  ],
  "metrics": [
    {
      "name": "on_time_rate",
      "description": "Share of shipments delivered on time.",
      "expression": {
        "dialects": [{ "dialect": "ANSI_SQL", "expression": "AVG(shipments.on_time)" }]
      }
    }
  ]
}

…and the ledger drops to a single entry — the type approximation that is inherent to the format. Commit that edited plan; it is now the reviewed, reusable definition of "how we convert this model".

Choosing a mapping kind

Three strategies produce plans, and all three feed the same gate.

inferred (the default) matches labels against a target hint and requires one — without it you get The inference resolver requires a target IR to match against. For a format conversion, pass the source as its own hint. For aligning an incoming model onto a governed schema, pass useProjectAsTargetHint: true (on schema/map) or use schema/mapImport, where the project's schema is the target. caseInsensitive defaults to true. One caveat specific to semantic models: matching is one-to-one, so two datasets sharing a field name make the gate reject the plan with Target Element id 'customersCustomerId' is produced by 2 operations. Warehouse-style prefixed columns avoid this entirely; unprefixed shared keys need an explicit plan.

explicit takes a SIA mapping guide as JSON text — autoMatchByMapsTo, fieldMappings (sourceElementIds, targetElementIds, transformName), taxonomyDirectives, drops. Unknown keys are rejected with a path-carrying error, because a silently ignored typo would execute a plan you did not intend. What the guide contributes is exactly what you wrote plus mapsTo-based auto-matching of constructs that declare the same cross-standard URI. An Ossie document on disk carries no mapsTo annotations, so a guide alone will not align your datasets; the productive pattern for this format is inference for the backbone plus a plan edit for the specifics — or an explicit guide when the source is a CoreModels project export whose nodes already carry mapsTo.

ai asks a server-side Claude proposer for a plan. The proposal faces the identical gate with at most one repair attempt; a rejected repair is a rejection. It needs a server-configured Anthropic key and Editor/Admin membership, and it sends the schema content to the Anthropic API server-side. Any self-reported confidence on an operation is advisory — the gate never trusts it. For deterministic pipelines, use it to draft a plan interactively, then commit the plan and replay it with plan/execute so the build stays reproducible and free of model calls.

A batch loop

Convert a directory of semantic models, keeping the plan and ledger for each:

#!/usr/bin/env bash
set -euo pipefail

HOST="https://coremodels.example.com"
mkdir -p out plans

for f in models/*.yaml; do
  name="$(basename "$f" .yaml)"

  jq -n --rawfile schema "$f" '{
    sourceFormat: "osi",
    sourceSchema: $schema,
    targetFormat: "osi-json",
    targetHintFormat: "osi",
    targetHintSchema: $schema,
    mapping: { kind: "inferred" }
  }' | curl -sS "$HOST/graph/transform/schema/map/$PROJECT_ID" \
        -H "Authorization: Bearer $TOKEN" \
        -H "Content-Type: application/json" \
        -d @- > "out/$name.response.json"

  jq -e '.success' "out/$name.response.json" > /dev/null
  jq -r '.schema' "out/$name.response.json" > "out/$name.json"
  jq '.plan'      "out/$name.response.json" > "plans/$name.plan.json"
  jq -c '.lossiness[]' "out/$name.response.json" > "out/$name.ledger.jsonl"
done

Then gate the build on the ledger rather than on the exit code alone. This rule accepts the inherent type approximation and fails on anything structural:

if jq -s 'map(select(.kind == "StructuralDrop")) | length > 0' out/*.ledger.jsonl | grep -q true; then
  echo "Structural drops in the conversion — review out/*.ledger.jsonl" >&2
  exit 1
fi

Once the plans are committed, the recurring job stops re-inferring anything: it loops over plans/*.plan.json and calls plan/execute with the matching source. Inference belongs to the day you author a mapping; replay belongs to every day after.

Operational notes

The mapping routes write nothing (they scope authorization only), so a scheduled conversion needs no more than Viewer — except mapping.kind: "ai", which requires Editor. schema/mapImport does write, and it takes dryRun: true for a plan-plus-summary preview before it does. The plan field on plan/execute is a JSON string, not an object; passing an object is the most common first-run mistake. And success: true never means "nothing changed" — the ledger is the source of truth for that, which is exactly why it makes a better build gate than an HTTP status.

The full endpoint reference, including the response envelope and the mapping-guide wire format, is in the CoreModels transform docs.