ODM at Scale: Batch Conversion, Deterministic Pipelines, and Replayable Plans

One entity document is a demo. A standards repository is dozens of them — identity, product, inventory, pricing — revised by pull request, consumed by teams who need formal schemas, not Markdown. The moment MACH Open Data Model documents become an input to your build, three engineering questions appear: can the conversion run unattended, will the same input always produce the same output, and what happens when one file in the batch is broken?

The CoreModels transform surface was built with those questions as requirements. This article shows the pipeline patterns we use for ODM: batch conversion with per-file failure isolation, lossiness gates in CI, dry-run-then-write imports, and the mapping plan as a committed, replayable artifact.

Determinism is the foundation

Every transform in CoreModels executes deterministically: the same source document produces the same output document, byte for byte. That single property is what makes the rest of this article possible. Converted schemas can live in git next to the entity documents they came from, a nightly job can regenerate them, and git diff becomes your change detector — an upstream edit to an entity document shows up as a reviewable schema diff, and an empty diff means nothing changed. No timestamps, no reordered keys, no noise.

Batch conversion with failure isolation

The converter endpoint takes multiple entities in one call. The body is entities: [{ name, markdown }, ...]; the response is a schemas bundle keyed by kebab-cased entity title. Building the body from a directory of Markdown files is a few lines of shell:

#!/usr/bin/env bash
set -euo pipefail

BODY=$(jq -n '{ entities: [] }')
for f in entities/*.md; do
  BODY=$(jq --rawfile md "$f" --arg name "$(basename "$f" .md)" \
    '.entities += [{ name: $name, markdown: $md }]' <<< "$BODY")
done

curl -s -X POST \
  "https://coremodels.example.com/graph/transform/odm/convert/$PROJECT_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  --data "$BODY" > bundle.json

The failure semantics are the point. Files that are not entity documents at all (a README with no ## YAML Schema Definition section) are simply skipped. A file that should convert but cannot — say the section exists but contains no fenced YAML block — does not poison the batch. It is reported in the lossiness ledger as a StructuralDrop naming the file, and every other entity still converts:

{
  "success": true,
  "lossiness": [
    {
      "kind": "StructuralDrop",
      "path": "pricing",
      "explanation": "Entity conversion failed and the file was skipped: The section contains no fenced yaml code blocks."
    }
  ],
  "errors": [],
  "schemas": {
    "product": { "...": "..." },
    "inventory": { "...": "..." }
  }
}

Only when zero entities convert does the call fail outright, with per-file errors. The error channel means "could not proceed"; per-file trouble rides the reported-and-continue channel. Splitting the bundle back into files is one jq loop:

for key in $(jq -r '.schemas | keys[]' bundle.json); do
  jq ".schemas[\"$key\"]" bundle.json > "schemas/$key.schema.json"
done

Gate on the ledger, not just the status code

success: true means the run completed — the lossiness array is where fidelity lives. In a pipeline, that array is machine-readable policy input. A strict gate that fails the build if any entity was dropped:

jq -e '[.lossiness[] | select(.kind == "StructuralDrop")] | length == 0' \
  bundle.json > /dev/null || { echo "ODM batch had dropped entities"; exit 1; }

The four lossiness kinds (StructuralDrop, TypeApproximation, ConstraintRelaxation, SemanticNarrowing) support graduated policy: fail on drops, warn on narrowing (a dangling $ref in an entity document surfaces as SemanticNarrowing at its exact path), and log the rest. Because every record carries kind, path, and explanation, the failure message in your CI log already says which file, which construct, and why.

Dry-run in the PR, write on merge

For pipelines that import ODM into a CoreModels project rather than just converting it, the import endpoint's dry-run contract maps cleanly onto CI stages. In the pull-request job, a Viewer token is enough, and nothing is written:

jq -n --rawfile md entities/product.md '{ markdown: $md, dryRun: true }' \
| curl -s -X POST \
  "https://coremodels.example.com/graph/transform/odm/import/$PROJECT_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  --data @- | tee dryrun.json

jq -e '.success and (.lossiness | length == 0)' dryrun.json > /dev/null

The dry-run returns the would-be summary — { "types": n, "elements": n, "taxonomies": n } — plus the full ledger, so reviewers see exactly what a merge will create before it creates anything. The merge job then repeats the call with "dryRun": false under an Admin token, writing the schema with its provenance stamped odm. Same body, two stages, least privilege at each.

The plan is a pipeline artifact

Conversion is only half the automation story; the other half is mapping — aligning ODM entities onto a target shape your systems already speak. Here the engine's design pays off in a specific way: every mapping strategy produces a plan, every plan passes the same universal validation gate, and the executed plan comes back in the response as a replayable artifact.

That enables a workflow we recommend for any recurring ODM mapping:

1. Once, interactively: call POST /graph/transform/schema/map/{projectId} with sourceFormat: "odm", your target, and a mapping kind. Review the returned plan, then commit it to the repository like any other build input. schema/map writes nothing — every call is inherently a dry run.
2. Every run thereafter: replay the committed plan with POST /graph/transform/plan/execute/{projectId}, passing the plan JSON as a string:

jq -n --rawfile md entities/product.md --rawfile plan plans/product-plan.json '{
  sourceFormat: "odm",
  sourceSchema: $md,
  plan: $plan,
  targetFormat: "jsonschema"
}' \
| curl -s -X POST \
  "https://coremodels.example.com/graph/transform/plan/execute/$PROJECT_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  --data @-

A stored plan earns no shortcut — it re-enters the same validation gate on every execution — but once validated it executes deterministically: same plan plus same source equals same output, a property proven by test. Your pipeline's mapping behavior is now versioned, diffable, and reviewable in the same pull request as the entity documents themselves.

Choosing a mapping kind for unattended runs

The three mapping kinds have distinct automation profiles when the source is ODM:

• inferred — label and type matching against a target hint, caseInsensitive by default. Zero configuration, ideal for the first pass and for targets whose field names track the ODM entity's. Deterministic, Viewer-role, safe on a schedule.
• explicit — an authored SIA mapping-guide JSON (fieldMappings with sourceElementIds/targetElementIds/transformName, taxonomyDirectives, drops, autoMatchByMapsTo). This is the production-pipeline kind: the guide is a committed file, and the parser rejects unknown keys with a path-carrying error — a typo in a guide fails the build instead of silently executing a plan you did not intend.
• ai — a server-side model proposes the plan, and the identical gate validates it with at most one repair attempt; a rejected repair is a rejection, and the proposal's self-reported confidence is advisory only — the gate never trusts it. It requires Editor or Admin project membership and a server-configured Anthropic key, and it sends schema content to the Anthropic API server-side. Our recommendation for pipelines: use ai interactively to draft a mapping, review the plan it produces, commit that plan — and let plan/execute do the unattended runs. The creativity happens once, under review; the automation replays a validated artifact.

The shape of a good ODM pipeline

Pull the entity documents, batch-convert, gate on StructuralDrop, diff the regenerated schemas, dry-run the import in review, write on merge, and replay committed plans for any onward mapping. Every step is stateless or dry-run until the one that is not, every step returns the same envelope, and every fidelity decision the engine made is on the record. That is what it takes for authored documentation to become build input you can trust.

For ready-to-paste request bodies and the complete endpoint reference, see the CoreModels transform API guide in the product docs.