Same Plan, Same Bytes: Automating JSON Schema Conversion

A generator you cannot re-run and get identical output from is not a build step. It is a rumor with a timestamp. That is the practical objection to putting schema conversion in a pipeline: if today's run can differ from yesterday's for reasons nobody logged, then the generated Avro, the generated DDL, and the generated proto files are all provisional, and reviewing them is theater.

The CoreModels transform surface is built around the opposite property. A conversion produces a plan — an explicit, readable list of operations — and that plan, replayed against the same source, produces the same output byte for byte. We test that by executing the same stored plan twice and comparing the results. This article is about building on that: how to structure a JSON Schema conversion pipeline, what to commit, what to gate on, and where the sharp edges are.

Throughout: host https://coremodels.example.com, token in $TOKEN, 32-hex project id in $PROJECT_ID. All routes take Authorization: Bearer $TOKEN and Content-Type: application/json.

Two phases, and the line between them

Phase one is propose, and it happens when a human is present. schema/map decodes the source, produces a plan through the strategy you chose, executes it, and hands back the schema, the plan, and the lossiness ledger. Nothing is written to the project, so the call is inherently a dry run.

Phase two is replay, and it happens in CI with nobody watching. plan/execute takes the stored plan plus the same source and re-runs it. Crucially, a stored plan gets no privileges: it passes through the identical validation gate a fresh plan does. A plan someone hand-edited into something invalid is rejected on its merits, not accepted because it came from a file in the repository.

The line between the phases is where review belongs. You review a plan once — it is small, named, and diffable — and after that the pipeline is mechanical.

Phase one: propose a plan per schema

Assume schemas/*.json holds your JSON Schema files. This loop asks for a conversion of each one and saves three artifacts: the plan, the target output, and the ledger.

#!/usr/bin/env bash
set -euo pipefail

HOST="https://coremodels.example.com"
TARGET="${1:-avro}"
mkdir -p plans out ledgers

for f in schemas/*.json; do
  name="$(basename "$f" .json)"

  jq -n --rawfile s "$f" --arg t "$TARGET" \
    '{ sourceFormat: "jsonschema", sourceSchema: $s,
       targetFormat: $t,
       targetHintFormat: "jsonschema", targetHintSchema: $s,
       mapping: { kind: "inferred", caseInsensitive: true } }' > "/tmp/$name.req.json"

  curl -sS -X POST "$HOST/graph/transform/schema/map/$PROJECT_ID" \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    --data-binary "@/tmp/$name.req.json" > "/tmp/$name.res.json"

  jq -e '.success' "/tmp/$name.res.json" > /dev/null \
    || { echo "FAILED: $name"; jq '.errors' "/tmp/$name.res.json"; exit 1; }

  jq -S '.plan'      "/tmp/$name.res.json" > "plans/$name.$TARGET.plan.json"
  jq -S '.lossiness' "/tmp/$name.res.json" > "ledgers/$name.$TARGET.json"
  jq -r 'if (.schema | type) == "string" then .schema else (.schema | tojson) end' \
     "/tmp/$name.res.json" > "out/$name.$TARGET"
done

Note the self-hint: targetHintSchema is the source itself. Inference needs something to map toward — without a hint it refuses, with The inference resolver requires a target IR to match against. — and when you are converting rather than reconciling, the source is the correct answer. When you are reconciling — aligning a legacy schema onto a canonical one — pass the canonical schema as the hint instead, and the ledger starts telling you about the differences.

Commit plans/ and ledgers/. The plan is the reviewable artifact; the ledger is the reviewed decision. A pull request that changes either one is a conversation worth having, and both are small enough that the conversation is short.

Reading a plan in review

A plan is a flat list of operations, each stamped with its origin:

{
  "operations": [
    {
      "kind": "TypeMapping",
      "origin": "Inferred",
      "sourceTypeId": "Order",
      "targetTypeId": "Order",
      "targetLabel": "Order"
    },
    {
      "kind": "Drop",
      "origin": "Explicit",
      "sourceNodeId": "Order::legacyRegionCode",
      "sourceKind": "Element",
      "reason": "Dropped by the SIA guide.",
      "declaredLossiness": [
        {
          "kind": "StructuralDrop",
          "path": "Element[Order::legacyRegionCode]",
          "explanation": "Dropped by the SIA guide."
        }
      ]
    }
  ]
}

Six kinds exist: TypeMapping, ElementMapping, TaxonomyMapping, RelationMapping, ComponentMapping, and Drop. origin is Explicit, Inferred, or AiAssisted, so a reviewer can see at a glance which lines are instructions and which are guesses. A drop carries its own lossiness record, which is why deletions cannot happen quietly.

Phase two: replay in CI

The check that keeps generated files honest: regenerate from the committed plan and diff.

#!/usr/bin/env bash
set -euo pipefail

HOST="https://coremodels.example.com"
TARGET="${1:-avro}"
status=0

for f in schemas/*.json; do
  name="$(basename "$f" .json)"
  plan="plans/$name.$TARGET.plan.json"
  [ -f "$plan" ] || { echo "no plan for $name"; status=1; continue; }

  jq -n --rawfile s "$f" --rawfile p "$plan" --arg t "$TARGET" \
    '{ sourceFormat: "jsonschema", sourceSchema: $s,
       plan: $p, targetFormat: $t }' > "/tmp/$name.replay.json"

  curl -sS -X POST "$HOST/graph/transform/plan/execute/$PROJECT_ID" \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    --data-binary "@/tmp/$name.replay.json" > "/tmp/$name.replayed.json"

  jq -r 'if (.schema | type) == "string" then .schema else (.schema | tojson) end' \
     "/tmp/$name.replayed.json" > "/tmp/$name.$TARGET"

  diff -u "out/$name.$TARGET" "/tmp/$name.$TARGET" || status=1
done

exit $status

plan travels as a string containing the plan JSON — the one detail that costs everyone an afternoon the first time. If the diff is empty, the committed outputs are exactly what the committed plans produce from the committed sources. If it is not, either a source changed or a plan changed, and the diff says which.

Gating on the ledger

The ledger is machine-readable, so decide once what your pipeline tolerates. The four kinds are StructuralDrop, TypeApproximation, ConstraintRelaxation, and SemanticNarrowing. A reasonable default: approximations and relaxations are facts of life between formats, but a structural drop should never appear without someone having asked for it.

# Fail the build on any structural drop that is not in the allowlist.
jq -s -e --slurpfile allow ci/allowed-drops.json '
  [ .[][] | select(.kind == "StructuralDrop")
          | select(.path as $p | ($allow[0] | index($p)) | not) ] | length == 0
' ledgers/*.json > /dev/null || { echo "unexpected structural drop"; exit 1; }

The allowlist is a JSON array of paths such as ["Element[Order::legacyRegionCode]"]. Fields you meant to retire stay listed; fields that vanish for any other reason stop the build.

Choosing a mapping kind per pipeline stage

inferred is the right default for one-to-one conversion. It matches by label and by type, it will not pair two same-named properties whose primitive types disagree, and anything it leaves unmatched is reported: Element is a member of the mapped type but no operation maps or drops it. That message is the entire value of the strategy — it never quietly retypes a field to make a match happen.

explicit is for repeatable alignment. The guide has exactly four keys:

{
  "autoMatchByMapsTo": true,
  "fieldMappings": [
    { "sourceElementIds": ["Order::customerName"],
      "targetElementIds": ["Order::firstName", "Order::lastName"],
      "transformName": "SplitName" }
  ],
  "taxonomyDirectives": [
    { "sourceTaxonomyId": "Order::status::enum",
      "targetTaxonomyId": "Order::status::enum",
      "treatment": "InlineEnum" }
  ],
  "drops": ["Order::legacyRegionCode"]
}

autoMatchByMapsTo aligns the elements carrying matching x-maps-to assertions on both sides — the annotation backbone doing the work instead of string similarity. fieldMappings handles the cases matching cannot express, with a named transform from the registry (SplitName, JoinName, CoerceStringToDateTime, CoerceDateTimeToString, CoerceIntegerToDouble, CoerceDoubleToInteger). taxonomyDirectives carries controlled lists. drops retires fields on the record. Unknown keys are rejected with their path, so a typo in a generated guide fails instead of quietly doing nothing.

ai belongs in phase one only, and only with a person reading the result. It requires Editor or Admin membership and a server-configured key, and it sends schema content to the Anthropic API server-side. The proposal is validated by the same gate as everything else, with at most one repair attempt. The pipeline discipline that makes it safe is the one already described: use it to propose a plan, review the plan, commit the plan, and let CI replay the plan. After review, the model is no longer in the loop — the artifact is.

The edge worth knowing before it finds you

Enums do not survive inference. A JSON Schema enum decodes into a controlled list identified as Type::property::enum, and label matching cannot pair an unnamed list with anything in the hint. The call fails clearly rather than dropping the constraint:

target:Element[Order::status].ValueType: Referenced taxonomy 'Order::status::enum' does not resolve.

The fix is one line of explicit guide — a taxonomyDirective naming the list on both sides, as in the guide above. If your schemas/ directory contains enums (most do), plan on explicit for those files and keep inferred for the rest; the pipeline shape is identical, only the mapping object differs.

Two smaller ones. A mapping that renames the root type and then encodes back to JSON Schema will report The schema has no root type to encode. — the document needs a root object, and the source's root marker still names the original. And label case matters more than you would expect: caseInsensitive defaults to true, so OrderId matches orderid unless you turn it off, which you should if your naming convention is meaningful.

For route details, roles, and the full mapping-guide reference, see the schema transformation guide in the CoreModels documentation.