ShEx in the Pipeline: Deterministic Transforms, Replayable Plans, Batch Conversion

A schema conversion you cannot reproduce is a liability with a timestamp on it. If your ShEx shapes are converted to JSON Schema by hand, or by a script whose behavior depends on who runs it and when, then the day the outputs disagree you have no way to say which conversion was right. We built the CoreModels transform surface so that conversion can live in a pipeline like any other build step: deterministic outputs, a reviewable plan artifact you can commit next to your shapes, and a change report you can gate a build on. This article shows the working patterns for ShEx (format key shex).

Three properties make this safe to automate, and all three are engineered rather than hoped for:

1. The engine is deterministic. Same validated plan plus same source produces the same output, every time — an invariant we prove by test, not a habit we hope holds.
2. The plan is an artifact. Every mapping call returns the executed plan as JSON. You store it; a separate endpoint replays it — and a stored plan earns no shortcut, it passes the same validation gate as a fresh one.
3. The ShEx encoder emits stable text. Prefix declarations are sorted and limited to the prefixes actually used; shapes and constraints follow the model's order; preserved predicates and datatypes are re-emitted exactly. Stable text means meaningful git diffs on generated shapes.

All calls below hit https://coremodels.example.com with -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json", against a project id in $PROJECT_ID (mapping calls are stateless — the project scopes authorization only, nothing is written).

Pattern 1 — Batch-convert a directory of shapes

One target hint, many ShEx files, one response and one extracted plan per file:

#!/usr/bin/env bash
set -euo pipefail
mkdir -p out plans

for f in shapes/*.shex; do
  name=$(basename "$f" .shex)

  jq -n --rawfile src "$f" --rawfile hint hints/service.schema.json '{
    sourceFormat: "shex",
    sourceSchema: $src,
    targetFormat: "jsonschema",
    targetHintFormat: "jsonschema",
    targetHintSchema: $hint,
    mapping: { kind: "inferred", caseInsensitive: true }
  }' > "out/$name.request.json"

  curl -s -X POST "https://coremodels.example.com/graph/transform/schema/map/$PROJECT_ID" \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d @"out/$name.request.json" > "out/$name.response.json"

  jq '.schema' "out/$name.response.json" > "out/$name.schema.json"
  jq '.plan'   "out/$name.response.json" > "plans/$name.plan.json"
done

Every response uses the same envelope — success, lossiness, errors, plus schema and plan — so the extraction lines never change per format. Want Avro or SQL alongside JSON Schema? Loop targetFormat over ["jsonschema", "avro", "sql"]; only sql takes an extra key ("vendor": "postgres" | "mysql" | "sqlserver").

Pattern 2 — Gate the build on the lossiness ledger

success: true means the transform ran, not that it was faithful. The faithfulness signal is the lossiness array, where each record carries one of four kinds: StructuralDrop, TypeApproximation, ConstraintRelaxation, SemanticNarrowing. A pragmatic CI policy: fail hard on structural drops, print and tolerate the rest.

for r in out/*.response.json; do
  drops=$(jq '[.lossiness[] | select(.kind == "StructuralDrop")] | length' "$r")
  if [ "$drops" -gt 0 ]; then
    echo "FAIL: $r dropped content:"
    jq -r '.lossiness[] | "\(.kind) at \(.path): \(.explanation)"' "$r"
    exit 1
  fi
done

Because every record has a path and a plain-English explanation, the failure message in the log is already the review comment.

Pattern 3 — Commit the plan, replay it forever

The plan produced in Pattern 1 is the reviewable statement of how source constructs map to target constructs. Commit plans/*.plan.json next to shapes/*.shex. From then on, CI does not re-derive the mapping — it replays it:

jq -n --rawfile src shapes/product.shex \
      --arg plan "$(jq -c . plans/product.plan.json)" '{
  sourceFormat: "shex",
  sourceSchema: $src,
  plan: $plan,
  targetFormat: "jsonschema"
}' | curl -s -X POST "https://coremodels.example.com/graph/transform/plan/execute/$PROJECT_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d @- > out/product.replayed.json

Note the shape of the request: plan is the plan JSON as a string, which is why the snippet compacts it with jq -c and passes it with --arg. Same plan plus same source produces the same output. And when someone edits product.shex in a way the committed plan no longer covers, the gate rejects the stale plan loudly at validation time instead of executing something the reviewer never saw. That failure is a feature: it forces the mapping change through review — regenerate with schema/map, diff the plan in the pull request, commit.

Choosing a mapping kind for ShEx sources

All three strategies flow through the identical validation gate; they differ only in who authors the plan.

inferred matches by label and compatible type against the target hint (caseInsensitive defaults to true). ShEx decodes with friendly labels — schema:ProductShape becomes the type label Product, each predicate's local name labels its element — so inference works well when the two sides share vocabulary.

explicit is an authored guide, passed as JSON text, and it is where ShEx pays a structural dividend. ShEx predicates are IRIs, and our decoder lifts every one into the element's mapsTo annotation automatically (schema:name becomes https://schema.org/name). Set autoMatchByMapsTo and elements align by shared URI even when labels differ completely on the two sides:

{
  "autoMatchByMapsTo": true,
  "fieldMappings": [],
  "drops": ["schema:ProductShape|schema:internalNote"]
}

Two details worth knowing. First, ShEx element ids follow the scheme <shapeName>|<predicate> — that is the id you reference in drops or fieldMappings (sourceElementIds, targetElementIds, transformName). A drop you author is a decision, recorded in the plan, not an accident. Second, the guide parser rejects unknown keys with a path-carrying error — a typo like "fieldMapings" fails the call instead of silently executing a plan you did not intend.

ai asks a server-side model to propose the plan. In a pipeline, treat it as a plan generator, not a pipeline step: it requires Editor/Admin project membership and a server-configured Anthropic key, it sends schema content to the Anthropic API server-side, and its output goes through the same gate with at most one repair attempt. Run it once at authoring time, review the returned plan, commit that, and let CI replay the committed artifact with plan/execute — which needs neither the key nor the elevated role.

The nightly drift check

One last pattern that costs almost nothing: re-run Pattern 1 on a schedule and diff today's outputs against the committed ones. Deterministic encoders mean any diff is a real change — an upstream shapes edit, or a mapping that no longer covers the source — never formatting noise. Surfacing that in a scheduled job turns "the shapes and the service schema quietly diverged three months ago" into "the shapes changed last night, here is the diff and the ledger."

Shapes in git, plans in git, generated schemas in git, a gate on the ledger: that is the whole architecture. Each piece is one curl call on the same envelope. For the endpoint reference and the other formats the same pipeline can carry, see the transform section of the CoreModels docs.