Plans, Not Scripts: Repeatable LinkML Conversion Pipelines

A conversion you cannot reproduce is not a pipeline; it is a favor someone did once. The interesting question for automation is not "can this tool turn LinkML into Postgres DDL" — it is "will the run in six months, on a build agent, with a token nobody remembers issuing, produce the same bytes and tell me if it did not."

CoreModels answers that with three properties, and everything below is built on them.

The three guarantees

The encoder is deterministic. Our LinkML output is written by hand, not handed to a general-purpose YAML serializer: fixed key order, two-space indent, a fixed quoting rule for keys and values. The same model always produces the same text, which is what makes an exported .yaml safe to commit — a diff means the model changed, never that a serializer reordered a map.

The executor is deterministic. The engine takes a validated plan and a source and emits a target with no I/O, no network, and no model calls anywhere in the path. Same plan plus same source yields the same output, every time. That is the property plan/execute sells.

The gate is universal. Every strategy — an authored guide, label inference, an AI proposal, or a plan you stored last quarter — goes through the identical validator before anything executes. Deserializing a plan grants it no engine-side privilege; a stored plan earns no shortcut.

The plan is the artifact

Every mapping call returns plan next to the result. It is small, readable JSON, and it is the thing you should be versioning:

{
  "operations": [
    {
      "kind": "Drop",
      "origin": "Explicit",
      "sourceNodeId": "SensorSamplingIntervalS",
      "sourceKind": "Element",
      "reason": "Dropped by the SIA guide.",
      "declaredLossiness": [
        { "kind": "StructuralDrop",
          "path": "Element[SensorSamplingIntervalS]",
          "explanation": "Dropped by the SIA guide." }
      ]
    },
    { "kind": "TypeMapping", "origin": "Explicit",
      "sourceTypeId": "Sensor", "targetTypeId": "Sensor", "targetLabel": "Sensor" },
    { "kind": "ElementMapping", "origin": "Explicit",
      "sourceElementIds": ["SensorName"], "targetElementIds": ["SensorName"] }
  ]
}

The anatomy is worth learning, because reviewing plans is the leverage point:

• kind is one of TypeMapping, ElementMapping, TaxonomyMapping, RelationMapping, ComponentMapping, Drop. Anything else is refused on parse with Unknown operation kind '...'. Use: TypeMapping | ElementMapping | TaxonomyMapping | RelationMapping | ComponentMapping | Drop.
• origin is Explicit, Inferred, or AiAssisted — the provenance of that specific line. A reviewer can scan for Inferred and AiAssisted and ignore the rest.
• confidence appears when the producing strategy scores itself, between 0 and 1. It orders human review. The gate neither reads it nor trusts it: a 0.99 operation is validated exactly like a 0.4 one.
• declaredLossiness travels with the operation, so a stored plan replays with the same honesty it was produced with.

Store the plan next to the schema it maps. plan/execute needs only Viewer, so the review-once/replay-forever split is a real privilege boundary: a human approves the plan, and the pipeline that replays it never needs write access to anything.

Choosing a mapping kind for LinkML

inferred matches labels between source and target and checks that the value types are compatible (two primitives must be the same kind; references must be the same shape). caseInsensitive defaults to true. It needs a target hint, always. For a straight format conversion, hand it the source document as its own hint — every label matches itself, the plan covers everything, and the ledger stays empty. For a projection onto a model you already govern, pass that model as the hint (or set useProjectAsTargetHint: true) and read the drops.

explicit is the one that pays off in a pipeline, and it has a LinkML-specific superpower. The guide's backbone is autoMatchByMapsTo: source and target constructs that declare the same cross-standard IRI are aligned automatically. LinkML is unusually good at supplying those IRIs, because class_uri, slot_uri, and permissible-value meaning are exactly that — declared meaning, expanded through your prefixes: block into full IRIs. A LinkML schema that grounds its terms is a schema whose mappings largely write themselves.

{
  "autoMatchByMapsTo": true,
  "fieldMappings": [],
  "taxonomyDirectives": [],
  "drops": ["SensorSamplingIntervalS"]
}

Fed that guide with a LinkML source and a LinkML target hint, the engine produced the three-operation plan above: the explicit drop, plus two alignments that came from meaning alone — the Sensor class (which declares class_uri: schema:Thing) and the name slot (which declares slot_uri: schema:name). The ledger reported the rest without editorializing:

[
  { "kind": "StructuralDrop", "path": "Element[SensorSamplingIntervalS]",
    "explanation": "Dropped by the SIA guide." },
  { "kind": "StructuralDrop", "path": "Type[Sensor].SensorSensorId",
    "explanation": "Element is a member of the mapped type but no operation maps or drops it." },
  { "kind": "StructuralDrop", "path": "Type[Sensor].SensorInstalledOn",
    "explanation": "Element is a member of the mapped type but no operation maps or drops it." }
]

That is the lesson in one screen: sensor_id and installed_on carry no declared meaning, so meaning-based matching had nothing to work with, and the ledger says so by name rather than quietly shipping a two-column table. The fix is a modeling fix — give those slots a slot_uri — not a tooling workaround.

Guides are validated strictly. A typo is an error, not a silently ignored key: Unknown mapping-guide key 'fieldMapping'. Known keys: autoMatchByMapsTo, fieldMappings, taxonomyDirectives, drops. The alternative — executing a plan you did not intend because a key was misspelled — is not a trade we make.

ai proposes a plan with Claude and then submits it to the same gate, with at most one repair attempt; a rejected repair is a rejection. Its operations arrive stamped AiAssisted, so they are visible in review forever after. Two facts to design around: it requires Editor or Admin membership on the scoping project, and it sends the schema content to the Anthropic API server-side. Our recommendation for scheduled pipelines is to use ai interactively, capture the plan it produced, review it, commit it, and let the schedule run plan/execute — which is deterministic, Viewer-level, and never calls out.

Batch conversion

The stateless routes make bulk work a shell loop. Converting a directory of LinkML files to Postgres DDL, failing the run if anything is lost:

#!/usr/bin/env bash
set -euo pipefail
API="https://coremodels.example.com"
mkdir -p out

for f in schemas/*.yaml; do
  base="$(basename "$f" .yaml)"
  resp="$(jq -n --rawfile s "$f" \
    '{sourceFormat:"linkml", sourceSchema:$s,
      targetFormat:"sql", vendor:"postgres",
      targetHintFormat:"linkml", targetHintSchema:$s,
      mapping:{kind:"inferred"}}' \
  | curl -sf "$API/graph/transform/schema/map/$PROJECT_ID" \
      -H "Authorization: Bearer $TOKEN" \
      -H "Content-Type: application/json" \
      -d @-)"

  jq -e '.success' <<<"$resp" >/dev/null || { echo "FAILED $base"; jq '.errors' <<<"$resp"; exit 1; }

  jq -r '.schema' <<<"$resp" > "out/$base.sql"
  jq    '.plan'   <<<"$resp" > "out/$base.plan.json"

  loss="$(jq '[.lossiness[] | select(.kind == "StructuralDrop")] | length' <<<"$resp")"
  if [ "$loss" -gt 0 ]; then
    echo "::error::$base dropped $loss construct(s)"
    jq -r '.lossiness[] | "  [\(.kind)] \(.path): \(.explanation)"' <<<"$resp"
    exit 1
  fi
  jq -r '.lossiness[] | "  [\(.kind)] \(.path): \(.explanation)"' <<<"$resp"
done

Three things make this a gate rather than a script. The .success check separates "could not proceed" from "ran with consequences". The severity filter is yours to set — here StructuralDrop fails the build while ConstraintRelaxation and TypeApproximation are printed and tolerated, which is the right default when the target is SQL and you already know an enum becomes a VARCHAR. And out/$base.plan.json is written every time, so the mapping that produced today's DDL is an artifact, not a memory.

Pinning a pipeline to reviewed plans

Once a plan has been reviewed, stop re-deriving it. Replace the mapping call with a replay:

for f in schemas/*.yaml; do
  base="$(basename "$f" .yaml)"
  jq -n --rawfile s "$f" --rawfile p "plans/$base.plan.json" \
    '{sourceFormat:"linkml", sourceSchema:$s, plan:$p,
      targetFormat:"sql", vendor:"postgres"}' \
  | curl -sf "$API/graph/transform/plan/execute/$PROJECT_ID" \
      -H "Authorization: Bearer $TOKEN" \
      -H "Content-Type: application/json" \
      -d @- | jq -r '.schema' > "out/$base.sql"
done

Note plan is passed as a string — the JSON text of the plan, not a nested object. If the source schema has drifted in a way the plan no longer fits, this fails at the gate with a path-carrying error instead of quietly producing something different. That is the tripwire: the plan is a contract between a reviewed model and a generated artifact, and when the model breaks the contract the build stops.

The same loop works in reverse. Export the governed model as LinkML on every merge, commit the file, and let code review do the rest — the encoder's byte-stability means a diff in that file is always a real change in the model. Pair it with a second export in whichever format your consumers actually run, and you have a repository where the semantic model and its generated shapes are checked in side by side, each traceable to the plan that produced it.

The mapping-guide wire format, the plan schema, and the roles each route requires are documented in the Schema Transformation API reference in the CoreModels docs.