Ten Minutes to a Governed Glue Catalog

Open a terminal. Everything in this tutorial is one AWS CLI command and two HTTP calls, and at the end of it your AWS Glue Data Catalog has a governed twin in CoreModels plus a first drift audit you can read line by line. Nothing gets installed in your AWS account, no IAM role is granted to us, and no credential of yours travels anywhere: you run the CLI, you upload the JSON it printed.

We use https://coremodels.example.com as the API base URL, $TOKEN as your CoreModels login token, and $PROJECT_ID as the 32-character hex id of the CoreModels project that will govern the estate. You need curl and jq locally, plus the AWS CLI authenticated against the account that owns the catalog.

Two roles matter. Import writes to the graph and requires the Admin role on the project. Audit never writes and requires only Viewer — which is why the automation you build later can run with a deliberately weak key.

The artifact contract

The connector is registered under the vendor key glue, display name AWS Glue Data Catalog, and it asks for exactly one artifact:

tables — required — aws glue get-tables JSON (one response, an array of responses, or a bare TableList)

That permissiveness is deliberate. The parser walks whatever you send it and collects table objects from three envelopes: a single {"TableList": [...]} response, an array of such responses, or a bare array of table objects. If the payload is valid JSON but contains no named tables, you get an explicit error rather than a successful import of nothing.

Step 1 — Export the catalog

aws glue get-tables --database-name lake > tables.json

If your estate spans several Glue databases, run the command once per database and merge the responses into a single array:

aws glue get-tables --database-name lake            > db-lake.json
aws glue get-tables --database-name reference_data  > db-reference.json
jq -s '.' db-lake.json db-reference.json > tables.json

Merging is safe because a table's governed identity is database.table, so tables from different databases can never collide. One consequence worth knowing up front: the vendor-side project name recorded for the import is taken from the first table's DatabaseName, so a merged export will be labeled with whichever database came first.

Step 2 — Import

Build the body with jq so the file content is embedded as a correctly escaped JSON string — never paste a multi-megabyte export onto a command line.

jq -n --rawfile tables tables.json '{artifacts: {tables: $tables}}' > import-request.json

curl -sS -X POST \
  "https://coremodels.example.com/graph/integrations/glue/import/$PROJECT_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  --data-binary @import-request.json

Step 3 — Read the counts

For a small lake database holding two tables — an events table with four columns and one partition key, and a user_profiles table with three columns — a first import answers:

{
  "success": true,
  "vendor": "glue",
  "projectName": "lake",
  "datasetsAdded": 2,
  "datasetsSkippedExisting": 0,
  "fieldsAdded": 0,
  "lineageEdgesAdded": 0,
  "lineageEdgesSkipped": 0,
  "nodesEnriched": 10,
  "snapshotStored": true,
  "lossiness": [
    {
      "kind": "TypeApproximation",
      "path": "lake.events.payload",
      "explanation": "Native type 'struct<action:string,value:double>' was approximated as String; the exact native type is preserved in the vendor metadata mixin."
    },
    {
      "kind": "TypeApproximation",
      "path": "lake.user_profiles.email",
      "explanation": "Native type 'varchar(255)' was approximated as String; the exact native type is preserved in the vendor metadata mixin."
    }
  ],
  "errors": []
}

Four of those numbers routinely surprise people, so here is what each one actually counts.

datasetsAdded: 2 — two tables became governed Types. Their columns and partition keys came in with them as Elements, which is why fieldsAdded reads 0: that counter is specifically "columns added to datasets that were already governed". On a first import it is always zero; on later re-imports it is the number that tells you the lake grew a column.

nodesEnriched: 10 — every Type and every Element received its vendor-metadata value: two tables plus eight columns (four events columns, one events partition key, three user_profiles columns).

lineageEdgesAdded: 0 — expected, permanently. get-tables output declares no dependencies between tables, and we do not infer lineage from naming conventions. The same restraint applies to keys and nullability: Glue declares neither, so the governed model contains neither.

snapshotStored: true — the parsed catalog was persisted next to the project. That is what makes a one-call re-audit possible later without re-exporting anything.

And lossiness is not an error list. It is the honesty channel: the import succeeded and it tells you exactly which two columns are governed by an approximation. The verbatim Hive type (struct<action:string,value:double>, varchar(255)) is preserved on the node's vendor metadata regardless — the approximation only concerns the cross-vendor primitive the graph reasons with.

Step 4 — The first audit

The audit compares fresh artifacts against the live governed model. It is read-only and runs at Viewer role. Run it now with the same file and you have your baseline:

jq -n --rawfile tables tables.json \
  '{artifacts: {tables: $tables}, recordHistory: true}' > audit-request.json

curl -sS -X POST \
  "https://coremodels.example.com/graph/integrations/glue/audit/$PROJECT_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  --data-binary @audit-request.json

recordHistory defaults to false — the audit verb stays strictly read-only unless asked. Set it to true when you want the run appended to the project's rolling trail, which is also what feeds the status badge.

Step 5 — Read the result

{
  "success": true,
  "vendor": "glue",
  "projectName": "lake",
  "errorCount": 0,
  "warningCount": 1,
  "infoCount": 2,
  "codes": {
    "semi-structured-column": 1,
    "table-no-description": 1,
    "classification-missing": 1
  },
  "driftedObjects": [],
  "fingerprint": "3f9c1d5a7b2e4086",
  "metrics": {
    "Datasets (estate)": "2",
    "Datasets governed": "2 / 2",
    "Fields governed": "8 / 8",
    "Governed nodes with canonical mappings": "0 / 10 (0%)",
    "Last import": "2026-08-04T09:14:22.1043117+00:00"
  },
  "findings": [
    {
      "section": "Conformance",
      "severity": "Warning",
      "code": "semi-structured-column",
      "subject": "lake.events",
      "message": "1 struct/map/array column(s) carry ungoverned inner schemas.",
      "detail": "payload"
    },
    {
      "section": "Conformance",
      "severity": "Info",
      "code": "table-no-description",
      "subject": "lake.user_profiles",
      "message": "Catalog table has no description — undocumented lake assets resist governance and agent grounding.",
      "detail": null
    },
    {
      "section": "Conformance",
      "severity": "Info",
      "code": "classification-missing",
      "subject": "lake.user_profiles",
      "message": "Table has no classification parameter — crawlers/Athena treat the format as unknown.",
      "detail": null
    }
  ],
  "markdown": "# glue Schema Audit — lake\n\n🟡 **0 errors · 1 warnings · 2 info**\n…",
  "historyRecorded": true,
  "lossiness": []
}

Read it in this order.

errorCount first. Zero means nothing in the catalog violates governed meaning. This single integer is the whole contract that automation keys off later.

Then metrics. Datasets governed: 2 / 2 and Fields governed: 8 / 8 confirm the import landed completely. Governed nodes with canonical mappings: 0 / 10 (0%) counts how many of your governed nodes also carry a mapping to a non-vendor standard — right after an import that is naturally zero, and it is the number that moves as you ground your model in shared vocabularies.

Then findings. Every finding carries a section (Coverage, Drift, or Conformance), a severity (Info, Warning, Error), a stable kebab-case code, the subject in vendor identity terms, a human message, and an optional detail. Coverage is what the catalog has and the model does not (dataset-unmapped, field-unmapped) — clean here, because we just imported. Drift is where the two disagree; driftedObjects collects those subjects separately for exactly this reason. Conformance is where the Glue connector's own rules live, and all three fired on this tiny estate:

• semi-structured-column (Warning) — lake.events has a struct column whose inner schema is not governed. One aggregate finding per table, with the column names in detail, because a 40-field struct is one decision to make, not 40 rows of noise.
• table-no-description (Info) — lake.user_profiles is undocumented.
• classification-missing (Info) — lake.user_profiles has no classification parameter, so crawlers and Athena treat its format as unknown.

None of those fail a build. They tell you where the catalog is thinner than your governance deserves — and they are a backlog you generated for the price of one API call.

Finally markdown. The same report rendered for humans: a verdict line, the metrics table, and each section folded into a collapsible block. Paste it into a review, an incident channel, or a job summary as-is.

Where to go from here

You now have a governed baseline, a stored catalog snapshot, and one recorded run. Three obvious next moves, each a single call: GET .../glue/status/$PROJECT_ID to see the last-import bookkeeping; POST .../glue/reaudit/$PROJECT_ID with an empty JSON body ({}) to re-check the stored snapshot after someone edits governed meaning; and POST .../glue/generate/$PROJECT_ID to get Athena/Hive CREATE EXTERNAL TABLE DDL back out of the governed model.

The AWS Glue quickstart in the CoreModels documentation collects every route, payload, and caveat on one page.