What Happens to a Glue Table on Its Way into the Graph

Take one entry out of an aws glue get-tables response — a table called events in database lake, a few columns, one partition key, a classification parameter, an S3 location — and follow it into CoreModels. By the end it is a governed Type with Elements, a vendor identity, a metadata mixin, an entry in an estate snapshot, and a set of audit rules watching it. This is that journey, plus the honest edges: what is approximated, what is deliberately not invented, and which behaviors will surprise you.

One neutral estate model

Every CoreModels connector parses its vendor's artifacts into the same neutral shape: datasets carrying fields and normalized checks, plus lineage edges between datasets. Vendor-specific detail that has no neutral slot rides in Meta bags rather than becoming new top-level concepts. The Glue connector itself is pure — it parses, maps native types, contributes audit rules and generates artifacts, and never touches the graph. All graph I/O lives in vendor-neutral services, which is why the Glue mapping behaves like every other connector's once parsing is done.

Parsing: what the connector reads

The tables artifact may arrive as a single {"TableList": […]} response, an array of such responses, or a bare array of table objects; the collector walks all three. Here is the input, trimmed to the keys the connector actually reads:

{
  "Name": "events",
  "DatabaseName": "lake",
  "Description": "Raw product events landed from Kinesis.",
  "TableType": "EXTERNAL_TABLE",
  "Parameters": { "classification": "parquet" },
  "StorageDescriptor": {
    "Location": "s3://example-lake/events/",
    "Columns": [
      { "Name": "event_id", "Type": "string", "Comment": "Event key." },
      { "Name": "user_id", "Type": "bigint" },
      { "Name": "payload", "Type": "struct<action:string,value:double>", "Comment": "Raw payload." }
    ]
  },
  "PartitionKeys": [
    { "Name": "event_date", "Type": "date", "Comment": "Partition." }
  ]
}

A table object needs a Name to be considered at all. From each one:

• Identity is DatabaseName and Name joined with a dot — lake.events. Empty segments are skipped, so a table object without DatabaseName gets the bare name as its identity.
• Kind comes from TableType (default EXTERNAL_TABLE): a value containing VIEW becomes a view with materialization view, anything else a table with materialization external_table.
• Physical name is StorageDescriptor.Location, falling back to the identity when absent.
• Classification is the classification entry from Parameters, lifted into the dataset's meta bag as glue.classification.
• Fields are Columns followed by PartitionKeys, in that order, de-duplicated case-insensitively (first wins). Each keeps its Comment and its Type verbatim; partition keys additionally carry glue.partitionKey in their own meta bag.

The whole artifact is hashed into a 16-character fingerprint at parse time; that value travels into audit reports, history records and the stored snapshot, and is what proves two runs saw byte-identical content. Two refusals are explicit rather than silent: an artifact that is not valid JSON, and one that parses but yields no named tables.

Two write paths, one import

Schema shape — types, fields, enums, references — travels through the intermediate representation the whole CoreModels transform stack shares, written by the same hardened graph writer every format import uses. Estate facts it cannot hold — lineage, checks, provenance, last-import state — are written natively by the integration enricher. For Glue the first path does little beyond structure, because Glue declares little:

Estate fact	Becomes	Notes for Glue
table	governed Type	label = table name
column / partition key	Element on that Type	in catalog order
accepted values	Taxonomy + controlled list	never — Glue declares none
foreign key / reference	expected-type relation	never — Glue declares none
not-null check	required flag	never — Glue declares none
vendor identity	mapsTo { standard: "glue", uri: "lake.events" }	queryable, survives exports
native type, description, tags, meta, materialization, physical name	vendor-metadata mixin	see below
lineage edge	Depends On relation	never — get-tables declares no dependencies

That table is the connector's design rule in one view: preserve verbatim, never invent. A governed model built from Glue has no primary keys and no nullability, because the catalog declares none — a guessed constraint would poison every audit that followed.

Node ids, labels and identity

CoreModels node ids must be camelCase alphanumerics, so identities are sanitized: lake.events becomes the Type id lakeEvents, and its event_id column the Element id lakeEventsEventId (dataset id plus the PascalCased field name — elements are per-dataset). The native names survive on the node's label, its mapsTo value and the mixin, and audit, reconciliation and re-import all join on the mapsTo identity — never on the id or the label.

Sanitization folds separators away, so two distinct catalog identities could in principle collapse onto one id. Rather than silently merging two tables, the import refuses and names both sources: "Distinct vendor identities collapse onto the same graph id after sanitization … Rename one of them in the vendor estate."

The type map

Alongside the verbatim Hive type, each Element gets a governed primitive so the graph can reason across vendors:

Hive type	Governed primitive	Approximated?
int, integer, bigint, smallint, tinyint	Integer	no
double, float	Double	no
decimal(p,s)	Double	yes — precision and scale are not carried
boolean	Boolean	no
timestamp, date	DateTime	no
string	String	no
char(n), varchar(n)	String	yes, when a length is declared
binary, array<…>, map<…>, struct<…>, uniontype<…>	String	yes

Every approximation is reported as a TypeApproximation lossiness record on the import response, naming the column: "Native type 'structaction:string,value:double' was approximated as String; the exact native type is preserved in the vendor metadata mixin." That is also why generated DDL re-emits decimal(19,4) and struct<…> exactly: the primitive is the cross-vendor working type, the recorded Hive string remains the authority.

What rides the vendor-metadata mixin

Every imported node gets one mixin value on a mixin type called AWS Glue Data Catalog Metadata; property ids are the vendor key plus the property name. On a Type: glueUniqueId (the catalog identity), glueResourceKind, glueMaterialization, glueRelationName (the S3 location), glueContract, glueAccess, glueDescription, glueTags, glueMeta (the meta bag as JSON — this is where glue.classification lives), and glueChecks. On an Element: glueDataType (the verbatim Hive type), glueDescription, glueTags, glueMeta (where glue.partitionKey lives) and glueChecks.

Note glueDescription in both lists: CoreModels has no schema-level description slot, so the catalog's Description and each column's Comment ride the mixin and are read back from there by generation and audit.

These values are the one thing an import refreshes rather than merely adds: they describe the estate, not the meaning. Governed nodes are never mutated by an import. That asymmetry is the posture in one line — import is additive, drift is the audit's job, and applying a meaning change is a human act.

Two more nodes appear beside the model. A state node, labeled AWS Glue Data Catalog Integration — lake, records the last import: vendor, project name, timestamp, fingerprint, counts and parser facts. An estate snapshot node stores the parsed catalog itself, gzip-compressed and base64-encoded, so re-audits can run without fresh artifacts. It is size-capped (roughly 1.5 MB encoded); over the cap, import reports snapshotStored: false with a lossiness record rather than failing.

The rules that watch it afterwards

Audits run in three sections against the live governed graph — no stored fingerprints to go stale, the graph is the baseline.

Coverage — dataset-unmapped (Warning: a catalog table nothing governs) and field-unmapped (Info: a column added since the last import).

Drift — dataset-removed (Error), field-removed (Error), field-type-drift (Error), enum-narrowed (Error), enum-widened (Warning), enum-constraint-removed (Warning) and contract-drift. A drift finding names both sides:

{ "section": "Drift", "severity": "Error", "code": "field-type-drift",
  "subject": "lake.events.user_id",
  "message": "Field type changed since the last import.",
  "detail": "governed: bigint, estate: string" }

Conformance — the three rules the Glue connector contributes:

Code	Severity	Fires when
table-no-description	Info	the catalog table has no description
classification-missing	Info	a table (views are exempt) has no classification parameter, so crawlers and Athena treat the format as unknown
semi-structured-column	Warning	the table has struct / map / array / uniontype columns — one aggregate finding per table, column names in the detail

Aggregating that last one is deliberate: a table with a 40-field struct is one governance decision, not 40 rows of noise.

Generation, in reverse

Generation emits coremodels_glue_tables.sql — Athena/Hive CREATE EXTERNAL TABLE IF NOT EXISTS statements. Each column prefers its recorded native type and falls back to the primitive mapping (Integer → bigint, Double → double, Boolean → boolean, DateTime → timestamp, otherwise string) for governed elements that never came from Glue; reference-typed elements are emitted as bigint. Descriptions and taxonomy allowed-values compose into column COMMENTs, partition-flagged elements return to PARTITIONED BY, tables are STORED AS PARQUET, and the LOCATION is a placeholder — the model knows your schema, not your bucket layout. Views and element-less types are skipped as declared StructuralDrop lossiness.

The edges worth knowing

• Governing a taxonomy on a Glue column produces a standing warning. Glue never declares accepted values, so once an element carries a governed taxonomy, every audit reports enum-constraint-removed. That is literally true, and for a Glue-only estate it is permanent. Expect it, and gate on codes rather than raw warning counts.
• contract-drift is inert here. Glue has no contract-enforcement flag, so both sides of the comparison are always false.
• dataset-removed is namespace-scoped. It only fires for governed datasets whose database prefix appears in the export being audited, so auditing one database never claims another database's tables disappeared.
• Merging databases is safe; the report name is first-come. Identities are database.table, so merged exports cannot collide — but the recorded vendor-side project name is taken from the first table seen.
• Cross-estate reconciliation matches on the physical relation. For Glue that is the S3 location, not database.schema.table, so pairing a Glue table with a warehouse table only works where those strings genuinely agree.
• Imports are not transactional. If a write fails midway the nodes already written stay, and the error says so — re-running is additive and safe by design.

Every one of these is reported in-band: as a lossiness record, a finding, an explicit flag, or an error that names the fix. The governed graph holds exactly what your catalog declared, annotated with exactly what it could not hold. For the operational side — extraction recipe, routes, payloads, the CI gate — see the AWS Glue quickstart in the CoreModels documentation.