The Catalog Knows Your Tables. It Doesn't Know What They Mean.

A crawler ran over the weekend. On Monday, a column in a table your finance dashboards depend on is no longer a bigint — a few malformed rows in a new S3 partition convinced the crawler it was looking at strings. Nothing failed. Nothing alerted. The Glue Data Catalog did exactly what it is designed to do: it recorded what the files look like now. The first person to notice is an analyst three days later, staring at a join that silently returns nothing.

If you run a data lake on AWS, some version of this story has happened to you. It is not a Glue bug. It is a category error we keep making about what a data catalog is.

An inventory is not a contract

The AWS Glue Data Catalog is a physical inventory. For every table it records a name, an S3 location, a storage descriptor, columns with Hive types, partition keys, and a bag of parameters. That is genuinely valuable — Athena, and everything else that treats Glue as its metastore, could not function without it.

But look at what the catalog cannot say. It has no concept of a primary key. It has no concept of nullability — not weakly supported, simply absent from the model. It cannot say that status may only ever contain three values, that subject_id must match an identifier in another table, or that col3 is a claim adjudication code and not a row counter. It records structure and stays entirely silent on meaning.

Teams paper over this gap with tribal knowledge: a wiki page from two re-orgs ago, a naming convention half the pipelines follow, the one engineer who remembers why the raw_events table has both ts and event_ts. Every one of those stopgaps decays, and none of them is machine-checkable.

Drift is the normal condition, not the exception

Warehouses with DDL-managed schemas drift when someone runs a migration. A lake catalog drifts whenever the data changes shape, because crawlers re-infer schemas from files. A partner starts sending a field as quoted numbers; a pipeline adds a column; someone re-partitions a bucket. The catalog updates itself — which is the feature — and every downstream assumption about that table is now unverified, which is the problem.

Without an independent record of what the schema is supposed to be, you cannot even ask the question "did anything drift?" The catalog is both the thing that changed and the only witness.

The struct-shaped blind spot

Then there are the columns the catalog describes only from the outside. Lake tables lean heavily on struct<>, map<>, and array<> types, and each one is a schema-within-a-schema that no governance process ever sees. A single payload struct<...> column can hide thirty fields, several of which are the ones your regulators actually care about. Ask most teams which of their struct columns contain identifiers or dates and the honest answer is: nobody has ever enumerated them.

The metadata nobody fills in

Finally, the mundane failures. Tables with no description at all — undocumented assets that resist both governance and any AI agent trying to ground itself in your estate. Tables missing the classification parameter, which means crawlers and Athena treat the format as unknown. None of this is hard to fix table by table; all of it is impossible to keep fixed across hundreds of tables and dozens of databases without something that checks continuously.

What a fix has to look like

We built the CoreModels Glue integration because the fix cannot be another wiki page, and it cannot be a tool that demands the keys to your AWS account either. The requirements fall out of the failure modes:

A governed model that lives beside the catalog, not inside it. You export your catalog yourself — one aws glue get-tables call per database, no credentials ever shared with us — and CoreModels imports it: tables become governed types identified as database.table, columns and partition keys become elements with their Hive types preserved verbatim, descriptions and classifications carried along. On top of that skeleton your team adds what Glue has no place for: definitions, allowed values, ownership of meaning.

An audit that measures the distance between the two. Once the governed model exists, every fresh catalog export can be checked against it — in CI, on a schedule, or on demand. A retyped column surfaces as a field-type-drift finding with the governed type and the observed type side by side. A vanished table is dataset-removed. Undocumented tables, missing classifications, and struct columns with ungoverned inner schemas each get their own named finding, so the blind spots at least have edges. And the report carries metrics that finally answer the coverage question with a number rather than an adjective: how many datasets in the estate are governed, and how many fields.

Honesty about what Glue doesn't say. Because the catalog declares no keys and no nullability, CoreModels invents none. Because a struct<> type cannot be faithfully flattened, it is approximated and the approximation is declared, not hidden. A governance layer that quietly fabricates precision would be a second version of the original problem.

The Monday-morning story ends differently with this in place: the crawler still updates the catalog, but the next audit run flags the drifted column by name, with evidence, before the dashboard lies to anyone. The catalog keeps doing what it is good at — being the inventory. The meaning finally has somewhere to live, and someone checking that it still holds.

If your lake is big enough that no one person can hold it in their head, that is exactly the point at which an inventory stops being enough. The AWS Glue quickstart in the CoreModels docs shows the full loop — export, import, audit — end to end.