After the Import: What a Governed Glue Catalog Changes Day to Day

The honest test of any governance tool is not the demo — it is what an ordinary Tuesday looks like six weeks after you adopted it. So instead of walking through features, let us walk through a week with a Glue Data Catalog that has a governed CoreModels project behind it, and contrast each moment with how the same moment used to go.

Monday: the import that starts it

A data engineer exports the catalog — one AWS CLI call per database, merged into a single JSON file — and imports it into a CoreModels project. No AWS credentials change hands at any point; the artifact is a file the team produced themselves. The response is a receipt, not a mystery: how many datasets were added, how many fields, whether the catalog snapshot was stored for later re-audits, and an explicit list of anything that had to be approximated on the way in.

What now exists is a governed model in which every Glue table is a type identified as database.table, carrying its S3 location as the physical name, its columns and partition keys as elements with Hive types preserved verbatim, and its descriptions wherever Glue had them. The import runs once with admin rights; everything the rest of the week needs is read-only.

Before: the catalog was the only description of the estate, and it described itself. After: there is a second, independent account of what the lake is supposed to look like — one you can annotate, review, and check against.

Tuesday: a question that used to take a meeting

An analyst wants to know whether event_type in the clinical events table is a free-text field or a controlled list, and what the values mean. The old ritual: open the AWS console, find the table, discover the column is typed string and described by nothing, then ask in a channel and wait.

Now the governed model answers directly. The team has spent the intervening weeks doing the one thing Glue gave them no place to do — attaching definitions and allowed values to the imported skeleton. And because every governed fact is also queryable by AI agents over MCP, the analyst's assistant can answer the question without a human in the loop at all.

Wednesday: drift arrives, and bounces off CI

An upstream team ships a pipeline change: one column quietly disappears from a shared table, and another comes back under a wider type. Previously this was invisible until something downstream broke — the catalog updated itself, and no independent record existed to disagree with it.

This week it goes differently. The team's pipeline calls the audit endpoint on the machine-to-machine API surface with a fresh catalog export; the build fails because the error count is nonzero. The findings name the missing column and the retyped one — governed type and observed type side by side — and the report's markdown lands in the build's job summary, where a reviewer can read it without decoding JSON. The team decides, deliberately: fix the data, or accept the new type and change the governed model. Either way it is a decision, made on Wednesday — not an incident, discovered next Monday.

Thursday: the model changes, and the check runs backwards

Governance cuts both ways. A data steward tightens the governed model — renames an element's meaning, narrows an allowed-value list. Does the last-known catalog still conform? Nobody re-exports anything: a single re-audit call replays the catalog snapshot stored at import time against the current governed model, and the run is recorded into the project's rolling audit history. Drift checking works in both directions — fresh estate against the model, changed model against the last-known estate — and the history accumulates into a trail you can point auditors at.

Before: a model change was an act of faith about its blast radius. After: the blast radius is a report, available in one call, on record.

Friday: the badge, the DDL, and the quiet win

Two small things close the week. First, the repository README carries a status badge served straight from the audit history — green when the latest recorded run is clean, yellow for warnings, red for errors. Schema health became ambient information; nobody has to ask.

Second, a new table needs to exist in the lake. Instead of hand-writing DDL from memory, the team generates it from the governed model: Athena-ready CREATE EXTERNAL TABLE statements in which every column comment carries the governed description — allowed values included — partition keys become the PARTITIONED BY clause, and the S3 location is a deliberate placeholder for a human to fill in. The generated file states its own philosophy in its header: meaning changes belong in CoreModels. The DDL is output, not authority.

What actually changed

Nothing about AWS changed. The crawlers still crawl, Athena still queries the same catalog, and CoreModels still holds no credentials and no connection into the account. What changed is the direction of information flow. Schema knowledge used to live in heads and be verified by accident; now it lives in a governed model and is verified by machinery — on every pull request, on every model change, on a badge anyone can see.

The compounding effect is the part that is hard to demo: every drift caught on Wednesday instead of next week, every question answered on Tuesday without a meeting, every table created on Friday with its documentation already attached. None of these is dramatic alone. Six weeks of them is the difference between a lake people trust and a lake people merely use.

The AWS Glue quickstart in the CoreModels docs walks the whole loop — export, import, audit, re-audit, badge, generate — with the exact calls.