How the category emerged

Storage was expensive and schemas had to be decided before any data could be kept.

Hadoop made it affordable to store everything on commodity hardware and decide what it meant later. Hive added a SQL interface and a metastore that mapped directories to tables, which is the ancestor of every catalog that followed.

Running Hadoop clusters to hold data meant paying for compute you were not using.

Object storage separated storage from compute properly. Data went to S3 and its equivalents, and clusters were started only when needed. Parquet became the default file format because columnar layout suited analytical scans.

Writing a thousand Parquet files to object storage is not atomic. A failed job left partial data, concurrent jobs overwrote each other, and there was no way to roll back.

Teams worked around it: write to a temporary prefix and swap, avoid concurrent writers by scheduling, or copy curated data into a warehouse and treat the lake as staging.

The workarounds did not scale, and the second copy was expensive in both money and trust.

Iceberg, Delta Lake, and Hudi each added a metadata layer over the same files: a manifest of which files belong to the table now, and an atomic pointer swap to commit. That single change restored ACID transactions, schema evolution, and time travel to data sitting in ordinary object storage.

A table format is only useful if engines implement it, and early support was uneven and read-only.

Spark, Trino, Flink, and later DuckDB and the cloud warehouses added read and then write support. The lakehouse became a practical architecture rather than a proposal, because you could finally use the engine that suited each job.

With several engines reading the same tables, the question moved from "can it read this" to "who is allowed to, and who decides".

The Iceberg REST catalog specification defined an open API for that authority. Polaris, Nessie, Lakekeeper, Unity Catalog, and the cloud catalogs now compete on governance, credential vending, and federation rather than on storage.

Language models and agents can write queries, but they cannot tell which of four revenue columns is the governed one.

Attention has moved up the stack to semantics: metric definitions, ontologies, and context that describes what data means rather than only where it is. The same governance the catalog provides for engines is now needed for agents.