OpenDataLakehouse
Menu

Definition

What is an open lakehouse?

An open lakehouse is a data architecture that stores data in open file and table formats on commodity object storage, so that any compliant engine can read and write it without vendor lock-in.

It combines the scale and low cost of a data lake with the transactional guarantees of a data warehouse, and it does so without putting the data inside any one vendor's system. The data stays in your object storage, in formats that are publicly specified, and engines are attached to it rather than the other way round.

The three claims packed into the term

"Open lakehouse" is three separate assertions, and a system can satisfy one without satisfying the others. Pulling them apart is the fastest way to judge whether something actually qualifies.

Claim 1

Open

The file format, the table format, and the catalog API are publicly specified and independently implementable. More than one engine can read and write the same tables. Nothing about the storage layer requires a particular vendor.

Claim 2

Lake

Data sits on commodity object storage that you own, in files you can list and copy. Storage and compute scale independently and are billed separately, which is what makes retaining everything affordable.

Claim 3

House

Those files behave like real tables: ACID transactions, schema enforcement and evolution, snapshot isolation between concurrent writers, and the ability to query or roll back to an earlier state.

How the layers fit together

An open lakehouse is assembled from layers, each with one job and a choice of implementations. The value of the architecture comes from the fact that you can swap any single layer without disturbing the others.

  1. Agents & AI 18 entries How do agents use any of this safely? Agentic analyticsRAGText-to-SQLVector search
  2. Semantics 12 entries What does this data mean? Semantic layerMetricsOntologyModeling
  3. Movement 15 entries How does data get in, and stay current? AirflowdbtDagsterCDCStreaming
  4. Interchange 3 entries How does data move between systems? Apache ArrowArrow FlightFlight SQL
  5. Compute 43 entries What actually runs the query? SparkTrinoFlinkDuckDBDremio
  6. Catalog 17 entries How do engines find and govern those tables? Apache PolarisIceberg RESTNessieGlueUnity
  7. Table format 50 entries What makes those files behave like a table? Apache IcebergDelta LakeApache HudiApache Paimon
  8. Storage format 26 entries How is the data physically written down? Apache ParquetORCAvroObject storage
  9. Foundations 17 entries What kind of system are we building? LakehouseData lakeData warehouseMedallion

Each layer links to the reference entries covering it. The reference architecture walks the same stack in more depth, including where the boundaries between layers tend to blur in practice.

What it replaced

The architecture exists because of a specific failure. The first generation of cloud data lakes wrote Parquet and CSV files to object storage with no transaction management. Partial writes left tables in corrupted states, concurrent jobs produced silently wrong results, and there was no reliable way to undo a bad load.

The common workaround was to keep the lake for raw data and copy a curated subset into a warehouse for anything that needed correctness. That meant maintaining two systems, two copies, and a pipeline between them that could drift.

Open table formats collapsed that split. By adding a metadata layer over the same files, they made the lake transactional, which removed the reason to copy the data into a warehouse at all. The history of the category traces how that happened.

Common questions

What makes a lakehouse "open"?

Openness is a property of the formats and interfaces, not the vendor. A lakehouse is open when the data sits in published file and table formats that anyone can implement, the catalog speaks a documented API, and no single engine is required to read or write the tables. If replacing your query engine means migrating your data, the architecture is not open.

How is an open lakehouse different from a data warehouse?

A warehouse couples storage and compute inside one system that owns the data in a proprietary internal format. An open lakehouse separates them: the data lives in your own object storage in open formats, and engines are attached to it. You gain engine choice and avoid egress-style lock-in, at the cost of assembling more of the stack yourself.

How is it different from a data lake?

A data lake is files on object storage with no transactional guarantees. Concurrent writers corrupt state, there is no schema enforcement, and there is no reliable way to undo a mistake. An open lakehouse adds a table format over those same files, which restores ACID transactions, schema evolution, and time travel without moving the data anywhere.

Do I need Apache Iceberg to build one?

No. Iceberg is the most widely adopted open table format, but Delta Lake, Apache Hudi, and Apache Paimon occupy the same layer. What matters architecturally is that the table format is openly specified and implemented by more than one engine.

Is a catalog required?

In practice, yes, once more than one engine is involved. The table format defines what a table is; the catalog tracks which tables exist, where their current metadata lives, and who may access them. Without a shared catalog, engines cannot safely agree on the current state of a table.

Does open mean free?

No. Open refers to the formats and specifications, not the price of the software that reads them. Commercial engines and managed catalogs are common in open lakehouses. The distinction that matters is whether you could switch those components without rewriting your data.