Reference architecture
How an open lakehouse is assembled
Each layer has one responsibility and several implementations. The architecture earns its keep when you can replace any single layer without touching the others, so the useful way to read this is as a set of independent decisions rather than a product.
- Agents & AI 18 entries How do agents use any of this safely? Agentic analyticsRAGText-to-SQLVector search
- Semantics 12 entries What does this data mean? Semantic layerMetricsOntologyModeling
- Movement 15 entries How does data get in, and stay current? AirflowdbtDagsterCDCStreaming
- Interchange 3 entries How does data move between systems? Apache ArrowArrow FlightFlight SQL
- Compute 43 entries What actually runs the query? SparkTrinoFlinkDuckDBDremio
- Catalog 17 entries How do engines find and govern those tables? Apache PolarisIceberg RESTNessieGlueUnity
- Table format 50 entries What makes those files behave like a table? Apache IcebergDelta LakeApache HudiApache Paimon
- Storage format 26 entries How is the data physically written down? Apache ParquetORCAvroObject storage
- Foundations 17 entries What kind of system are we building? LakehouseData lakeData warehouseMedallion
Drawn top to bottom as the stack is built: storage at the base, agents at the top. Each row links to the reference entries for that layer.
Layer by layer
Every layer below answers one question, and the choice you make there constrains the layers above it more than the ones below.
What kind of system are we building?
The architectural ideas the rest of the stack assumes: what a lakehouse is, and how it differs from a lake or a warehouse.
How is the data physically written down?
How bytes are laid out on object storage, and how they are encoded and compressed for analytical reads.
| Option | What distinguishes it |
|---|---|
| Apache Parquet | Columnar, the default for analytical reads |
| ORC | Columnar, common in Hive-era estates |
| Avro | Row-oriented, used for metadata and streaming payloads |
| Object storage | S3, GCS, Azure Blob, MinIO, or on-premise equivalents |
What makes those files behave like a table?
The metadata that turns a directory of files into a table with transactions, schema evolution, and time travel.
| Option | What distinguishes it |
|---|---|
| Apache Iceberg | Widest multi-engine support |
| Delta Lake | Strong within the Databricks ecosystem |
| Apache Hudi | Built around upserts and incremental pulls |
| Apache Paimon | Streaming-first, close ties to Flink |
How do engines find and govern those tables?
The service that tracks which tables exist, where their metadata lives, and who is allowed to touch them.
| Option | What distinguishes it |
|---|---|
| Apache Polaris | Open catalog with RBAC and credential vending |
| Iceberg REST | The API specification the others implement |
| Project Nessie | Git-style branching and tagging |
| AWS Glue | Managed, deeply integrated with AWS |
| Unity Catalog | Governance across the Databricks estate |
What actually runs the query?
The engines that plan and execute queries, and the optimisations that make them fast over object storage.
| Option | What distinguishes it |
|---|---|
| Apache Spark | Batch and large-scale transformation |
| Trino | Interactive federated SQL |
| Apache Flink | Stream processing |
| DuckDB | Single-node analytics, increasingly on lakehouse tables |
| Dremio | Lakehouse query engine with a semantic layer |
How does data move between systems?
The in-memory format and wire protocols that move results between systems without paying serialisation costs.
| Option | What distinguishes it |
|---|---|
| Apache Arrow | The in-memory columnar standard |
| Arrow Flight | High-throughput transport for Arrow data |
| Flight SQL | A SQL protocol over Flight |
How does data get in, and stay current?
How data arrives and keeps arriving: ingestion, transformation, orchestration, and streaming.
What does this data mean?
The layer that maps physical tables to business meaning, so queries are written against concepts rather than schemas.
| Option | What distinguishes it |
|---|---|
| Semantic layer | Maps physical tables to business concepts |
| Metrics definitions | One governed definition per measure |
| Ontologies and graphs | Relationships agents can traverse |
How do agents use any of this safely?
How language models and agents consume governed lakehouse data, and what they need from the layers beneath.
Where the model blurs
Clean layer diagrams are useful and slightly dishonest. These are the seams where real systems cross boundaries, and they are usually where migrations get expensive.
Catalogs that also store table metadata
The table format owns metadata and the catalog owns the pointer to it. Several catalogs blur this by caching or managing metadata directly, which improves performance and complicates migration.
Engines that ship their own catalog
Most engines can act as a catalog for their own tables. It is convenient for a single-engine deployment and becomes the thing you have to undo when a second engine arrives.
Table formats reaching into storage layout
Sort order, partitioning, and file sizing are declared at the table layer but decide how bytes land at the storage layer. Most real performance work happens across that seam.
Semantics implemented inside the engine
A semantic layer bound to one engine is the fastest to adopt and the hardest to keep when engines change. Whether semantics belong to the platform or to one query engine is the live architectural argument.