# OpenDataLakehouse

> A vendor-neutral reference for open lakehouse architecture: what an open lakehouse is,
> how its layers fit together, and how to choose at each one.

Written and maintained by Alex Merced. Canonical author entity: https://alexmerced.com/#alexmerced

Apache Iceberg, Apache Polaris, Apache Parquet, and Apache Arrow are trademarks of the
Apache Software Foundation. This site is independent and not affiliated with the ASF;
project names describe subject matter only.

## Start here
- [What is an open lakehouse?](https://opendatalakehouse.com/what-is-an-open-lakehouse/): The canonical definition, the three claims inside the term, and what the architecture replaced
- [Principles](https://opendatalakehouse.com/principles/): Five commitments that make an architecture open, each with a test you can run against a system
- [Reference architecture](https://opendatalakehouse.com/architecture/): Every layer, the options at each, and where the clean model blurs in practice
- [History](https://opendatalakehouse.com/history/): How the category emerged, one failure and response at a time

## Reference formats
- [Reference library](https://opendatalakehouse.com/kb/): 201 in-depth entries grouped by layer
- [Glossary](https://opendatalakehouse.com/glossary/): Every term defined in one line on a single page
- [Comparisons](https://opendatalakehouse.com/compare/): Side-by-side tables for the choices at each layer
- [FAQ](https://opendatalakehouse.com/faq/): Direct answers to common questions

## Comparisons
- [Lakehouse vs warehouse vs lake](https://opendatalakehouse.com/compare/lakehouse-vs-warehouse-vs-lake/): How the three architectures differ across storage, transactions, engine choice, and exit cost
- [Iceberg vs Delta vs Hudi vs Paimon](https://opendatalakehouse.com/compare/table-formats/): Choosing an open table format
- [Comparing lakehouse catalogs](https://opendatalakehouse.com/compare/catalogs/): Polaris, Nessie, Glue, Unity, Hive Metastore, and Lakekeeper

## Reference index, by architecture layer

### Foundations (17 entries)
The architectural ideas the rest of the stack assumes: what a lakehouse is, and how it differs from a lake or a warehouse.
Layer index: https://opendatalakehouse.com/kb/layer/foundation/

- [Bronze Layer](https://opendatalakehouse.com/kb/bronze-layer/): In the Medallion Architecture that structures modern data lakehouses, data does not simply arrive and immediately become analytically ready.
- [Data Fabric](https://opendatalakehouse.com/kb/data-fabric/): Modern enterprises rarely store their data in a single place.
- [Data Gravity](https://opendatalakehouse.com/kb/data-gravity/): Data Gravity is an analogy coined by Dave McCrory in 2010 to describe the phenomenon by which large concentrations of data attract applications, services.
- [Data Lake](https://opendatalakehouse.com/kb/data-lake/): The explosion of digital information over the last decade created a massive storage problem.
- [Data Lakehouse](https://opendatalakehouse.com/kb/data-lakehouse/): A comprehensive definition of Data Lakehouse architecture, combining data warehouse reliability with data lake scalability via open table formats.
- [Data Mesh](https://opendatalakehouse.com/kb/data-mesh/): For most of the 2010s, the standard blueprint for a modern data platform involved building a centralized data lake, staffing a central data engineering te…
- [Data Swamp](https://opendatalakehouse.com/kb/data-swamp/): A comprehensive guide to what causes a Data Lake to become a Data Swamp, how to recognize the warning signs, and the governance practices that prevent it.
- [Data Warehouse](https://opendatalakehouse.com/kb/data-warehouse/): A data warehouse is a centralized repository engineered specifically to store highly structured, historical data.
- [Gold Layer](https://opendatalakehouse.com/kb/gold-layer/): At the top of the Medallion Architecture sits the Gold Layer: the final destination for data that has been ingested, cleaned, validated, enriched, and now.
- [Kappa Architecture](https://opendatalakehouse.com/kb/kappa-architecture/): Understanding Kappa Architecture, the simplified alternative to Lambda that treats everything as a stream.
- [Lambda Architecture](https://opendatalakehouse.com/kb/lambda-architecture/): Lambda Architecture is a data deployment model introduced by Nathan Marz designed to handle massive quantities of data by taking advantage of both batch a…
- [Medallion Architecture](https://opendatalakehouse.com/kb/medallion-architecture/): When organizations first started building data lakehouses, they faced a structuring problem.
- [Open Lakehouse](https://opendatalakehouse.com/kb/open-lakehouse/): An open lakehouse is a data architecture that stores data in open file and table formats on commodity object storage, so that any compliant engine can rea…
- [Polyglot Persistence](https://opendatalakehouse.com/kb/polyglot-persistence/): For a long time, the default answer to any data storage question in enterprise software development was a relational database.
- [Separation of Compute and Storage](https://opendatalakehouse.com/kb/separation-of-compute-and-storage/): Separation of Compute and Storage is an architectural principle in which the processing layer (compute engines that execute queries and transformations) a…
- [Silver Layer](https://opendatalakehouse.com/kb/silver-layer/): In the Medallion Architecture, data quality is not enforced at the point of ingestion.
- [Zero-ETL](https://opendatalakehouse.com/kb/zero-etl/): Data pipelines are expensive to build and expensive to maintain.

### Storage format (26 entries)
How bytes are laid out on object storage, and how they are encoded and compressed for analytical reads.
Layer index: https://opendatalakehouse.com/kb/layer/storage/

- [Amazon S3](https://opendatalakehouse.com/kb/amazon-s3/): Amazon Simple Storage Service (Amazon S3) is an object storage service provided by Amazon Web Services (AWS) that offers industry-leading scalability, dat…
- [Avro Format](https://opendatalakehouse.com/kb/avro-format/): While Apache Parquet dominates the field of analytical data storage, it is not the only file format found in a modern data lakehouse.
- [Azure Blob Storage](https://opendatalakehouse.com/kb/azure-blob-storage/): Azure Blob Storage is Microsoft's massively scalable object storage solution for the cloud.
- [Bloom Filters](https://opendatalakehouse.com/kb/bloom-filters/): A Bloom Filter is a space-efficient probabilistic data structure invented by Burton Howard Bloom in 1970.
- [Column-Level Statistics](https://opendatalakehouse.com/kb/column-level-statistics/): Column-Level Statistics are metadata measurements computed over the values in individual columns of data files, stored alongside the data to enable two.
- [Columnar Formats](https://opendatalakehouse.com/kb/columnar-formats/): Columnar formats are data storage layouts where data is physically organized and stored by column, rather than by row.
- [Data File](https://opendatalakehouse.com/kb/data-file/): If you peel back the complex layers of an open data lakehouse (past the query engines, past the Catalog, past the metadata tree of Manifest Lists and.
- [Dictionary Encoding](https://opendatalakehouse.com/kb/dictionary-encoding/): Dictionary Encoding is a highly effective data compression technique predominantly used in columnar storage formats like Apache Parquet and Apache ORC.
- [File Block Size](https://opendatalakehouse.com/kb/file-block-size/): File Block Size (also referred to as row group size or split size) is a critical physical configuration parameter in big data storage systems.
- [File Format](https://opendatalakehouse.com/kb/file-format/): In the architecture of a modern data lakehouse, there is a strict separation between the Table Format (like Apache Iceberg or Delta Lake) and the File For…
- [Google Cloud Storage (GCS)](https://opendatalakehouse.com/kb/google-cloud-storage/): Google Cloud Storage (GCS) is a managed, highly scalable object storage service provided by Google Cloud Platform (GCP).
- [GZIP Compression](https://opendatalakehouse.com/kb/gzip-compression/): GZIP (GNU zip) is one of the most widely used lossless data compression utilities in the history of computing.
- [LZ4 Compression](https://opendatalakehouse.com/kb/lz4-compression/): LZ4 is a lossless data compression algorithm focused on incredibly fast compression and decompression speeds.
- [Min-Max Statistics](https://opendatalakehouse.com/kb/min-max-statistics/): Min-Max Statistics are per-column metadata stored alongside data in Parquet files and in Iceberg Manifest Files, recording the minimum value, maximum valu…
- [MinIO](https://opendatalakehouse.com/kb/minio/): MinIO is an open-source, high-performance, distributed object storage server.
- [Object Storage](https://opendatalakehouse.com/kb/object-storage/): Object Storage is a data storage architecture designed to manage massive amounts of unstructured and structured data.
- [ORC Format](https://opendatalakehouse.com/kb/orc-format/): Apache ORC (Optimized Row Columnar) is the second major columnar file format found in modern data lakehouses.
- [Parquet Format](https://opendatalakehouse.com/kb/parquet-format/): Apache Parquet is an open-source, columnar file format designed specifically for fast data processing and massive storage efficiency in the Hadoop and dat…
- [Row-Oriented Formats](https://opendatalakehouse.com/kb/row-oriented-formats/): Row-oriented formats are data storage layouts where the data associated with a single record (a row) is stored contiguously on physical storage media.
- [Run-Length Encoding (RLE)](https://opendatalakehouse.com/kb/run-length-encoding/): Understanding Run-Length Encoding (RLE), a foundational compression algorithm for sorted columnar data.
- [S3 API Compatibility](https://opendatalakehouse.com/kb/s3-api-compatibility/): S3 API Compatibility refers to the industry-wide phenomenon where competing cloud providers, independent software vendors, and hardware manufacturers have.
- [Small File Problem](https://opendatalakehouse.com/kb/small-file-problem/): The Small File Problem refers to the performance and operational degradation that occurs when a data lake or lakehouse accumulates a very large number of.
- [Snappy Compression](https://opendatalakehouse.com/kb/snappy-compression/): Snappy is a fast, lossless data compression and decompression library written in C++ and originally developed by Google.
- [Storage Layer](https://opendatalakehouse.com/kb/storage-layer/): In the architecture of a modern data lakehouse, the Storage Layer is the foundational bedrock upon which everything else is built.
- [Target File Size](https://opendatalakehouse.com/kb/target-file-size/): Target File Size is the configured desired size (in bytes) for data files written to or produced by a data lakehouse table format like Apache Iceberg.
- [Zstandard (Zstd)](https://opendatalakehouse.com/kb/zstandard/): Zstandard, commonly abbreviated as Zstd, is a fast, lossless data compression algorithm developed by Yann Collet at Facebook (Meta).

### Table format (50 entries)
The metadata that turns a directory of files into a table with transactions, schema evolution, and time travel.
Layer index: https://opendatalakehouse.com/kb/layer/table/

- [ACID Transactions](https://opendatalakehouse.com/kb/acid-transactions/): ACID is the foundational set of properties that define the correctness guarantees a data storage system must provide for transactions to be considered.
- [Apache Hudi](https://opendatalakehouse.com/kb/apache-hudi/): Apache Hudi (Hadoop Upserts Deletes and Incrementals) is the third major pillar of the Open Table Format ecosystem alongside Apache Iceberg and Delta Lake.
- [Apache Iceberg](https://opendatalakehouse.com/kb/apache-iceberg/): As data lakes grew in popularity, organizations quickly realized that simply dumping raw Parquet or ORC files into Amazon S3 was not a viable long-term.
- [Apache Paimon](https://opendatalakehouse.com/kb/apache-paimon/): Apache Paimon is the youngest and most architecturally distinctive of the major Open Table Formats.
- [Apache XTable (OneTable)](https://opendatalakehouse.com/kb/apache-xtable/): Apache XTable, originally released by Onehouse as \"OneTable\" and donated to the Apache Software Foundation, represents a fundamentally different categor…
- [Branching (WAP)](https://opendatalakehouse.com/kb/branching-wap/): Software engineering relies heavily on version control systems like Git.
- [Commit (Iceberg)](https://opendatalakehouse.com/kb/commit-iceberg/): In a traditional file-system-based data lake, \"committing\" data usually just meant finishing the upload of a Parquet file to a directory.
- [Compaction](https://opendatalakehouse.com/kb/compaction/): A modern data lakehouse is often fed by continuous, real-time data streams (like Apache Kafka or Flink).
- [Copy-on-Write (CoW)](https://opendatalakehouse.com/kb/copy-on-write/): In a traditional relational database, if you execute an UPDATE statement to change a user's address, the database engine physically overwrites the specifi…
- [Delete Files](https://opendatalakehouse.com/kb/delete-files/): In an immutable storage layer like Amazon S3, you cannot open a Parquet file, locate a specific row, and hit the \"delete\" key.
- [Delta Lake](https://opendatalakehouse.com/kb/delta-lake/): Delta Lake is one of the three dominant Open Table Formats that define the modern data lakehouse.
- [Delta UniForm](https://opendatalakehouse.com/kb/delta-uniform/): Delta UniForm, short for Universal Format, is a feature introduced in Delta Lake 3.0 that allows a Delta Lake table to be simultaneously readable as an.
- [Equality Deletes](https://opendatalakehouse.com/kb/equality-deletes/): When dealing with high-velocity streaming data, such as a Change Data Capture (CDC) pipeline ingesting thousands of database updates per second, data.
- [Expire Snapshots](https://opendatalakehouse.com/kb/expire-snapshots/): A defining feature of Apache Iceberg is its ability to create a new Snapshot for every single transaction.
- [Format Conversion](https://opendatalakehouse.com/kb/format-conversion/): Format Conversion is the process of reading data stored in one physical file format and rewriting it in a different physical file format.
- [Format Interoperability](https://opendatalakehouse.com/kb/format-interoperability/): The promise of the Open Data Lakehouse is simple to state and brutally difficult to execute: store data once, on cheap cloud object storage, and make it.
- [Hidden Partitioning](https://opendatalakehouse.com/kb/hidden-partitioning/): Partitioning is a core optimization strategy in massive data lakehouses.
- [Manifest File](https://opendatalakehouse.com/kb/manifest-file/): In the Apache Iceberg metadata hierarchy, the Manifest File is the critical layer sitting directly above the raw data.
- [Manifest List](https://opendatalakehouse.com/kb/manifest-list/): In the hierarchical metadata tree of Apache Iceberg, the Snapshot defines the state of the table, but it does not directly list the millions of data files.
- [Merge-on-Read (MoR)](https://opendatalakehouse.com/kb/merge-on-read/): While Copy-on-Write (CoW) provides blistering read performance, its massive Write Amplification makes it unsuitable for high-frequency updates, such as.
- [Metadata Layer](https://opendatalakehouse.com/kb/metadata-layer/): If data files (like Parquet or ORC) are the muscle of a data lakehouse, and compute engines (like Spark or Dremio) are the brain, the Metadata Layer is th…
- [Metadata Log](https://opendatalakehouse.com/kb/metadata-log/): Apache Iceberg achieves ACID transactions on object storage without requiring a continuous compute engine.
- [Metadata Pointer](https://opendatalakehouse.com/kb/metadata-pointer/): In an open data lakehouse architecture based on Apache Iceberg, a single table might consist of millions of Parquet data files, thousands of Avro manifest.
- [Metadata Translation](https://opendatalakehouse.com/kb/metadata-translation/): Metadata Translation is the technical process of reading the metadata representation of a data table in one Open Table Format and generating an equivalent.
- [Open Table Formats](https://opendatalakehouse.com/kb/open-table-formats/): A definitive, detailed look guide into Open Table Formats, exploring the architectural shift in approach that bridges the gap between data lakes and data …
- [Optimistic Concurrency Control (OCC)](https://opendatalakehouse.com/kb/occ/): When multiple systems attempt to write data to the same table at the exact same time, a database must have a mechanism to resolve the conflict.
- [Partition Evolution](https://opendatalakehouse.com/kb/partition-evolution/): In the lifecycle of a data lakehouse, data volume rarely remains static.
- [Partition Spec](https://opendatalakehouse.com/kb/partition-spec/): Partitioning is a fundamental technique in data engineering.
- [Position Deletes](https://opendatalakehouse.com/kb/position-deletes/): Within the Merge-on-Read (MoR) architecture of Apache Iceberg, there are two methods for creating a logical tombstone to hide a record.
- [Read Amplification](https://opendatalakehouse.com/kb/read-amplification/): Read Amplification is the phenomenon where a query must read more data from storage than is logically required to satisfy the query's result, due to the.
- [Remove Orphan Files](https://opendatalakehouse.com/kb/remove-orphan-files/): Data lakehouse operations are inherently distributed and prone to environmental failures.
- [Rewrite Data Files](https://opendatalakehouse.com/kb/rewrite-data-files/): In Apache Iceberg, the abstract concept of \"Compaction\" is practically executed using a specific maintenance API called rewriteDataFiles.
- [Rewrite Manifests](https://opendatalakehouse.com/kb/rewrite-manifests/): While rewriteDataFiles focuses on optimizing the physical Parquet data, it does not optimize the metadata layer.
- [Rollback](https://opendatalakehouse.com/kb/rollback/): Despite the best data quality checks and Write-Audit-Publish patterns, human error inevitably occurs in data engineering.
- [Schema Evolution](https://opendatalakehouse.com/kb/schema-evolution/): A comprehensive guide to Schema Evolution in Apache Iceberg, detailing how metadata-only operations provide safe, instantaneous updates to data structures.
- [Schema Spec](https://opendatalakehouse.com/kb/schema-spec/): A comprehensive guide to the Schema Spec in Apache Iceberg, detailing how strict column ID tracking enables safe, instantaneous schema evolution without r…
- [Sequence Number](https://opendatalakehouse.com/kb/sequence-number/): In a highly concurrent data lakehouse, determining the exact order in which events occurred is critical.
- [Snapshot](https://opendatalakehouse.com/kb/snapshot/): In a traditional relational database, if you execute an UPDATE statement to change a user's address, the database physically overwrites the old address on.
- [Snapshot Isolation](https://opendatalakehouse.com/kb/snapshot-isolation/): When multiple systems interact with the same data simultaneously, chaos can easily ensue.
- [Sort Order Spec](https://opendatalakehouse.com/kb/sort-order-spec/): While the Partition Spec determines how data is logically divided into coarse-grained directories (like by year or month), the Sort Order Spec determines …
- [Staged Commits](https://opendatalakehouse.com/kb/staged-commits/): The Write-Audit-Publish (WAP) pattern is the gold standard for maintaining data quality in a lakehouse.
- [Strict Metrics](https://opendatalakehouse.com/kb/strict-metrics/): When a query engine like Apache Spark or Trino runs a query against a data lakehouse, its primary goal during the planning phase is to read as little data…
- [Table Format](https://opendatalakehouse.com/kb/table-format/): If you look inside a traditional data lake, you will find a collection of directories and files.
- [Table Maintenance](https://opendatalakehouse.com/kb/table-maintenance/): Table Maintenance is the collection of recurring operational procedures that keep Apache Iceberg tables performant, storage-efficient, and operationally.
- [Table UUID](https://opendatalakehouse.com/kb/table-uuid/): When a user interacts with a database, they use human-readable names. They write SELECT * FROM sales_data.
- [Tagging (Iceberg)](https://opendatalakehouse.com/kb/tagging-iceberg/): While Time Travel allows you to query historical data by providing a specific Snapshot ID or Timestamp, remembering a random 19-digit number like.
- [Time Travel](https://opendatalakehouse.com/kb/time-travel/): In traditional relational databases, querying the past is extremely difficult.
- [Transaction Log](https://opendatalakehouse.com/kb/transaction-log/): The Transaction Log is one of the most fundamental data structures in computer science.
- [Write Amplification](https://opendatalakehouse.com/kb/write-amplification/): Write Amplification is a phenomenon in data storage and processing systems where the actual amount of data written to storage is significantly larger than.
- [Write-Audit-Publish (WAP)](https://opendatalakehouse.com/kb/write-audit-publish/): The \"silent failure\" is the most dangerous event in data engineering. A pipeline succeeds, no errors are thrown, and data is written to production.

### Catalog (17 entries)
The service that tracks which tables exist, where their metadata lives, and who is allowed to touch them.
Layer index: https://opendatalakehouse.com/kb/layer/catalog/

- [Attribute-Based Access Control (ABAC)](https://opendatalakehouse.com/kb/attribute-based-access-control/): Attribute-Based Access Control is an access management model that makes authorization decisions based on the attributes of the subject (who is requesting.
- [AWS Glue Data Catalog](https://opendatalakehouse.com/kb/aws-glue-data-catalog/): The AWS Glue Data Catalog is Amazon Web Services' fully managed, serverless metadata catalog service: the central metadata registry for the entire AWS.
- [Catalog Migration](https://opendatalakehouse.com/kb/catalog-migration/): Catalog Migration is the process of transitioning Apache Iceberg tables from one catalog implementation to another, from Hive Metastore to a REST Catalog.
- [Credential Vending](https://opendatalakehouse.com/kb/credential-vending/): Credential Vending is the security mechanism in the Apache Iceberg REST Catalog specification that enables catalog services to dynamically generate.
- [Dremio Arctic](https://opendatalakehouse.com/kb/dremio-arctic/): A historical overview of Dremio Arctic, the Git-for-data catalog that evolved into Apache Polaris and the Nessie open-source project.
- [Dynamic Catalogs](https://opendatalakehouse.com/kb/dynamic-catalogs/): Dynamic Catalogs refers to the architectural pattern of configuring query engines and data platforms to connect with multiple Iceberg catalog instances.
- [Fine-Grained Access Control (FGAC)](https://opendatalakehouse.com/kb/fine-grained-access-control/): Fine-Grained Access Control refers to the collection of data access enforcement mechanisms that operate below the table level: controlling which rows.
- [Hadoop Catalog](https://opendatalakehouse.com/kb/hadoop-catalog/): The Hadoop Catalog (also referred to as the Filesystem Catalog or HadoopCatalog in the Iceberg codebase) is the simplest possible catalog implementation f…
- [Hive Metastore (HMS)](https://opendatalakehouse.com/kb/hive-metastore/): The Hive Metastore is the metadata management service at the heart of the Apache Hive data warehouse ecosystem, and by extension the foundational catalog.
- [Iceberg Catalog](https://opendatalakehouse.com/kb/iceberg-catalog/): The Iceberg Catalog is the central nervous system of any Apache Iceberg deployment.
- [JDBC Catalog](https://opendatalakehouse.com/kb/jdbc-catalog/): The JDBC Catalog is a catalog implementation for Apache Iceberg that uses any JDBC-compatible relational database as its persistent metadata backend.
- [Polaris Catalog](https://opendatalakehouse.com/kb/polaris-catalog/): Apache Polaris is the premier open-source implementation of the Apache Iceberg REST Catalog specification: a production-grade, vendor-neutral catalog serv…
- [Project Nessie](https://opendatalakehouse.com/kb/project-nessie/): Project Nessie is an open-source transactional catalog for Apache Iceberg that introduces Git-like version control semantics to data lakehouse metadata.
- [REST Catalog](https://opendatalakehouse.com/kb/rest-catalog/): The Iceberg REST Catalog specification, commonly called the REST Catalog or IRC, is the single most architecturally significant standard to emerge from th…
- [Role-Based Access Control (RBAC)](https://opendatalakehouse.com/kb/role-based-access-control/): Role-Based Access Control is the dominant access management model in enterprise data infrastructure: a system that grants permissions to named roles, and.
- [Tabular](https://opendatalakehouse.com/kb/tabular/): Tabular was a managed data lakehouse platform built entirely around Apache Iceberg, founded in 2021 by the original creators of the Apache Iceberg project.
- [Unity Catalog](https://opendatalakehouse.com/kb/unity-catalog/): Unity Catalog is Databricks' enterprise governance layer for the lakehouse: a centralized metadata, access control, and data discovery service that manage…

### Compute (43 entries)
The engines that plan and execute queries, and the optimisations that make them fast over object storage.
Layer index: https://opendatalakehouse.com/kb/layer/compute/

- [Amazon Athena](https://opendatalakehouse.com/kb/amazon-athena/): Amazon Athena is a serverless, interactive query service provided by Amazon Web Services (AWS) that allows users to analyze data directly in Amazon Simple.
- [Apache Doris](https://opendatalakehouse.com/kb/apache-doris/): Apache Doris is a modern, open-source Massively Parallel Processing (MPP) analytical database designed for blazing-fast, real-time data warehousing.
- [Apache Flink](https://opendatalakehouse.com/kb/apache-flink/): Apache Flink is an open-source, unified stream-processing and batch-processing framework.
- [Apache Spark](https://opendatalakehouse.com/kb/apache-spark/): Apache Spark is a unified analytics engine designed for large-scale data processing.
- [Broadcast Join](https://opendatalakehouse.com/kb/broadcast-join/): A Broadcast Join is a distributed join algorithm where the smaller of the two join inputs is completely replicated (\"broadcast\") to every worker node in…
- [Caching](https://opendatalakehouse.com/kb/caching/): Caching in data systems is the practice of storing copies of frequently accessed data in a faster, closer storage layer so that subsequent requests for th…
- [ClickHouse](https://opendatalakehouse.com/kb/clickhouse/): ClickHouse is an open-source, column-oriented database management system (DBMS) built expressly for online analytical processing (OLAP).
- [Compute Engine](https://opendatalakehouse.com/kb/compute-engine/): In traditional data warehousing, storage and compute were inextricably linked.
- [Cost-Based Optimizer (CBO)](https://opendatalakehouse.com/kb/cost-based-optimizer/): A Cost-Based Optimizer (CBO) is the intelligent \"brain\" within a modern relational database or distributed compute engine.
- [Data Skew](https://opendatalakehouse.com/kb/data-skew/): Data Skew in distributed query execution is the unequal distribution of data or computational work across the worker nodes of a cluster, where some worker…
- [Data Skipping](https://opendatalakehouse.com/kb/data-skipping/): Data Skipping is a query optimization technique in which a query engine uses pre-computed metadata statistics about data files to determine, at planning.
- [Databricks](https://opendatalakehouse.com/kb/databricks/): An extensive overview of Databricks, the unified data analytics platform that pioneered the data lakehouse paradigm and developed Delta Lake and Apache Sp…
- [Deserialization](https://opendatalakehouse.com/kb/deserialization/): An in-depth look at deserialization and its performance impacts on analytical query engines.
- [Distributed Compute](https://opendatalakehouse.com/kb/distributed-compute/): A foundational overview of distributed compute architectures in data processing, explaining master-worker topologies, data shuffling, and fault tolerance.
- [Dremio](https://opendatalakehouse.com/kb/dremio/): Dremio is an open lakehouse platform purpose-built for Apache Iceberg, providing a high-performance query engine, a semantic layer, an integrated Apache.
- [DuckDB](https://opendatalakehouse.com/kb/duckdb/): DuckDB is an open-source, in-process SQL OLAP database management system.
- [File Skipping](https://opendatalakehouse.com/kb/file-skipping/): File Skipping is the collective term for the multi-level system of techniques that allow a data lakehouse query engine to identify and eliminate data file…
- [Google BigQuery](https://opendatalakehouse.com/kb/google-bigquery/): Google BigQuery is a fully managed, serverless, enterprise data warehouse that enables scalable analysis over petabytes of data.
- [Hash Join](https://opendatalakehouse.com/kb/hash-join/): A Hash Join is a join algorithm that uses a hash table to match rows from two input relations based on their join key values.
- [Hilbert Curves](https://opendatalakehouse.com/kb/hilbert-curves/): The Hilbert Curve is a mathematically elegant space-filling curve first described by mathematician David Hilbert in 1891.
- [Indexing (Data Lakes)](https://opendatalakehouse.com/kb/indexing-data-lakes/): Indexing in the context of data lakes and lakehouses refers to the set of auxiliary data structures that enable query engines to quickly locate relevant d…
- [Join Strategies](https://opendatalakehouse.com/kb/join-strategies/): A Join Strategy is the specific algorithm a query engine uses to physically implement a SQL JOIN operation: matching rows from two (or more) tables based …
- [Materialized Views](https://opendatalakehouse.com/kb/materialized-views/): A Materialized View is a pre-computed, physically stored result of a SQL query whose output is saved to storage and can be queried directly, rather than.
- [MPP (Massively Parallel Processing)](https://opendatalakehouse.com/kb/mpp/): Massively Parallel Processing (MPP) is an architectural design paradigm for distributed computing and databases.
- [Out-of-Memory (OOM) Errors](https://opendatalakehouse.com/kb/out-of-memory-errors/): An Out-of-Memory (OOM) error occurs when a query execution process attempts to allocate more memory than is available to it, either on a single node.
- [Partition Pruning](https://opendatalakehouse.com/kb/partition-pruning/): Partition Pruning is the most powerful form of data skipping available to query engines in the data lakehouse.
- [Predicate Pushdown](https://opendatalakehouse.com/kb/predicate-pushdown/): Predicate Pushdown is a query optimization technique in which filter conditions (predicates) from the WHERE clause of a SQL query are applied as early as.
- [Presto](https://opendatalakehouse.com/kb/presto/): Presto (often referred to as PrestoDB to distinguish it from its fork, Trino) is an open-source, distributed SQL query engine designed for running.
- [Projection Pushdown](https://opendatalakehouse.com/kb/projection-pushdown/): Projection Pushdown (also called column pruning) is a query optimization technique in which only the specific columns required by a SQL query are read fro…
- [Pushdown Optimization](https://opendatalakehouse.com/kb/pushdown-optimization/): Pushdown Optimization (specifically Predicate Pushdown and Projection Pushdown) is one of the most critical performance techniques in distributed data.
- [Query Execution](https://opendatalakehouse.com/kb/query-execution/): Query Execution is the runtime phase in which a database or query engine carries out the physical execution plan produced by the query planner, reading da…
- [Query Planning](https://opendatalakehouse.com/kb/query-planning/): Query Planning is the process by which a database or query engine transforms a declarative SQL query into a detailed, optimized execution plan: a concrete.
- [Rule-Based Optimizer (RBO)](https://opendatalakehouse.com/kb/rule-based-optimizer/): A Rule-Based Optimizer (RBO) is a critical component of a database's query planning phase.
- [Serialization](https://opendatalakehouse.com/kb/serialization/): Serialization is the process of translating data structures or object state into a format that can be stored (for example, in a file or memory buffer) or.
- [Shuffle](https://opendatalakehouse.com/kb/shuffle/): In distributed query execution, a Shuffle (also called an Exchange or Repartition) is the operation of redistributing data rows across the worker nodes of…
- [Snowflake](https://opendatalakehouse.com/kb/snowflake/): Snowflake is a fully managed cloud data platform that fundamentally reshaped the data warehousing industry.
- [Sort-Merge Join](https://opendatalakehouse.com/kb/sort-merge-join/): A Sort-Merge Join (SMJ) is a join algorithm that sorts both input relations by their join key and then performs a single linear merge pass through both.
- [Spilling to Disk](https://opendatalakehouse.com/kb/spilling-to-disk/): Spilling to Disk is a query engine mechanism that writes intermediate query results (hash tables, sort buffers, shuffle data) to local disk storage when t…
- [SQL Dialects](https://opendatalakehouse.com/kb/sql-dialects/): SQL (Structured Query Language) is the lingua franca of data analytics.
- [StarRocks](https://opendatalakehouse.com/kb/starrocks/): StarRocks is an open-source, next-generation Massive Parallel Processing (MPP) database designed for blazing-fast, real-time analytics.
- [Trino](https://opendatalakehouse.com/kb/trino/): Trino (formerly known as PrestoSQL) is a highly parallel and distributed open-source SQL query engine.
- [Vectorized Execution](https://opendatalakehouse.com/kb/vectorized-execution/): A detailed explanation of vectorized execution, the hardware-optimized processing model that allows modern compute engines to achieve blistering speeds.
- [Z-Ordering](https://opendatalakehouse.com/kb/z-ordering/): Z-Ordering is a multi-dimensional data clustering technique used in data lakehouse environments to physically co-locate records with similar values across.

### Interchange (3 entries)
The in-memory format and wire protocols that move results between systems without paying serialisation costs.
Layer index: https://opendatalakehouse.com/kb/layer/interchange/

- [Apache Arrow](https://opendatalakehouse.com/kb/apache-arrow/): Apache Arrow is a cross-language development platform for in-memory data.
- [Arrow Flight](https://opendatalakehouse.com/kb/arrow-flight/): Arrow Flight is an open-source, high-performance Remote Procedure Call (RPC) framework developed as part of the Apache Arrow project.
- [Arrow Flight SQL](https://opendatalakehouse.com/kb/arrow-flight-sql/): Arrow Flight SQL is a protocol extension built on top of the Apache Arrow Flight framework.

### Movement (15 entries)
How data arrives and keeps arriving: ingestion, transformation, orchestration, and streaming.
Layer index: https://opendatalakehouse.com/kb/layer/pipeline/

- [Apache Airflow](https://opendatalakehouse.com/kb/apache-airflow/): Apache Airflow is an open-source platform created by Airbnb in 2014 and later donated to the Apache Software Foundation.
- [Batch Processing](https://opendatalakehouse.com/kb/batch-processing/): Batch Processing is the execution of a series of jobs in a computer program without manual intervention (non-interactive).
- [Change Data Capture (CDC)](https://opendatalakehouse.com/kb/change-data-capture/): Change Data Capture (CDC) is a set of software design patterns and technologies used to determine and track the data that has changed within a source.
- [Dagster](https://opendatalakehouse.com/kb/dagster/): Dagster is an open-source data orchestration platform designed to address some of the architectural limitations of Apache Airflow.
- [Data Pipeline](https://opendatalakehouse.com/kb/data-pipeline/): A Data Pipeline is an automated set of processes and infrastructure that extracts data from various source systems, transforms it into a clean and usable.
- [dbt (data build tool)](https://opendatalakehouse.com/kb/dbt/): Understanding dbt, the transformative framework that brought software engineering best practices to SQL-based data transformations.
- [Directed Acyclic Graph (DAG)](https://opendatalakehouse.com/kb/directed-acyclic-graph/): A Directed Acyclic Graph (DAG) is a conceptual mathematical model heavily utilized in computer science, specifically within the area of data engineering a…
- [ELT (Extract, Load, Transform)](https://opendatalakehouse.com/kb/elt/): For decades, the dominant pattern for moving data from source systems into analytical databases was ETL: Extract, Transform, Load.
- [ETL (Extract, Transform, Load)](https://opendatalakehouse.com/kb/etl/): Before a business analyst can run a query against a data warehouse, someone has to get the data there.
- [Eventual Consistency](https://opendatalakehouse.com/kb/eventual-consistency/): In any discussion of data reliability in distributed systems, the conversation inevitably polarizes around two competing consistency models: ACID (Atomici…
- [Micro-batching](https://opendatalakehouse.com/kb/micro-batching/): Exploring Micro-batching, the architectural compromise that simulates streaming using rapid, tiny batch jobs.
- [Orchestration](https://opendatalakehouse.com/kb/orchestration/): In data engineering, Orchestration is the automated configuration, coordination, and management of complex computer systems, software, and services.
- [Prefect](https://opendatalakehouse.com/kb/prefect/): Exploring Prefect, the dynamic, Python-native workflow orchestration framework.
- [Streaming Data](https://opendatalakehouse.com/kb/streaming-data/): Streaming Data refers to data that is continuously generated by thousands of data sources, which typically send in the data records simultaneously, and in.
- [Strong Consistency](https://opendatalakehouse.com/kb/strong-consistency/): Strong Consistency is the most demanding correctness guarantee a distributed storage system can provide.

### Semantics (12 entries)
The layer that maps physical tables to business meaning, so queries are written against concepts rather than schemas.
Layer index: https://opendatalakehouse.com/kb/layer/semantic/

- [Data Lineage](https://opendatalakehouse.com/kb/data-lineage/): Data Lineage is the documented record of how data moves, transforms, and evolves through an analytical system, from its origination at source systems thro…
- [Data Modeling](https://opendatalakehouse.com/kb/data-modeling/): Data Modeling is the process of creating a visual and logical representation of either a whole information system or parts of it to communicate connection…
- [Data Quality](https://opendatalakehouse.com/kb/data-quality/): Data Quality is the set of properties, practices, and enforcement mechanisms that ensure data assets in a lakehouse are fit for the purposes for which the…
- [Dimension Table](https://opendatalakehouse.com/kb/dimension-table/): Understanding Dimension Tables, the descriptive context that gives meaning to analytical data.
- [Dimensional Modeling](https://opendatalakehouse.com/kb/dimensional-modeling/): Dimensional Modeling is a specialized data design methodology primarily utilized for data warehouses, data marts, and modern data lakehouses.
- [Fact Table](https://opendatalakehouse.com/kb/fact-table/): In dimensional modeling and data warehousing (specifically within a Star Schema or Snowflake Schema), a Fact Table is the central table that stores the.
- [Knowledge Graphs](https://opendatalakehouse.com/kb/knowledge-graphs/): A Knowledge Graph is a structured representation of knowledge as a network of entities and the relationships between them.
- [Ontology](https://opendatalakehouse.com/kb/ontology/): An ontology is a formal, explicit specification of a shared conceptualization: a structured vocabulary that defines the types of entities that exist in a.
- [Semantic Layer](https://opendatalakehouse.com/kb/semantic-layer/): One of the most persistent and costly problems in enterprise analytics is metric inconsistency.
- [Slowly Changing Dimensions (SCD)](https://opendatalakehouse.com/kb/slowly-changing-dimensions/): Slowly Changing Dimensions (SCD) is a fundamental concept in data warehousing that deals with a critical problem: How do you handle dimensional data that.
- [Snowflake Schema](https://opendatalakehouse.com/kb/snowflake-schema/): The Snowflake Schema is a logical arrangement of tables in a multidimensional database that is an extension and variation of the Star Schema.
- [Star Schema](https://opendatalakehouse.com/kb/star-schema/): Understanding the Star Schema, the fundamental dimensional modeling technique optimized for analytical query performance.

### Agents & AI (18 entries)
How language models and agents consume governed lakehouse data, and what they need from the layers beneath.
Layer index: https://opendatalakehouse.com/kb/layer/ai/

- [Agentic Analytics](https://opendatalakehouse.com/kb/agentic-analytics/): Agentic Analytics represents the next frontier in business intelligence and data engineering.
- [Agentic Workflows](https://opendatalakehouse.com/kb/agentic-workflows/): An Agentic Workflow is a structured sequence of AI-driven steps in which one or more autonomous agents coordinate to accomplish a complex, multi-stage.
- [AI Agents](https://opendatalakehouse.com/kb/ai-agents/): An AI Agent is an autonomous software system that uses a Large Language Model (LLM) as its core reasoning engine to perceive its environment, form plans.
- [Autonomous Analytics](https://opendatalakehouse.com/kb/autonomous-analytics/): Autonomous Analytics is the discipline and set of technologies that enable AI systems to independently discover, retrieve, analyze, and interpret enterpri…
- [Context Window](https://opendatalakehouse.com/kb/context-window/): The context window of a Large Language Model is the total amount of text, measured in tokens, that the model can process and reason about simultaneously.
- [Hallucination Mitigation](https://opendatalakehouse.com/kb/hallucination-mitigation/): Hallucination in Large Language Models refers to the generation of content that is factually incorrect, unverifiable, or entirely fabricated, but presente…
- [Large Language Models (LLMs)](https://opendatalakehouse.com/kb/large-language-models/): A Large Language Model (LLM) is an artificial intelligence model trained on massive quantities of text data to understand, generate, and reason with human.
- [Model Fine-Tuning](https://opendatalakehouse.com/kb/model-fine-tuning/): Model fine-tuning is the process of taking a pre-trained Large Language Model and continuing its training on a curated, domain-specific dataset to adapt i…
- [Multi-Agent Systems](https://opendatalakehouse.com/kb/multi-agent-systems/): A Multi-Agent System (MAS) is an architecture in which multiple independent AI agents collaborate to accomplish complex tasks that exceed what any single.
- [Observability (AI Systems)](https://opendatalakehouse.com/kb/observability-ai-systems/): AI System Observability is the practice of instrumenting, collecting, and analyzing telemetry data from Large Language Model applications and agentic syst…
- [Prompt Engineering](https://opendatalakehouse.com/kb/prompt-engineering/): Prompt Engineering is the discipline of designing, structuring, and optimizing the text inputs (prompts) provided to a Large Language Model to elicit.
- [Retrieval-Augmented Generation (RAG)](https://opendatalakehouse.com/kb/retrieval-augmented-generation/): Retrieval-Augmented Generation (RAG) is an AI architecture pattern that enhances a Large Language Model's responses by dynamically retrieving relevant.
- [Semantic Search](https://opendatalakehouse.com/kb/semantic-search/): Semantic Search is a search methodology that understands the intent and contextual meaning behind a query rather than performing a literal word-for-word.
- [Text Embeddings](https://opendatalakehouse.com/kb/text-embeddings/): A text embedding is a numerical representation of a piece of text (a word, sentence, paragraph, or entire document) expressed as a dense vector of.
- [Text-to-SQL](https://opendatalakehouse.com/kb/text-to-sql/): Text-to-SQL is the task of automatically translating a natural language question or instruction into a valid SQL query that, when executed against the tar…
- [Tool Use (Function Calling)](https://opendatalakehouse.com/kb/tool-use/): Tool Use, also called Function Calling, is the capability of modern Large Language Models to generate structured requests to execute predefined external.
- [Vector Databases](https://opendatalakehouse.com/kb/vector-databases/): A Vector Database is a specialized database management system engineered to store, index, and efficiently query high-dimensional vector embeddings at scal…
- [Vector Search](https://opendatalakehouse.com/kb/vector-search/): Vector Search (also called semantic search or similarity search) is a retrieval technique that finds results based on the conceptual meaning and semantic.

## Related sites in the same network
- https://datalakehouse.help: Task-oriented lakehouse documentation
- https://semanticlakehouse.com: The semantic layer in depth
- https://agenticlakehouse.com: AI agents on governed lakehouse data
- https://dataengnr.com: General data engineering reference
- https://alexmerced.com: Author profile and machine-readable entity data