Table format
What makes those files behave like a table?
The metadata that turns a directory of files into a table with transactions, schema evolution, and time travel.
50 reference entries
ACID Transactions ACID is the foundational set of properties that define the correctness guarantees a data storage system must provide for transactions to be considered. Apache Hudi Apache Hudi (Hadoop Upserts Deletes and Incrementals) is the third major pillar of the Open Table Format ecosystem alongside Apache Iceberg and Delta Lake. Apache Iceberg As data lakes grew in popularity, organizations quickly realized that simply dumping raw Parquet or ORC files into Amazon S3 was not a viable long-term. Apache Paimon Apache Paimon is the youngest and most architecturally distinctive of the major Open Table Formats. Apache XTable (OneTable) Apache XTable, originally released by Onehouse as "OneTable" and donated to the Apache Software Foundation, represents a fundamentally different category of. Branching (WAP) Software engineering relies heavily on version control systems like Git. Commit (Iceberg) In a traditional file-system-based data lake, "committing" data usually just meant finishing the upload of a Parquet file to a directory. Compaction A modern data lakehouse is often fed by continuous, real-time data streams (like Apache Kafka or Flink). Copy-on-Write (CoW) In a traditional relational database, if you execute an UPDATE statement to change a user's address, the database engine physically overwrites the specific. Delete Files In an immutable storage layer like Amazon S3, you cannot open a Parquet file, locate a specific row, and hit the "delete" key. Delta Lake Delta Lake is one of the three dominant Open Table Formats that define the modern data lakehouse. Delta UniForm Delta UniForm, short for Universal Format, is a feature introduced in Delta Lake 3.0 that allows a Delta Lake table to be simultaneously readable as an. Equality Deletes When dealing with high-velocity streaming data, such as a Change Data Capture (CDC) pipeline ingesting thousands of database updates per second, data. Expire Snapshots A defining feature of Apache Iceberg is its ability to create a new Snapshot for every single transaction. Format Conversion Format Conversion is the process of reading data stored in one physical file format and rewriting it in a different physical file format. Format Interoperability The promise of the Open Data Lakehouse is simple to state and brutally difficult to execute: store data once, on cheap cloud object storage, and make it. Hidden Partitioning Partitioning is a core optimization strategy in massive data lakehouses. Manifest File In the Apache Iceberg metadata hierarchy, the Manifest File is the critical layer sitting directly above the raw data. Manifest List In the hierarchical metadata tree of Apache Iceberg, the Snapshot defines the state of the table, but it does not directly list the millions of data files. Merge-on-Read (MoR) While Copy-on-Write (CoW) provides blistering read performance, its massive Write Amplification makes it unsuitable for high-frequency updates, such as. Metadata Layer If data files (like Parquet or ORC) are the muscle of a data lakehouse, and compute engines (like Spark or Dremio) are the brain, the Metadata Layer is the. Metadata Log Apache Iceberg achieves ACID transactions on object storage without requiring a continuous compute engine. Metadata Pointer In an open data lakehouse architecture based on Apache Iceberg, a single table might consist of millions of Parquet data files, thousands of Avro manifest. Metadata Translation Metadata Translation is the technical process of reading the metadata representation of a data table in one Open Table Format and generating an equivalent. Open Table Formats A definitive, detailed look guide into Open Table Formats, exploring the architectural shift in approach that bridges the gap between data lakes and data warehouses, featuring an exhaustive analysis of Apache Iceberg, Delta Lake, and Apache Hudi. Optimistic Concurrency Control (OCC) When multiple systems attempt to write data to the same table at the exact same time, a database must have a mechanism to resolve the conflict. Partition Evolution In the lifecycle of a data lakehouse, data volume rarely remains static. Partition Spec Partitioning is a fundamental technique in data engineering. Position Deletes Within the Merge-on-Read (MoR) architecture of Apache Iceberg, there are two methods for creating a logical tombstone to hide a record. Read Amplification Read Amplification is the phenomenon where a query must read more data from storage than is logically required to satisfy the query's result, due to the. Remove Orphan Files Data lakehouse operations are inherently distributed and prone to environmental failures. Rewrite Data Files In Apache Iceberg, the abstract concept of "Compaction" is practically executed using a specific maintenance API called rewriteDataFiles. Rewrite Manifests While rewriteDataFiles focuses on optimizing the physical Parquet data, it does not optimize the metadata layer. Rollback Despite the best data quality checks and Write-Audit-Publish patterns, human error inevitably occurs in data engineering. Schema Evolution A comprehensive guide to Schema Evolution in Apache Iceberg, detailing how metadata-only operations provide safe, instantaneous updates to data structures. Schema Spec A comprehensive guide to the Schema Spec in Apache Iceberg, detailing how strict column ID tracking enables safe, instantaneous schema evolution without rewriting data. Sequence Number In a highly concurrent data lakehouse, determining the exact order in which events occurred is critical. Snapshot In a traditional relational database, if you execute an UPDATE statement to change a user's address, the database physically overwrites the old address on. Snapshot Isolation When multiple systems interact with the same data simultaneously, chaos can easily ensue. Sort Order Spec While the Partition Spec determines how data is logically divided into coarse-grained directories (like by year or month), the Sort Order Spec determines how. Staged Commits The Write-Audit-Publish (WAP) pattern is the gold standard for maintaining data quality in a lakehouse. Strict Metrics When a query engine like Apache Spark or Trino runs a query against a data lakehouse, its primary goal during the planning phase is to read as little data as. Table Format If you look inside a traditional data lake, you will find a collection of directories and files. Table Maintenance Table Maintenance is the collection of recurring operational procedures that keep Apache Iceberg tables performant, storage-efficient, and operationally. Table UUID When a user interacts with a database, they use human-readable names. They write SELECT * FROM sales_data. Tagging (Iceberg) While Time Travel allows you to query historical data by providing a specific Snapshot ID or Timestamp, remembering a random 19-digit number like. Time Travel In traditional relational databases, querying the past is extremely difficult. Transaction Log The Transaction Log is one of the most fundamental data structures in computer science. Write Amplification Write Amplification is a phenomenon in data storage and processing systems where the actual amount of data written to storage is significantly larger than. Write-Audit-Publish (WAP) The "silent failure" is the most dangerous event in data engineering. A pipeline succeeds, no errors are thrown, and data is written to production.