Table format
What makes those files behave like a table?
The metadata that turns a directory of files into a table with transactions, schema evolution, and time travel.
50 reference entries
ACID TransactionsACID is the foundational set of properties that define the correctness guarantees a data storage system must provide for transactions to be considered.Apache HudiApache Hudi (Hadoop Upserts Deletes and Incrementals) is the third major pillar of the Open Table Format ecosystem alongside Apache Iceberg and Delta Lake.Apache IcebergAs data lakes grew in popularity, organizations quickly realized that simply dumping raw Parquet or ORC files into Amazon S3 was not a viable long-term.Apache PaimonApache Paimon is the youngest and most architecturally distinctive of the major Open Table Formats.Apache XTable (OneTable)Apache XTable, originally released by Onehouse as "OneTable" and donated to the Apache Software Foundation, represents a fundamentally different category of.Branching (WAP)Software engineering relies heavily on version control systems like Git.Commit (Iceberg)In a traditional file-system-based data lake, "committing" data usually just meant finishing the upload of a Parquet file to a directory.CompactionA modern data lakehouse is often fed by continuous, real-time data streams (like Apache Kafka or Flink).Copy-on-Write (CoW)In a traditional relational database, if you execute an UPDATE statement to change a user's address, the database engine physically overwrites the specific.Delete FilesIn an immutable storage layer like Amazon S3, you cannot open a Parquet file, locate a specific row, and hit the "delete" key.Delta LakeDelta Lake is one of the three dominant Open Table Formats that define the modern data lakehouse.Delta UniFormDelta UniForm, short for Universal Format, is a feature introduced in Delta Lake 3.0 that allows a Delta Lake table to be simultaneously readable as an.Equality DeletesWhen dealing with high-velocity streaming data, such as a Change Data Capture (CDC) pipeline ingesting thousands of database updates per second, data.Expire SnapshotsA defining feature of Apache Iceberg is its ability to create a new Snapshot for every single transaction.Format ConversionFormat Conversion is the process of reading data stored in one physical file format and rewriting it in a different physical file format.Format InteroperabilityThe promise of the Open Data Lakehouse is simple to state and brutally difficult to execute: store data once, on cheap cloud object storage, and make it.Hidden PartitioningPartitioning is a core optimization strategy in massive data lakehouses.Manifest FileIn the Apache Iceberg metadata hierarchy, the Manifest File is the critical layer sitting directly above the raw data.Manifest ListIn the hierarchical metadata tree of Apache Iceberg, the Snapshot defines the state of the table, but it does not directly list the millions of data files.Merge-on-Read (MoR)While Copy-on-Write (CoW) provides blistering read performance, its massive Write Amplification makes it unsuitable for high-frequency updates, such as.Metadata LayerIf data files (like Parquet or ORC) are the muscle of a data lakehouse, and compute engines (like Spark or Dremio) are the brain, the Metadata Layer is the.Metadata LogApache Iceberg achieves ACID transactions on object storage without requiring a continuous compute engine.Metadata PointerIn an open data lakehouse architecture based on Apache Iceberg, a single table might consist of millions of Parquet data files, thousands of Avro manifest.Metadata TranslationMetadata Translation is the technical process of reading the metadata representation of a data table in one Open Table Format and generating an equivalent.Open Table FormatsA definitive, detailed look guide into Open Table Formats, exploring the architectural shift in approach that bridges the gap between data lakes and data warehouses, featuring an exhaustive analysis of Apache Iceberg, Delta Lake, and Apache Hudi.Optimistic Concurrency Control (OCC)When multiple systems attempt to write data to the same table at the exact same time, a database must have a mechanism to resolve the conflict.Partition EvolutionIn the lifecycle of a data lakehouse, data volume rarely remains static.Partition SpecPartitioning is a fundamental technique in data engineering.Position DeletesWithin the Merge-on-Read (MoR) architecture of Apache Iceberg, there are two methods for creating a logical tombstone to hide a record.Read AmplificationRead Amplification is the phenomenon where a query must read more data from storage than is logically required to satisfy the query's result, due to the.Remove Orphan FilesData lakehouse operations are inherently distributed and prone to environmental failures.Rewrite Data FilesIn Apache Iceberg, the abstract concept of "Compaction" is practically executed using a specific maintenance API called rewriteDataFiles.Rewrite ManifestsWhile rewriteDataFiles focuses on optimizing the physical Parquet data, it does not optimize the metadata layer.RollbackDespite the best data quality checks and Write-Audit-Publish patterns, human error inevitably occurs in data engineering.Schema EvolutionA comprehensive guide to Schema Evolution in Apache Iceberg, detailing how metadata-only operations provide safe, instantaneous updates to data structures.Schema SpecA comprehensive guide to the Schema Spec in Apache Iceberg, detailing how strict column ID tracking enables safe, instantaneous schema evolution without rewriting data.Sequence NumberIn a highly concurrent data lakehouse, determining the exact order in which events occurred is critical.SnapshotIn a traditional relational database, if you execute an UPDATE statement to change a user's address, the database physically overwrites the old address on.Snapshot IsolationWhen multiple systems interact with the same data simultaneously, chaos can easily ensue.Sort Order SpecWhile the Partition Spec determines how data is logically divided into coarse-grained directories (like by year or month), the Sort Order Spec determines how.Staged CommitsThe Write-Audit-Publish (WAP) pattern is the gold standard for maintaining data quality in a lakehouse.Strict MetricsWhen a query engine like Apache Spark or Trino runs a query against a data lakehouse, its primary goal during the planning phase is to read as little data as.Table FormatIf you look inside a traditional data lake, you will find a collection of directories and files.Table MaintenanceTable Maintenance is the collection of recurring operational procedures that keep Apache Iceberg tables performant, storage-efficient, and operationally.Table UUIDWhen a user interacts with a database, they use human-readable names. They write SELECT * FROM sales_data.Tagging (Iceberg)While Time Travel allows you to query historical data by providing a specific Snapshot ID or Timestamp, remembering a random 19-digit number like.Time TravelIn traditional relational databases, querying the past is extremely difficult.Transaction LogThe Transaction Log is one of the most fundamental data structures in computer science.Write AmplificationWrite Amplification is a phenomenon in data storage and processing systems where the actual amount of data written to storage is significantly larger than.Write-Audit-Publish (WAP)The "silent failure" is the most dangerous event in data engineering. A pipeline succeeds, no errors are thrown, and data is written to production.