Storage format
How is the data physically written down?
How bytes are laid out on object storage, and how they are encoded and compressed for analytical reads.
26 reference entries
Amazon S3Amazon Simple Storage Service (Amazon S3) is an object storage service provided by Amazon Web Services (AWS) that offers industry-leading scalability, data.Avro FormatWhile Apache Parquet dominates the field of analytical data storage, it is not the only file format found in a modern data lakehouse.Azure Blob StorageAzure Blob Storage is Microsoft's massively scalable object storage solution for the cloud.Bloom FiltersA Bloom Filter is a space-efficient probabilistic data structure invented by Burton Howard Bloom in 1970.Column-Level StatisticsColumn-Level Statistics are metadata measurements computed over the values in individual columns of data files, stored alongside the data to enable two.Columnar FormatsColumnar formats are data storage layouts where data is physically organized and stored by column, rather than by row.Data FileIf you peel back the complex layers of an open data lakehouse (past the query engines, past the Catalog, past the metadata tree of Manifest Lists and.Dictionary EncodingDictionary Encoding is a highly effective data compression technique predominantly used in columnar storage formats like Apache Parquet and Apache ORC.File Block SizeFile Block Size (also referred to as row group size or split size) is a critical physical configuration parameter in big data storage systems.File FormatIn the architecture of a modern data lakehouse, there is a strict separation between the Table Format (like Apache Iceberg or Delta Lake) and the File Format.Google Cloud Storage (GCS)Google Cloud Storage (GCS) is a managed, highly scalable object storage service provided by Google Cloud Platform (GCP).GZIP CompressionGZIP (GNU zip) is one of the most widely used lossless data compression utilities in the history of computing.LZ4 CompressionLZ4 is a lossless data compression algorithm focused on incredibly fast compression and decompression speeds.Min-Max StatisticsMin-Max Statistics are per-column metadata stored alongside data in Parquet files and in Iceberg Manifest Files, recording the minimum value, maximum value.MinIOMinIO is an open-source, high-performance, distributed object storage server.Object StorageObject Storage is a data storage architecture designed to manage massive amounts of unstructured and structured data.ORC FormatApache ORC (Optimized Row Columnar) is the second major columnar file format found in modern data lakehouses.Parquet FormatApache Parquet is an open-source, columnar file format designed specifically for fast data processing and massive storage efficiency in the Hadoop and data.Row-Oriented FormatsRow-oriented formats are data storage layouts where the data associated with a single record (a row) is stored contiguously on physical storage media.Run-Length Encoding (RLE)Understanding Run-Length Encoding (RLE), a foundational compression algorithm for sorted columnar data.S3 API CompatibilityS3 API Compatibility refers to the industry-wide phenomenon where competing cloud providers, independent software vendors, and hardware manufacturers have.Small File ProblemThe Small File Problem refers to the performance and operational degradation that occurs when a data lake or lakehouse accumulates a very large number of.Snappy CompressionSnappy is a fast, lossless data compression and decompression library written in C++ and originally developed by Google.Storage LayerIn the architecture of a modern data lakehouse, the Storage Layer is the foundational bedrock upon which everything else is built.Target File SizeTarget File Size is the configured desired size (in bytes) for data files written to or produced by a data lakehouse table format like Apache Iceberg.Zstandard (Zstd)Zstandard, commonly abbreviated as Zstd, is a fast, lossless data compression algorithm developed by Yann Collet at Facebook (Meta).