Storage format
How is the data physically written down?
How bytes are laid out on object storage, and how they are encoded and compressed for analytical reads.
26 reference entries
Amazon S3 Amazon Simple Storage Service (Amazon S3) is an object storage service provided by Amazon Web Services (AWS) that offers industry-leading scalability, data. Avro Format While Apache Parquet dominates the field of analytical data storage, it is not the only file format found in a modern data lakehouse. Azure Blob Storage Azure Blob Storage is Microsoft's massively scalable object storage solution for the cloud. Bloom Filters A Bloom Filter is a space-efficient probabilistic data structure invented by Burton Howard Bloom in 1970. Column-Level Statistics Column-Level Statistics are metadata measurements computed over the values in individual columns of data files, stored alongside the data to enable two. Columnar Formats Columnar formats are data storage layouts where data is physically organized and stored by column, rather than by row. Data File If you peel back the complex layers of an open data lakehouse (past the query engines, past the Catalog, past the metadata tree of Manifest Lists and. Dictionary Encoding Dictionary Encoding is a highly effective data compression technique predominantly used in columnar storage formats like Apache Parquet and Apache ORC. File Block Size File Block Size (also referred to as row group size or split size) is a critical physical configuration parameter in big data storage systems. File Format In the architecture of a modern data lakehouse, there is a strict separation between the Table Format (like Apache Iceberg or Delta Lake) and the File Format. Google Cloud Storage (GCS) Google Cloud Storage (GCS) is a managed, highly scalable object storage service provided by Google Cloud Platform (GCP). GZIP Compression GZIP (GNU zip) is one of the most widely used lossless data compression utilities in the history of computing. LZ4 Compression LZ4 is a lossless data compression algorithm focused on incredibly fast compression and decompression speeds. Min-Max Statistics Min-Max Statistics are per-column metadata stored alongside data in Parquet files and in Iceberg Manifest Files, recording the minimum value, maximum value. MinIO MinIO is an open-source, high-performance, distributed object storage server. Object Storage Object Storage is a data storage architecture designed to manage massive amounts of unstructured and structured data. ORC Format Apache ORC (Optimized Row Columnar) is the second major columnar file format found in modern data lakehouses. Parquet Format Apache Parquet is an open-source, columnar file format designed specifically for fast data processing and massive storage efficiency in the Hadoop and data. Row-Oriented Formats Row-oriented formats are data storage layouts where the data associated with a single record (a row) is stored contiguously on physical storage media. Run-Length Encoding (RLE) Understanding Run-Length Encoding (RLE), a foundational compression algorithm for sorted columnar data. S3 API Compatibility S3 API Compatibility refers to the industry-wide phenomenon where competing cloud providers, independent software vendors, and hardware manufacturers have. Small File Problem The Small File Problem refers to the performance and operational degradation that occurs when a data lake or lakehouse accumulates a very large number of. Snappy Compression Snappy is a fast, lossless data compression and decompression library written in C++ and originally developed by Google. Storage Layer In the architecture of a modern data lakehouse, the Storage Layer is the foundational bedrock upon which everything else is built. Target File Size Target File Size is the configured desired size (in bytes) for data files written to or produced by a data lakehouse table format like Apache Iceberg. Zstandard (Zstd) Zstandard, commonly abbreviated as Zstd, is a fast, lossless data compression algorithm developed by Yann Collet at Facebook (Meta).