Compute
What actually runs the query?
The engines that plan and execute queries, and the optimisations that make them fast over object storage.
43 reference entries
Amazon Athena Amazon Athena is a serverless, interactive query service provided by Amazon Web Services (AWS) that allows users to analyze data directly in Amazon Simple. Apache Doris Apache Doris is a modern, open-source Massively Parallel Processing (MPP) analytical database designed for blazing-fast, real-time data warehousing. Apache Flink Apache Flink is an open-source, unified stream-processing and batch-processing framework. Apache Spark Apache Spark is a unified analytics engine designed for large-scale data processing. Broadcast Join A Broadcast Join is a distributed join algorithm where the smaller of the two join inputs is completely replicated ("broadcast") to every worker node in the. Caching Caching in data systems is the practice of storing copies of frequently accessed data in a faster, closer storage layer so that subsequent requests for the. ClickHouse ClickHouse is an open-source, column-oriented database management system (DBMS) built expressly for online analytical processing (OLAP). Compute Engine In traditional data warehousing, storage and compute were inextricably linked. Cost-Based Optimizer (CBO) A Cost-Based Optimizer (CBO) is the intelligent "brain" within a modern relational database or distributed compute engine. Data Skew Data Skew in distributed query execution is the unequal distribution of data or computational work across the worker nodes of a cluster, where some workers. Data Skipping Data Skipping is a query optimization technique in which a query engine uses pre-computed metadata statistics about data files to determine, at planning. Databricks An extensive overview of Databricks, the unified data analytics platform that pioneered the data lakehouse paradigm and developed Delta Lake and Apache Spark. Deserialization An in-depth look at deserialization and its performance impacts on analytical query engines. Distributed Compute A foundational overview of distributed compute architectures in data processing, explaining master-worker topologies, data shuffling, and fault tolerance. Dremio Dremio is an open lakehouse platform purpose-built for Apache Iceberg, providing a high-performance query engine, a semantic layer, an integrated Apache. DuckDB DuckDB is an open-source, in-process SQL OLAP database management system. File Skipping File Skipping is the collective term for the multi-level system of techniques that allow a data lakehouse query engine to identify and eliminate data files. Google BigQuery Google BigQuery is a fully managed, serverless, enterprise data warehouse that enables scalable analysis over petabytes of data. Hash Join A Hash Join is a join algorithm that uses a hash table to match rows from two input relations based on their join key values. Hilbert Curves The Hilbert Curve is a mathematically elegant space-filling curve first described by mathematician David Hilbert in 1891. Indexing (Data Lakes) Indexing in the context of data lakes and lakehouses refers to the set of auxiliary data structures that enable query engines to quickly locate relevant data. Join Strategies A Join Strategy is the specific algorithm a query engine uses to physically implement a SQL JOIN operation: matching rows from two (or more) tables based on. Materialized Views A Materialized View is a pre-computed, physically stored result of a SQL query whose output is saved to storage and can be queried directly, rather than. MPP (Massively Parallel Processing) Massively Parallel Processing (MPP) is an architectural design paradigm for distributed computing and databases. Out-of-Memory (OOM) Errors An Out-of-Memory (OOM) error occurs when a query execution process attempts to allocate more memory than is available to it, either on a single node. Partition Pruning Partition Pruning is the most powerful form of data skipping available to query engines in the data lakehouse. Predicate Pushdown Predicate Pushdown is a query optimization technique in which filter conditions (predicates) from the WHERE clause of a SQL query are applied as early as. Presto Presto (often referred to as PrestoDB to distinguish it from its fork, Trino) is an open-source, distributed SQL query engine designed for running. Projection Pushdown Projection Pushdown (also called column pruning) is a query optimization technique in which only the specific columns required by a SQL query are read from. Pushdown Optimization Pushdown Optimization (specifically Predicate Pushdown and Projection Pushdown) is one of the most critical performance techniques in distributed data. Query Execution Query Execution is the runtime phase in which a database or query engine carries out the physical execution plan produced by the query planner, reading data. Query Planning Query Planning is the process by which a database or query engine transforms a declarative SQL query into a detailed, optimized execution plan: a concrete. Rule-Based Optimizer (RBO) A Rule-Based Optimizer (RBO) is a critical component of a database's query planning phase. Serialization Serialization is the process of translating data structures or object state into a format that can be stored (for example, in a file or memory buffer) or. Shuffle In distributed query execution, a Shuffle (also called an Exchange or Repartition) is the operation of redistributing data rows across the worker nodes of a. Snowflake Snowflake is a fully managed cloud data platform that fundamentally reshaped the data warehousing industry. Sort-Merge Join A Sort-Merge Join (SMJ) is a join algorithm that sorts both input relations by their join key and then performs a single linear merge pass through both. Spilling to Disk Spilling to Disk is a query engine mechanism that writes intermediate query results (hash tables, sort buffers, shuffle data) to local disk storage when the. SQL Dialects SQL (Structured Query Language) is the lingua franca of data analytics. StarRocks StarRocks is an open-source, next-generation Massive Parallel Processing (MPP) database designed for blazing-fast, real-time analytics. Trino Trino (formerly known as PrestoSQL) is a highly parallel and distributed open-source SQL query engine. Vectorized Execution A detailed explanation of vectorized execution, the hardware-optimized processing model that allows modern compute engines to achieve blistering speeds. Z-Ordering Z-Ordering is a multi-dimensional data clustering technique used in data lakehouse environments to physically co-locate records with similar values across.