Compute
What actually runs the query?
The engines that plan and execute queries, and the optimisations that make them fast over object storage.
43 reference entries
Amazon AthenaAmazon Athena is a serverless, interactive query service provided by Amazon Web Services (AWS) that allows users to analyze data directly in Amazon Simple.Apache DorisApache Doris is a modern, open-source Massively Parallel Processing (MPP) analytical database designed for blazing-fast, real-time data warehousing.Apache FlinkApache Flink is an open-source, unified stream-processing and batch-processing framework.Apache SparkApache Spark is a unified analytics engine designed for large-scale data processing.Broadcast JoinA Broadcast Join is a distributed join algorithm where the smaller of the two join inputs is completely replicated ("broadcast") to every worker node in the.CachingCaching in data systems is the practice of storing copies of frequently accessed data in a faster, closer storage layer so that subsequent requests for the.ClickHouseClickHouse is an open-source, column-oriented database management system (DBMS) built expressly for online analytical processing (OLAP).Compute EngineIn traditional data warehousing, storage and compute were inextricably linked.Cost-Based Optimizer (CBO)A Cost-Based Optimizer (CBO) is the intelligent "brain" within a modern relational database or distributed compute engine.Data SkewData Skew in distributed query execution is the unequal distribution of data or computational work across the worker nodes of a cluster, where some workers.Data SkippingData Skipping is a query optimization technique in which a query engine uses pre-computed metadata statistics about data files to determine, at planning.DatabricksAn extensive overview of Databricks, the unified data analytics platform that pioneered the data lakehouse paradigm and developed Delta Lake and Apache Spark.DeserializationAn in-depth look at deserialization and its performance impacts on analytical query engines.Distributed ComputeA foundational overview of distributed compute architectures in data processing, explaining master-worker topologies, data shuffling, and fault tolerance.DremioDremio is an open lakehouse platform purpose-built for Apache Iceberg, providing a high-performance query engine, a semantic layer, an integrated Apache.DuckDBDuckDB is an open-source, in-process SQL OLAP database management system.File SkippingFile Skipping is the collective term for the multi-level system of techniques that allow a data lakehouse query engine to identify and eliminate data files.Google BigQueryGoogle BigQuery is a fully managed, serverless, enterprise data warehouse that enables scalable analysis over petabytes of data.Hash JoinA Hash Join is a join algorithm that uses a hash table to match rows from two input relations based on their join key values.Hilbert CurvesThe Hilbert Curve is a mathematically elegant space-filling curve first described by mathematician David Hilbert in 1891.Indexing (Data Lakes)Indexing in the context of data lakes and lakehouses refers to the set of auxiliary data structures that enable query engines to quickly locate relevant data.Join StrategiesA Join Strategy is the specific algorithm a query engine uses to physically implement a SQL JOIN operation: matching rows from two (or more) tables based on.Materialized ViewsA Materialized View is a pre-computed, physically stored result of a SQL query whose output is saved to storage and can be queried directly, rather than.MPP (Massively Parallel Processing)Massively Parallel Processing (MPP) is an architectural design paradigm for distributed computing and databases.Out-of-Memory (OOM) ErrorsAn Out-of-Memory (OOM) error occurs when a query execution process attempts to allocate more memory than is available to it, either on a single node.Partition PruningPartition Pruning is the most powerful form of data skipping available to query engines in the data lakehouse.Predicate PushdownPredicate Pushdown is a query optimization technique in which filter conditions (predicates) from the WHERE clause of a SQL query are applied as early as.PrestoPresto (often referred to as PrestoDB to distinguish it from its fork, Trino) is an open-source, distributed SQL query engine designed for running.Projection PushdownProjection Pushdown (also called column pruning) is a query optimization technique in which only the specific columns required by a SQL query are read from.Pushdown OptimizationPushdown Optimization (specifically Predicate Pushdown and Projection Pushdown) is one of the most critical performance techniques in distributed data.Query ExecutionQuery Execution is the runtime phase in which a database or query engine carries out the physical execution plan produced by the query planner, reading data.Query PlanningQuery Planning is the process by which a database or query engine transforms a declarative SQL query into a detailed, optimized execution plan: a concrete.Rule-Based Optimizer (RBO)A Rule-Based Optimizer (RBO) is a critical component of a database's query planning phase.SerializationSerialization is the process of translating data structures or object state into a format that can be stored (for example, in a file or memory buffer) or.ShuffleIn distributed query execution, a Shuffle (also called an Exchange or Repartition) is the operation of redistributing data rows across the worker nodes of a.SnowflakeSnowflake is a fully managed cloud data platform that fundamentally reshaped the data warehousing industry.Sort-Merge JoinA Sort-Merge Join (SMJ) is a join algorithm that sorts both input relations by their join key and then performs a single linear merge pass through both.Spilling to DiskSpilling to Disk is a query engine mechanism that writes intermediate query results (hash tables, sort buffers, shuffle data) to local disk storage when the.SQL DialectsSQL (Structured Query Language) is the lingua franca of data analytics.StarRocksStarRocks is an open-source, next-generation Massive Parallel Processing (MPP) database designed for blazing-fast, real-time analytics.TrinoTrino (formerly known as PrestoSQL) is a highly parallel and distributed open-source SQL query engine.Vectorized ExecutionA detailed explanation of vectorized execution, the hardware-optimized processing model that allows modern compute engines to achieve blistering speeds.Z-OrderingZ-Ordering is a multi-dimensional data clustering technique used in data lakehouse environments to physically co-locate records with similar values across.