Movement
How does data get in, and stay current?
How data arrives and keeps arriving: ingestion, transformation, orchestration, and streaming.
15 reference entries
Apache Airflow Apache Airflow is an open-source platform created by Airbnb in 2014 and later donated to the Apache Software Foundation. Batch Processing Batch Processing is the execution of a series of jobs in a computer program without manual intervention (non-interactive). Change Data Capture (CDC) Change Data Capture (CDC) is a set of software design patterns and technologies used to determine and track the data that has changed within a source. Dagster Dagster is an open-source data orchestration platform designed to address some of the architectural limitations of Apache Airflow. Data Pipeline A Data Pipeline is an automated set of processes and infrastructure that extracts data from various source systems, transforms it into a clean and usable. dbt (data build tool) Understanding dbt, the transformative framework that brought software engineering best practices to SQL-based data transformations. Directed Acyclic Graph (DAG) A Directed Acyclic Graph (DAG) is a conceptual mathematical model heavily utilized in computer science, specifically within the area of data engineering and. ELT (Extract, Load, Transform) For decades, the dominant pattern for moving data from source systems into analytical databases was ETL: Extract, Transform, Load. ETL (Extract, Transform, Load) Before a business analyst can run a query against a data warehouse, someone has to get the data there. Eventual Consistency In any discussion of data reliability in distributed systems, the conversation inevitably polarizes around two competing consistency models: ACID (Atomicity. Micro-batching Exploring Micro-batching, the architectural compromise that simulates streaming using rapid, tiny batch jobs. Orchestration In data engineering, Orchestration is the automated configuration, coordination, and management of complex computer systems, software, and services. Prefect Exploring Prefect, the dynamic, Python-native workflow orchestration framework. Streaming Data Streaming Data refers to data that is continuously generated by thousands of data sources, which typically send in the data records simultaneously, and in. Strong Consistency Strong Consistency is the most demanding correctness guarantee a distributed storage system can provide.