ETL, which stands for Extract, Transform, and Load, involves a three-phase process for moving data. Unlike ELT, ETL requires pre-defining the data schema and completing transformations before the data reaches its final destination. In contrast, ELT involves loading data first and then transforming it.

ETL processes are evolving. Historically, they were executed using tools like Informatica, IBM Datastage, Cognos, AbInitio, or Microsoft SSIS. Nowadays, the trend leans towards more programmatic or configuration-driven platforms such as Apache Airflow, Dagster, and Temporal. This shift coincides with growing data demands and the need for quicker data accessibility, steering the trend towards ELT.

History: Initially, data management focused on OLTP (Online Transactional Processing) systems optimized for fast, concurrent read/write operations, ensuring ACID properties (atomicity, consistency, isolation, durability). As businesses sought to derive insights from their operational data, OLAP (Online Analytical Processing) systems emerged, utilizing data warehouses optimized for read-only, complex queries for analytics. The bridge between OLTP and OLAP systems was formed by ETL processes, facilitating the flow of current operational data to historical analytical data.

Present: The advent of real-time data integration solutions, like CDC (Change Data Capture) and ELT (Extract, Load, Transform), reflects the shift towards immediate data processing needs. CDC focuses on moving only changed data in real-time, offering low latency but limited transformation capabilities. ELT, on the other hand, moves the transformation phase to the target data platform, allowing for in-situ data transformations. This shift is driven by the increasing volume of data, the decreasing costs of storage and compute resources, and the rise of cloud data platforms that support analyzing diverse data sources efficiently.

Future Directions: The document discusses potential future directions for ETL technology, including the adaptation to new data ecosystem challenges, such as streaming ETL, cloud-native ELT, and the integration of user-defined functions (UDFs) for in-engine ETL processing. These advancements aim to leverage modern data architectures and optimization techniques to improve the efficiency, scalability, and flexibility of ETL processes.

Read more on The History, Present, and Future of ETL Technology ( Zotero).

ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) are two methodologies for data handling. Traditionally, ETL has been the backbone of Data Warehouse processes, while ELT is more commonly associated with Data Lake creation.

See also: ETL is changing, ETL with Apache Airflow.

See ETL Tools.

See Data Integration.

There are at least three types of sources:

Common APIs: They are well served by the current ETL tools (Fivetran, Stitch, etc.)Custom & long-tail connectors: Right now, 95% of companies will have in-house data engineering teams that will build in-house scripts to handle those connectorsHigh-volume databases and files: The row-based pricing model is incompatible with those use cases, making it prohibitive to use those sources.

References: ETL vs ELT, ELT