Transaction logs are the fundamental backbone of modern Open Table Formats, enabling ACID transactions, metadata management, time travel capabilities, and concurrent operations on Data Lakes. This note explores how transaction logs are implemented across three major open table formats: Delta Lake, Apache Hudi, and Apache Iceberg.
# Core Concepts of Transaction Logs
A transaction log is an ordered record of all operations performed on a table since its creation. It serves as:
- Single Source of Truth: The definitive record of a table’s state and historyACID Transaction Enabler: Ensures atomicity, consistency, isolation, and durabilityMetadata Management System: Tracks schema, partitioning, and file informationConcurrency Controller: Manages multiple simultaneous reads and writesTime Travel Facilitator: Enables querying historical table states
# Examples
# Delta Lake Transaction Log
Delta Lake Transaction Log Structure:
| |
See detail deep dive in: Transaction Log (Delta Lake).
# Apache Iceberg Transaction Log
Structure and Implementation
- Layered Architecture: Comprises catalog layer, metadata layer, and data layerMetadata Files: Store global table metadata (schema, partitioning, properties)Snapshots: Represent table state at specific points in timeManifest Files: Track data files, including their locations, sizes, and statisticsAtomic Swaps: Table state updates create new metadata files replaced via atomic swaps
Key Functions
- Catalog Operations: Atomic operations at the catalog level ensure transaction correctnessOptimistic Concurrency: Uses sequence numbers to maintain consistency with concurrent transactionsMetadata Logging: Records history of metadata changes for rollback capabilitiesSchema Evolution: Supports schema changes without table rewrites
Apache Iceberg Transaction Log Structure:
| |
More details:
# Apache Hudi Transaction Log
Structure and Implementation
- Timeline-Based Architecture: Organizes transactions as actions along a timelineFile Organization: Uses directory-based approach with timestamped files and log files tracking changesMetadata Table: Tracks file information for query optimization (default since v0.11.0)Commit Files: Uses files with naming convention [timestamp].[transaction state] to track transaction states
Key Functions:
- Record-Level Index: Maintains mapping between record keys and file groupsOptimistic Concurrency: File-level, log-based concurrency control based on instant timesAsynchronous Operations: Supports background operations like compaction without blocking ingestionCopy-on-Write vs. Merge-on-Read: Offers two table types with different performance characteristics
Apache Hudi Transaction Log Structure:
| |
More details:
- Apache Hudi Concepts - Official documentation explaining Hudi’s timeline, file organization, and table structure.
# DuckLake Transaction Table
Since a short while, we also have DuckLake. This stores the metadata in a SQL database instead of metadata on disk.
# DuckLake Table Structure
The data model as of 2025-06-05 looks like this:
Check out more at DuckLake.
# Comparison of Transaction Log Approaches
| Feature | Delta Lake | Apache Hudi | Apache Iceberg |
|---|---|---|---|
| Concurrency Control | Optimistic concurrency control with mutual exclusion and retry mechanism | File-level, log-based concurrency control ordered by start instant times | Sequence number-based optimistic concurrency control |
| Metadata Management | JSON log files with Parquet checkpoints every 10 commits | Timeline-based approach with metadata table for query optimization | Layered approach with catalog pointing to metadata files |
| Update Handling | Breaks operations into atomic commits recorded sequentially | Offers Copy-on-Write and Merge-on-Read approaches for different performance needs | Supports eager data file rewrites or delete deltas for faster updates |
| Performance Characteristics | Efficient for append-heavy workloads with Spark integration | Excels at update-heavy workloads with upserts and record-level indexing | Offers strong query performance with optimized metadata handling |
| Time Travel | Supports via transaction log processing | Supports via timeline-based snapshots | Supports via versioned metadata and snapshots |
| Origins | Developed by Databricks | Developed by Uber | Developed by Netflix |
| Primary Integration | Apache Spark | Multiple engines with Spark, Flink, and Hive focus | Multi-engine with strong Spark, Flink, Trino support |
| Schema Evolution | Supported with column additions/deletions | Supported with schema tracking | Extensive support with in-place evolution |
See also more on Open Table Formats, The Open Table Format Revolution, and Composable Open Data Platform: How Iceberg and Open Table Formats Are Reshaping Data Architecture.
Origin: Data Lake Table Format
References:
Created 2025-04-29
