Product area
Delta Lake
Delta Lake as the lakehouse storage format: transactions, time travel, medallion architecture, gold-layer objects, liquid clustering, predictive optimization.
Concepts
- 1Delta Lake, the lakehouse table formatDelta Lake is the default table format on Databricks. A Delta table is a folder in object storage with two things inside: the data files in Parquet and a deltalog/ subfolder holding the transaction log, a sequence of JSON files (plus Parquet checkpoints) that…Beginner
- 2Time travel and table historyEvery write to a Delta table (see delta lake overview) becomes a numbered version in the transaction log. Time travel is the ability to query, or restore, the table as it looked at any past version or timestamp, as long as the pieces that version needs — the…Beginner
- 3Medallion architecture: bronze, silver, goldThe medallion architecture organizes the tables of a lakehouse into three layers, each with a different quality guarantee:Beginner
- 4Gold objects: tables, views, materialized views, streaming tablesGold is the layer that dashboards, analysts, and models read from (see medallion architecture). In Unity Catalog you can expose it through four different objects, all queried with a plain SELECT but with very different behavior underneath:Intermediate
- 5Upsert with MERGE INTOMERGE INTO applies a batch of changes to a Delta table in a single atomic commit. You give it a target table, a source (a table, a view, a subquery or a DataFrame), a join condition, and then clauses describing what to do with rows that match, rows that exist…Intermediate
- 6Liquid clusteringLiquid Clustering is how Delta Lake (see delta lake overview) physically organizes a table's files around one or more columns, so that queries filtering on those columns read fewer files. It replaces two older techniques: folder-based partitioning and the…Intermediate
- 7OPTIMIZE, VACUUM, and file layoutOPTIMIZE and VACUUM are the two file-maintenance commands every Delta table (see delta lake overview) eventually needs. OPTIMIZE compacts many small data files into fewer, larger ones and, optionally, reorders their contents for faster filtering. VACUUM…Intermediate
- 8Change Data FeedChange Data Feed (CDF) makes a Delta table (see delta lake overview) emit a row-level log of what changed on each write, not just the resulting state. Once enabled, every INSERT, UPDATE, DELETE, and MERGE is queryable as a stream of change records, each…Intermediate
- 9Predictive optimizationPredictive optimization is a managed service that runs three maintenance commands on Unity Catalog managed tables without being asked: OPTIMIZE, VACUUM, and ANALYZE. It looks at how each table is written and queried, decides which tables would benefit from…Intermediate
- 10Partitioning, Z-order, and data skippingBefore liquid clustering, getting a Delta table to read quickly meant four separate levers, and you operated all of them yourself:Intermediate
- 11Deletion vectorsDeletion vectors are a table feature, available on both Delta Lake and Apache Iceberg tables, that turns a row-level change into a metadata write. Without them, removing one row from a 500 MB Parquet file means reading that file, dropping the row, and writing…Intermediate
- 12Iceberg on DatabricksApache Iceberg is the other open table format, and Databricks meets it in four different places. They are easy to confuse because all four involve the word Iceberg and only one of them is about writing Iceberg tables on Databricks.Intermediate
- 13Table history and transaction log checkpointsEvery operation that modifies a Delta Lake or managed Iceberg table creates a new version, and DESCRIBE HISTORY returns one row per version in reverse chronological order, 14 columns wide. It is the table's own record of what happened to it: who ran what…Intermediate
News
No news for this area.