Change Data Feed

What it is

Change Data Feed (CDF) makes a Delta Lake table (see Delta Lake, the lakehouse table format) or an Apache Iceberg v3 table expose a row-level view of what changed between table versions, not just the resulting state. Databricks now describes two forms: automatic CDF, which is available for tables that meet the requirements without setting a CDF table property, and legacy CDF, which is turned on per Delta table and materializes change records during writes. Change records are tagged with _change_type, _commit_version, and _commit_timestamp.

Automatic CDF uses the same table_changes() and readChangeFeed APIs as legacy CDF. The documentation also says automatic CDF works with batch reads, Structured Streaming, and Databricks-to-Databricks OpenSharing.

Why it exists

Without CDF, propagating a change from one layer of the medallion (see Medallion architecture: bronze, silver, gold) to the next means either reprocessing the whole source table on every run, or hand-building your own audit columns and diffing logic to figure out what’s new. Full reprocessing is simple and safe, but it doesn’t scale: recomputing a multi-billion-row silver table to pick up a few thousand changed rows wastes most of the compute it uses. CDF gives you exactly the rows that changed, in the order they changed, without that cost.

How it works

Enabling it

Legacy CDF for Delta Lake is enabled with a table property:

ALTER TABLE main.silver.orders SET TBLPROPERTIES (delta.enableChangeDataFeed = true);

CREATE TABLE main.silver.orders (...)
TBLPROPERTIES (delta.enableChangeDataFeed = true);

Legacy CDF is off by default on a plain Delta table; turning it on only affects writes made after that point — there’s no way for legacy CDF to retroactively generate change records for history that already happened. Automatic CDF is different: it does not use this table property, and it applies automatically when the table meets the documented requirements, including Unity Catalog registration and the supported table format and row-tracking conditions.

Databricks says you can’t use automatic CDF and legacy CDF together on the same table. To move a Delta table from legacy CDF to automatic CDF, unset delta.enableChangeDataFeed after verifying that the table meets the automatic CDF requirements.

Reading changes

SELECT * FROM table_changes('main.silver.orders', 10, 20);
SELECT * FROM table_changes('main.silver.orders', '2026-09-01', '2026-09-05');

table_changes takes either a version range or a timestamp range and returns one row per change, decorated with the three metadata columns:

An UPDATE produces two rows: update_preimage (the row before) and update_postimage (the row after). Anything that just counts rows by _change_type without accounting for both will double-count updates.

Reading CDF as a stream

(spark.readStream
  .option("readChangeFeed", "true")
  .table("main.silver.orders"))

This turns the change feed itself into a Structured Streaming source (see Structured Streaming on Databricks). Databricks uses the stream checkpoint to track which table versions have already been processed. When a stream starts without an explicit starting version, CDF first emits the current table snapshot as insert records, then emits later table changes.

Incremental silver → gold propagation

The usual pattern reads the CDF stream from silver and applies it to gold with foreachBatch, using _change_type to decide the operation:

def apply_changes(batch_df, batch_id):
    batch_df.createOrReplaceTempView("changes")
    batch_df.sparkSession.sql("""
        MERGE INTO main.gold.orders t
        USING (
          SELECT * FROM changes
          QUALIFY row_number() OVER (
            PARTITION BY order_id ORDER BY _commit_version DESC
          ) = 1
        ) s
        ON t.order_id = s.order_id
        WHEN MATCHED AND s._change_type = 'delete' THEN DELETE
        WHEN MATCHED THEN UPDATE SET *
        WHEN NOT MATCHED AND s._change_type != 'delete' THEN INSERT *
    """)

(spark.readStream
  .option("readChangeFeed", "true")
  .table("main.silver.orders")
  .writeStream
  .foreachBatch(apply_changes)
  .option("checkpointLocation", "/Volumes/shop/streaming/_checkpoints/gold_orders")
  .trigger(availableNow=True)
  .start())

Keeping only the last change per key per batch (the QUALIFY) avoids applying an out-of-order sequence of updates to the same row within one micro-batch.

Limits

CDF is not a permanent audit archive. Change records are transient, and when table history is cleaned up, the corresponding change feed versions stop being readable too. If you need a permanent history, write the CDF output into another Delta table before the source table’s retention window removes the versions you need.

Legacy CDF is only available from the version where it was turned on: there’s no legacy change history before that point, and an interval when legacy CDF was turned off is not queryable through legacy CDF. CDF also isn’t a replacement for the table itself — it’s a log of transitions, not a snapshot you can query on its own for the current state.

Automatic CDF has its own conditions. It needs Databricks Runtime 19 or above, and the table must be registered in Unity Catalog: a managed Delta table with row tracking enabled, a managed Iceberg v3 table, or an external Delta table with row tracking enabled. It isn’t supported on tables with row filters or column masks, or when the source table is modified inside a multi-statement transaction. Only Databricks readers can query it; external Iceberg clients can’t. For either form, a query can’t span a table version where a non-additive schema change happened (renaming or dropping a column, changing a type or nullability), so split the range on either side of it.

CDF vs. reprocessing everything

Common mistakes

• Counting _change_type = 'update_postimage' rows as inserts, or not filtering out update_preimage from a simple downstream count.
• Expecting legacy CDF to reconstruct changes from before it was enabled: it only sees what happens after that point.
• Never reconciling with a full recompute: a schema change or a bug in the propagation logic can drift gold away from silver over time in ways a change-only pipeline won’t catch on its own.
• Leaving legacy CDF enabled on a high-churn table without factoring in the extra storage: legacy change files persist for the same retention window as the data they describe.