Product area
Streaming
Structured Streaming on Databricks: triggers, checkpoints, watermarks, stateful operations, real-time mode.
Concepts
- 1Structured Streaming on DatabricksStructured Streaming is Spark's model for incremental processing: a stream is an unbounded table that keeps getting new rows appended to it, and a streaming query is a regular DataFrame query that Spark re-runs, incrementally, every time new rows show up. You…Intermediate
- 2Trigger intervals in Structured StreamingThe trigger is the single setting on writeStream that decides when a streaming query goes looking for new data and how long a micro-batch is allowed to run. It says nothing about how much data ends up in a batch: that is the job of source-side limits like…Intermediate
- 3Reading and writing Apache KafkaKafka is reachable from Structured Streaming through the kafka format, both as a source (spark.readStream.format("kafka")) and as a sink (writeStream.format("kafka")). The same format works in batch mode with spark.read and spark.write, which is how you…Intermediate
- 4Arbitrary sinks with foreachBatchforeachBatch is the escape hatch in writeStream. Instead of naming a sink, you hand it a function with the signature (df, batchId): df is the output of one micro-batch as an ordinary batch DataFrame, and batchId is the monotonically increasing number…Advanced
- 5Watermarks and stateful streamingA watermark is a moving threshold on event time — the timestamp recorded inside each record, as opposed to processing time, when Spark happens to see it — that tells a stateful streaming query how late a record is allowed to be before it's ignored. Anything…Advanced
Also covered here
- Auto LoaderAuto Loader is a Structured Streaming source, identified by the cloudFiles format, that watches a directory in S3, ADLS, GCS, or a Unity Catalog volume and processes files as they arrive. It reads JSON, CSV, XML, Parquet, Avro, ORC, text, and binary files. It…
- Change Data FeedChange Data Feed (CDF) makes a Delta Lake table (see delta lake overview) or an Apache Iceberg v3 table expose a row-level view of what changed between table versions, not just the resulting state. Databricks now describes two forms: automatic CDF, which is…
- Streaming tables from Databricks SQLA streaming table is a Unity Catalog managed table that only ever appends, and that keeps itself up to date by reading new rows from its source. The standalone version is the one you declare in the SQL editor with CREATE OR REFRESH STREAMING TABLE, without…
News
Resources
Configure Structured Streaming trigger intervalsDocumentation · Databricks · checked 17 Oct 2026Free
Nothing of that kind for this area. Try the full list.