Product area
Streaming
Structured Streaming on Databricks: triggers, checkpoints, watermarks, stateful operations, real-time mode.
Concepts
- 1Structured Streaming on DatabricksStructured Streaming is Spark's model for incremental processing: a stream is an unbounded table that keeps getting new rows appended to it, and a streaming query is a regular DataFrame query that Spark re-runs, incrementally, every time new rows show up. You…Intermediate
- 2Trigger intervals in Structured StreamingThe trigger is the single setting on writeStream that decides when a streaming query goes looking for new data and how long a micro-batch is allowed to run. It says nothing about how much data ends up in a batch: that is the job of source-side limits like…Intermediate
- 3Reading and writing Apache KafkaKafka is reachable from Structured Streaming through the kafka format, both as a source (spark.readStream.format("kafka")) and as a sink (writeStream.format("kafka")). The same format works in batch mode with spark.read and spark.write, which is how you…Intermediate
- 4Arbitrary sinks with foreachBatchforeachBatch is the escape hatch in writeStream. Instead of naming a sink, you hand it a function with the signature (df, batchId): df is the output of one micro-batch as an ordinary batch DataFrame, and batchId is the monotonically increasing number…Advanced
- 5Watermarks and stateful streamingA watermark is a moving threshold on event time — the timestamp recorded inside each record, as opposed to processing time, when Spark happens to see it — that tells a stateful streaming query how late a record is allowed to be before it's ignored. Anything…Advanced
News
No news for this area.