Streaming
ReleaseWorth evaluating
On-demand state repartitioning added for Structured Streaming
From Databricks blog
Databricks announced on-demand state repartitioning for Apache Spark Structured Streaming on Databricks. The entry is aimed at people running stateful Structured Streaming queries in production.
Learners working with stateful streaming should know that Databricks now has a feature for repartitioning streaming state on demand.
What to re-read
More from Streaming
- ReleaseReal-Time Mode reaches GA for Lakeflow pipelines, with ForEach sinks to operational stores
- PreviewSeveral real-time flows can share one serverless pipeline; both new options are in Beta
- ReleaseAsynchronous processing for Python transformWithState is generally available
- ReleaseOn-demand state repartitioning is generally available for stateful streaming