Streaming
ReleaseWorth evaluating
On-demand state repartitioning is generally available for stateful streaming
From Databricks release notes (AWS)
You can now change the number of shuffle partitions of a stateful Structured Streaming query or a Lakeflow pipeline without losing its checkpoint state. The feature is generally available.
What to re-read
More from Streaming
- ReleaseOn-demand state repartitioning added for Structured Streaming
- PreviewManaged tables can be created in Lakeflow pipelines in Beta