Streaming
PreviewNice to know
Several real-time flows can share one serverless pipeline; both new options are in Beta
From Databricks release notes (AWS)
On serverless pipelines using the preview channel, multiple real-time mode flows can run in one pipeline without competing for task slots. Source stage adaptive repartitioning adjusts the number of read tasks per Kafka or Kinesis source to its traffic, so maxPartitions or shardsPerTask need not be set. Both are in Beta.
What to re-read
More from Streaming
- ReleaseReal-Time Mode reaches GA for Lakeflow pipelines, with ForEach sinks to operational stores
- ReleaseAsynchronous processing for Python transformWithState is generally available
- ReleaseOn-demand state repartitioning is generally available for stateful streaming
- ReleaseOn-demand state repartitioning added for Structured Streaming