Compute
ReleaseWorth evaluating
Apache Spark 4.3 is available in Databricks Runtime 19
From Databricks blog
Spark 4.3 ships in Databricks Runtime 19, with stateful operations (aggregation, deduplication, transformWithState) in Real-Time Mode and SCD Type 2 support in Auto CDC. The post also describes opt-in small-data optimisations under Project Feather and says Spark moves to a feature release every three months.
What to re-read
More from Compute
- ReleaseServerless compute access controls reach GA; rate limits and compute tags in Beta
- PreviewServerless job logs can be delivered to a Unity Catalog volume (Beta)
- PreviewDataFrame caching on serverless compute enters Public Preview
- PreviewShared Serverless lets notebooks and files in one bundle share compute and an environment (Beta)