|
In the last seven days: nine shipped, nine in preview, six with a date on it and six new concepts on Lakenaut.
24 changes for you: seven needs action, eight worth evaluating and nine nice to know.
 |
New features |
|
-
|
|
From 18 November 2026, new tables with a VARIANT column created on Databricks Runtime 17.3 or above get the variantShredding table feature automatically. Action required, by 18 Nov 2026: Upgrade any cluster still on Databricks Runtime 15.4 that touches Variant tables to 16.4 or above before the change. |
-
|
|
Spark 4.3 ships in Databricks Runtime 19, with stateful operations (aggregation, deduplication, transformWithState) in Real-Time Mode and SCD Type 2 support in Auto CDC. |
-
|
|
When a Structured Streaming query writes to Lakebase, merge strategies now control how an upsert updates each column: keep the first or last value, ignore nulls. |
-
|
|
From November 1, 2026, Unity Gateway budgets include spend from task-specific AI Functions such as ai_parse_document, ai_extract and ai_classify. |
-
|
|
Debezium Server 3.7 ships a sink that sends change events from a source database straight to Zerobus Ingest. |
-
|
|
The cap on Unity Catalog secrets rises from 100 to 1,000 per schema and from 1,000 to 10,000 per metastore. |
-
|
|
At GA it is offered in asia-southeast1, europe-west3, us-central1, us-east4 and us-west1. |
-
|
|
Zerobus Ingest in Lakeflow Connect can now write to tables backed by default storage with general availability status. |
-
|
|
Auto mode reviews tool calls so that routine permission prompts appear less often. |
 |
Previews and beta |
|
-
|
|
Jobs can now include a task that runs ai_parse_document, ai_extract or ai_classify over files in a volume or rows in a table, with no SQL to write. |
-
|
|
Lakeflow Connect now has a managed Okta connector in Beta that ingests identity and access data: users, groups, group memberships, app integrations, devices. |
-
|
|
A Lakeflow Job can be set to run when one or more upstream jobs complete, including jobs in other workspaces attached to the same Unity Catalog metastore. |
-
-
|
|
Lakeflow Connect database connector pipelines now show a Data freshness column: the time since the latest source commit ingested for each change data capture table. |
-
|
|
Read queries through Lakehouse//RT can now run against tables protected by row filters, column masks and attribute-based access control policies. |
-
|
|
MLflow runs the trace queries on a SQL warehouse it creates in the workspace where supported, and on serverless compute otherwise. |
-
|
|
It ingests security alerts from Microsoft Defender XDR, Microsoft Sentinel and other Microsoft security products. |
-
|
|
Audit logs record skill list, load and file fetch events under the aiGatewaySkills service, and the usage dashboard has a Skills tab showing downloads, loads and activity by user. |
 |
Deprecations and end of support |
|
-
|
|
DeepSeek V4.1 Flash is the recommended replacement. |
-
|
|
Anything still relying on it needs a plan before then. |
-
|
|
Anything still relying on it needs a plan before then. |
-
|
|
Anything still relying on it needs a plan before then. |
-
|
|
The release notes name Kimi K3 as the recommended replacement and point to the deprecated and retired models page for migration guidance. Action required, by 30 Oct 2026: Move any workload that calls Kimi K2.7 to Kimi K3, following the deprecated and retired models page. |
-
|
|
The release notes recommend GLM 5.3 or Kimi K3 as replacements and link to the deprecated and retired models page for migration guidance. Action required, by 30 Oct 2026: Move any workload that calls Inkling to GLM 5.3 or Kimi K3, following the deprecated and retired models page. |
 |
New and rewritten concepts |
|
-
|
|
How a Lakebase project is organised into copy-on-write branches, each with its own computes, roles and databases, and how expiry, protection. |
-
|
|
A materialized view created from Databricks SQL gets its own serverless pipeline. How to schedule the refresh, when it is incremental. |
-
|
|
The three generally available warehouse types and the Beta fourth one, which acceleration features each has, how fast each starts, and what the Preview channel is for. |
-
|
|
Writing records straight into a Unity Catalog table from an application, over gRPC for throughput or REST for edge fleets, with no message bus in between. |
-
|
|
Deletion vectors record deleted and updated rows in metadata instead of rewriting whole Parquet files. |
-
|
|
The four ways a dashboard becomes interactive: field filters, query parameters, dashboard variables, and click-driven cross-filtering and drill-through. |
|
From 18 November 2026, new tables with a VARIANT column created on Databricks Runtime 17.3 or above get the variantShredding table feature automatically. Action required, by 18 Nov 2026: Upgrade any cluster still on Databricks Runtime 15.4 that touches Variant tables to 16.4 or above before the change.
Jobs can now include a task that runs ai_parse_document, ai_extract or ai_classify over files in a volume or rows in a table, with no SQL to write.
DeepSeek V4.1 Flash is the recommended replacement.
How a Lakebase project is organised into copy-on-write branches, each with its own computes, roles and databases, and how expiry, protection.