In the last seven days: eight shipped, eight in preview, five with a date on it and six new concepts on Lakenaut.

21 changes for you: one breaks something, five needs action, ten worth evaluating and five nice to know.

New features

• AI Runtime, serverless GPU training on Databricks, announced as generally available Databricks announced general availability of AI Runtime, its managed serverless GPU compute for training and fine-tuning models next to Unity Catalog data.

• Genie Code tasks in Lakeflow Jobs are generally available A Genie Code task runs a prompt autonomously as a step in a job, can read upstream task outputs and call tools, and returns a link to the resulting Genie Code chat.

• Databricks Runtime 19 hides row filter and column mask metrics in Spark Connect In the 19 October 2026 Databricks Runtime 19 update, Spark Connect DataFrame execution metrics no longer show internal operators and metrics tied to row filters and column masks.

• Service principal OAuth tokens can carry a custom identity claim for per-user row filtering A custom identity claim can now be added to a service principal's OAuth token and used to filter rows in a dynamic view or an ABAC policy.

• Asynchronous processing for Python transformWithState is generally available The Python row-based transformWithState API can now run state operations and user logic concurrently, which the release note says raises throughput with only minor code changes.

• Lakeflow system tables for pipelines reach general availability They record the pipelines and pipeline updates in an account, and are documented in the jobs system table reference.

• Saving an unchanged job no longer writes rows to the jobs system tables Saving a Lakeflow job without changing its settings no longer adds rows to system.lakeflow.jobs or system.lakeflow.job_tasks, and change_time stays at the last real change.

• DeepSeek V4.1 Flash (1017) is available on Unity Gateway with text and image input A new DeepSeek V4.1 Flash version, labelled 1017, can now be called on Unity Gateway through Foundation Model APIs and accepts text and images.

Previews and beta

• Lakebase Insights is configured per project; observability-configuration schedules have stopped Lakebase Insights (Beta, formerly Lakebase Advanced Postgres Telemetry) is now set up in each project's Settings by users with CAN MANAGE, with on-demand runs. Breaking: If you scheduled Insights through an observability configuration, enable Insights and set an analysis frequency in each project's Settings…

• DataFrame caching on serverless compute enters Public Preview df.cache() and df.persist() now work on serverless compute, so an expensive result can be reused for the rest of a session.

• Files API 3.0 for Unity Catalog volumes enters Beta with multipart uploads It adds multipart uploads, so files larger than 5 GB can be uploaded, and conditional operations, which upload, download or delete a file only if its current version matches.

• Serverless job tasks can configure their environment from pyproject.toml (Beta) Serverless notebook and Python script tasks in Lakeflow Jobs can now take their job environment from a pyproject.toml file, set in the Jobs UI, the Jobs API or a bundle.

• Notebooks and files in one bundle can share serverless compute (Beta) Shared Serverless for Asset Bundles lets notebooks and files in the same bundle share one serverless compute and an environment defined in the bundle-root pyproject.toml.

• Unity Gateway can screen MCP tool calls and results for jailbreaks (Beta) It screens MCP tool calls and their results for jailbreak and prompt injection attempts, and can be applied on input, output, or both.

• Lakeflow Connect adds a Box Audit Logs connector in Beta It ingests enterprise audit events from Box into Databricks.

• Lakeflow Connect adds a Microsoft Entra ID connector in Beta It ingests users, groups, applications, service principals, directory roles, identity configuration and audit and sign-in logs.

Deprecations and end of support

• DeepSeek V4 Flash (0731) will be retired on 5 November 2026 DeepSeek V4.1 Flash is the recommended replacement.

• Databricks Runtime 10.4 LTS ends on 2026-11-01 Anything still relying on it needs a plan before then.

• Genie One MCP Beta endpoint (/api/2.0/mcp/genie) ends on 2026-10-31 Anything still relying on it needs a plan before then.

• DeepSeek V4 Pro (0813) ends on 2026-10-30 Anything still relying on it needs a plan before then.

• DeepSeek V4.1 Flash will be retired on 19 November 2026 The release note names DeepSeek V4.1 Flash (1017) as the recommended replacement and points to the deprecated and retired models page for migration guidance. Action required, by 19 Nov 2026: Move workloads that call DeepSeek V4.1 Flash to DeepSeek V4.1 Flash (1017).

New and rewritten concepts

• Architecture of the Data Intelligence Platform Databricks separates a vendor-managed control plane from a compute plane that processes the data, with storage in the customer's cloud, Delta Lake as the format.

• Using Genie outside the UI: API, embedding and agents The Conversation API, embedding, Slack and Teams, MCP and the Supervisor Agent let a Genie Agent answer from anywhere, always as an identity whose permissions apply.

• Choosing SQL or Python for a pipeline Both pipeline interfaces build the same dataflow graph, so most of the choice is taste. The asymmetries are few and they all run one way: Python covers the whole feature set.

• Privileges: GRANT, REVOKE, and DENY Unity Catalog privileges are granted to users, groups and service principals and inherit from the catalog down. USE CATALOG and USE SCHEMA are the entry door.

• Domains and Pages The human half of the Genie Ontology. Domains group assets by business purpose, Pages define what a business term actually means, and Genie prefers both over what it infers.

• SQL scripting and stored procedures BEGIN ... END compound blocks with local variables, loops, EXECUTE IMMEDIATE, condition handlers and cursors, and how to persist a working script as a Unity Catalog procedure.