What changes for you

Key-value parameters for Lakeflow pipelines have left preview. One pipeline's SQL source can run against different catalogs, schemas or tables: defaults live in the pipeline settings, an update can override them, and the SQL refers to them with named parameter syntax.

Feature Views, the declarative way to define and compute features, are generally available for batch, streaming and request-time data. For workspaces with the compliance security profile, they will be switched on automatically in mid-November 2026.

Genie One can now run code in a secure, isolated sandbox for analyses that go beyond SQL. The code runs with the user's existing permissions, so Unity Catalog governance and data access controls still apply. The feature is in Public Preview.

Discover, a curated way to browse assets without knowing the catalog hierarchy, and domains with subdomains are now generally available. Domains group tables, dashboards, notebooks and Genie Agents by business area, allow curated highlights, and surface AI-recommended assets.

Dataset materialization for AI/BI dashboards is in Beta. Dataset results are computed on a schedule and stored, so a published dashboard renders from the stored results instead of rerunning expensive queries on every load or interaction.

In Beta, an external service policy on Unity Gateway forwards request and response content to a third-party guardrail vendor and blocks the call if the vendor denies it. Applications need no changes.

Databricks Apps can now call supported APIs as the signed-in user, so Unity Catalog row filters and column masks apply to that user, while API scopes cap what the app may do. A read-only query app can request sql:restricted-query rather than the broader sql scope, and workspace admins can restrict which scopes developers may add.

DeepSeek V4.1 Flash can now be served with priority pay-per-token for latency-sensitive traffic, or with reserved provisioned throughput for dedicated capacity.

Genie Code CLI is a coding agent that runs in a terminal and works on local files. It uses the Databricks CLI to find data, answer questions, and build and deploy pipelines, models and apps. Model access goes through Unity Gateway. It is labelled Beta.

Metric views can now define window measures, which cover rolling windows, cumulative totals and semiadditive aggregations such as moving averages, period-over-period change and running totals. The feature is generally available as of this note. Anyone modelling time-based KPIs in metric views no longer has to treat it as a preview.

A Lakeflow job can now carry up to five triggers. Trigger types can be mixed, several triggers of the same type are allowed, and each trigger can be paused on its own. The release note labels this Beta.

NEAREST BY is a SQL join that finds, for each row on the left, the k closest rows on the right by vector similarity or distance, either EXACT or APPROX. It runs in Photon with a fused operator, and APPROX queries can use an IVF index stored as a liquid-clustered Delta table. The post does not state a release stage, so availability needs checking against the documentation.

Databricks contributed an UNREGISTER endpoint to the Apache Iceberg REST catalog specification. It removes a table from one catalog without touching its files and returns the latest metadata location, which another catalog then takes over with REGISTER, so a table has one managing catalog throughout the move. REGISTER and UNREGISTER on Unity Catalog are in private preview and available through the account team.

A Lakeflow pipeline's run-as identity can now be an account group, in addition to a user or a service principal. Pipeline updates run with the group's permissions, so a team can own an automated pipeline without it depending on one person's account.

A customer-managed key set on the system catalog can now be used to decrypt the statement_text and error_message fields in the query history system table. The release notes mark this generally available as of 2 October 2026. Teams that encrypt query history with their own key should read the docs section on reading encrypted fields.

OpenJev (Qwen3.5 4B) is now offered on Unity Gateway as a pay-per-token model service. The release notes say it is meant for yes/no, choice and scoring questions that return structured answers with probabilities, and that customers must check compliance with the applicable model terms.

The standard (default) mode of ai_extract now runs on a new model checkpoint that the release notes describe as 2.3x faster, with better overall quality and no extra cost. Queries that already use the default mode pick up the change without edits.

JDBC connections in Unity Catalog have reached general availability on serverless compute, Databricks Runtime 19 and above, and SQL warehouses 2025.40 and above, with SSL certificate support added. OAuth M2M authentication for these connections is still in Beta. Anyone who read the earlier Public Preview material should treat the GA scope above as the current one.

Lakeflow Connect now has a managed GitLab connector in Beta, ingesting repository, project, group, CI/CD and security data. It uses OAuth U2M authentication and supports incremental ingestion, UI and API pipeline authoring, and Declarative Automation Bundles. Workspace admins control access from the Previews page.

Supported external engines can now read Unity Catalog managed tables while ABAC policies, row filters and column masks are enforced by Databricks serverless compute. The feature is generally available; requirements, supported clients and serverless compute costs are listed on the linked documentation page.

LTAP Direct Writes, which writes the initial load and full refreshes of a Lakebase synced table straight to the storage layer instead of through the compute endpoint, has left Beta. You opt in per synced table under Synchronization settings, and it works on Lakebase projects running Postgres 16, 17 or 18.

Databricks now ships native SQL, PySpark and Scala functions for IPv4 and IPv6 addresses and CIDR blocks: parsing, canonicalizing, inspecting, converting to binary, and ip_cidr_contains for containment joins. try_ variants return NULL on bad input instead of failing. They are generally available on Databricks Runtime 18 LTS and above.

Writing data back to Databricks from the Excel add-in or from Google Sheets now requires a workspace admin to turn on the "Allow spreadsheet writeback" security setting. Workspaces where it stays off will see writeback stop working, so admins who have spreadsheet users should check the setting.

ai_decide is a new AI Function in Beta that answers one or more questions about an input in a single call. For each question it returns a probability, one label from criteria you define, or a score on an ordered scale. The input can be plain text, a JSON object or array, or the VARIANT output of ai_parse_document or ai_extract. It runs from SQL for batch work and through a REST API for real-time use; the SQL function needs Databricks Runtime 15.4 LTS or above and is not available on Databricks SQL Classic. Typical uses are tagging, routing prompts to a model and judging agent answers.

Query tags are now generally available. You attach custom key-value tags to SQL workloads to group, filter and attribute cost on Databricks SQL warehouses and Lakehouse//RT compute. The tags show up in system.query.history and on the Query History page.

Lakeflow Jobs table update triggers can now watch OpenSharing objects and system tables, and this is generally available. A job can start when a shared object or a system table changes, instead of on a schedule. Anyone polling these tables on a timer can consider switching to a trigger.

Genie One can keep facts you give it, such as preferences and reporting conventions, and apply them in later conversations. Memories are private to each user and are not shared; you can ask Genie One to remember something or accept a suggestion it makes.

The Agent Bricks CLI (databricks-agentbricks) is a command-line tool in Beta for custom code agents. It scaffolds a project from a bundled framework template, runs it locally and deploys it to the Databricks agent runtime, with managed memory, sessions, tools and MLflow tracing set up from one authenticated command.

GPT-6.1 Sol from OpenAI can now be called through Unity Gateway using Foundation Model APIs. The release note adds that customers remain responsible for complying with the applicable terms of the model provider.

A new lakebase_tokenizer extension controls how Lakebase Search splits text into terms: normalization and accent stripping, your own stopword and synonym lists, and stemming. It produces standard tsvector values, so it works with to_tsvector, GIN indexes and the lakebase_bm25 index; install it after enabling Lakebase Search in project settings.

Grok 4.7 from xAI is now available on Unity Gateway and can be called through pay-per-token Foundation Model APIs. Customers remain responsible for complying with the applicable terms.

Lakebase Postgres now ships two search extensions, lakebase_vector for approximate nearest-neighbour search and lakebase_text for BM25 full-text search, generally available on AWS and Azure. Together they allow semantic, keyword and hybrid search in one SQL query next to operational tables, without a separate search system. The vendor's own benchmarks compare it favourably with pgvector; the post also says Databricks AI Search remains the better fit for managed retrieval with little tuning.

The 1Password Event Logs connector is in Beta in Lakeflow Connect. It uses the 1Password Events API to ingest audit events, item usages and sign-in attempts, and authenticates with an Events Reporting bearer token. Workspace admins control access from the Previews page.

The Akamai WAF connector is in Beta in Lakeflow Connect and ingests web application firewall security events from Akamai. It authenticates with Akamai EdgeGrid API client credentials. Workspace admins control access from the Previews page.

The CrowdStrike Falcon Event Stream connector is in Beta in Lakeflow Connect and ingests Falcon Event Stream events. It authenticates with OAuth 2.0 client credentials from a CrowdStrike API client that has Event streams Read permission. Workspace admins control access from the Previews page.

The Workday Activity Logging connector is in Beta in Lakeflow Connect. It incrementally ingests Workday user activity log events and authenticates with an OAuth refresh token. Workspace admins control access from the Previews page.

A managed Microsoft 365 Unified Audit Logs connector is in Beta in Lakeflow Connect. It ingests unified audit events from Microsoft Entra ID, Exchange, SharePoint and other Microsoft 365 workloads, uses OAuth machine-to-machine authentication, and loads incrementally.

A managed Okta System Logs connector is in Beta in Lakeflow Connect. It incrementally ingests audit and security events from the Okta System Log, authenticates with an SSWS API token, and supports Declarative Automation Bundles and API-based pipeline authoring.

The system.ai_gateway.usage system table now records coding-agent session metadata and the invocation source. New token-detail fields count tokens written to the 5-minute and 1-hour prompt caches, plus file and web search counts. The release notes label this Public Preview.

A Databricks app can now run across multiple instances behind one app URL instead of just one, for higher availability and concurrency. Deployments get zero-downtime updates and best-effort session affinity between instances. This horizontal scaling capability has reached general availability.

Deleting a Unity Catalog pipeline in the default publishing mode, from the UI or the Pipelines DELETE API, now leaves its materialized views, streaming tables and views in place. They can still be queried but cannot be updated until they move into an active pipeline. To remove the tables too, the API request must set the cascade field to true; Hive metastore pipelines are unaffected.

Genie Code can now orchestrate an end-to-end document processing pipeline, handling parsing, classification, extraction, and validation in one flow. Pointed at unstructured PDFs, contracts, or invoices, it checks extraction quality against labeled data and iterates to improve it.

Tags can now be applied to dashboards, notebooks, Databricks apps, and Genie Agents to organize workspace assets. The release is generally available and supports both governed and ungoverned key-value tags.

The ai_search function can now retrieve from Lakebase synced tables that have Lakebase Search enabled, as well as from AI Search indexes. A call adds a lakebase_search knowledge source, and it can be combined with vector_search sources in the same call. The release notes label this Beta.

In Beta, Lakebase records query plans, session history, wait events, schema-change history, health data and Postgres logs into the system.lakebase system tables with no setup. Records are kept for 7 days, and account and metastore admins can read the tables and grant access to others.

In Databricks Runtime 19, time travel can be used on qualifying Unity Catalog managed tables while ABAC row filter and column mask policies are enforced. The feature is in Beta, so learners should treat it as preview behaviour rather than generally available behaviour.

Databricks says you can now build governed data and AI apps without code. Genie App Builder authors apps in natural language, App Spaces enforce governance, and Serverless Micro Apps scale to zero when idle.

Session-scoped and Unity Catalog Scala UDFs can now access secrets managed in Unity Catalog. Session-scoped UDFs use the caller's permissions, while Unity Catalog Scala UDFs declare secrets in a SECRETS clause and use the function owner's permissions. Users must check the stated environment, Databricks Runtime, access mode, and SQL warehouse requirements before using this in a UDF.

Scalar and batch Unity Catalog Python UDFs are now generally available on all supported compute types. The release adds named handlers for row-at-a-time scalar UDFs, removes the five-UDF-call-per-query limit, and aligns scalar TIMESTAMP inputs with PySpark behavior. Some capabilities require environment_version 6 or above, and requirements vary by feature and compute type.

Starting November 18, 2026, Databricks will automatically add the variantShredding table feature to new tables with a VARIANT column when they are created on Databricks Runtime 17.3 or above. Variant shredding stores frequently accessed fields in a typed, columnar layout to improve query performance. Clusters on Databricks Runtime 15.4 cannot read from or write to those tables, so affected clusters should be upgraded to Databricks Runtime 16.4 or above before the change takes effect.

Organizations can now centrally configure coding agents, including models, MCP servers, skills, a default harness and models, and Smart Routing. The settings apply when an agent starts, or can be rolled out through mobile device management.

Unity Gateway adds coding-agent activity to unified tracing through the Unity Gateway CLI. Traces can show the tools and skills used alongside model requests in Govern > Traces or with Genie.

The Unity Gateway CLI (ug) can connect Claude Code, Codex, Pi, and other coding agents to Unity Gateway with one command. It handles authentication and configuration, adds approved MCP servers and skills, and shows spend in the terminal.

Databricks describes Unity Catalog Pages as a governed home for business knowledge used with Genie Ontology. Readers working with Genie Ontology or Unity Catalog Pages should treat Pages as part of where business context for agents is managed.

OpenAI GPT-6 Luna is now available on Unity Gateway. It can be accessed through Foundation Model APIs pay-per-token, and customers are responsible for complying with applicable terms.

OpenAI GPT-6 Sol is now available on Unity Gateway. It can be accessed through Foundation Model APIs pay-per-token, and customers are responsible for meeting the applicable terms.

Databricks added a Beta snapshot schedule API for Lakebase. It lets you read and set a root branch's automated snapshot cadences, including daily, weekly, or monthly schedules, and the retention period for each cadence.

Lakeflow Connect now has a Shopify connector in Beta. It ingests Shopify e-commerce data, including products, orders, customers, inventory, fulfillments, and discounts, into Databricks, and workspace admins can manage access from the Previews page.

A model provider service API key can now be backed by a customer-owned Unity Catalog secret. The service reads the secret at query time, so rotating the secret takes effect without changing the service.

The system.query.history system table is now generally available. It provides account-wide records for queries run on SQL warehouses or serverless compute, including execution status, duration metrics, and query source.

Lakebase Search is now generally available for Lakebase projects. It uses the lakebase_vector and lakebase_text Postgres extensions for vector search and BM25 full-text search in the same database, instead of using a separate search stack. Lakebase users who want this must enable Lakebase Search in project settings and install the extensions.

Configuring telemetry for Databricks Apps is now generally available. It collects traces, logs, and metrics using the OpenTelemetry protocol and stores them in Unity Catalog tables. After telemetry is enabled, Databricks captures system logs and usage events automatically, and app developers can add custom instrumentation with an OpenTelemetry SDK.

Base environments can now be used to manage Python dependencies on classic compute. The beta supports both Databricks-provided environments and custom workspace-level environments.

Unity Catalog now has Beta support for attaching an ABAC policy at the metastore level. A row filter, column mask, GRANT, or DENY policy attached there applies across every catalog in the metastore, including catalogs created later, instead of being replicated per catalog.

Genie One skills can now include multiple files and sub-folders. Each skill can use folders up to three levels deep and up to 50 files or folders, and skills can load text files only.

Databricks added Beta managed agent memory and managed agent sessions, two fully managed Lakebase-backed stores for agent state that work with agents built on any framework. Memory persists across conversations and supports semantic search; sessions store conversation history for the agent to read at the start of a turn and append to while it runs.

ABAC column masks can now use a single VARIANT function for ARRAY and MAP columns, in addition to STRUCT and other column types. This requires Databricks Runtime 19 or above, so readers using masks on ARRAY or MAP columns need to check their runtime before using it.

Databricks now deletes a service principal's OAuth client secret after 90 days without use. If that deleted secret is still needed, create a new OAuth secret.

The Unity Gateway API is generally available for managing model services, model provider services, and MCP services. Create, read, update, list, and delete operations are supported across Terraform, the Databricks CLI, and the listed Databricks SDKs; Declarative Automation Bundles support through the CLI is in Beta.

The blog introduces MATCH_RECOGNIZE as a SQL feature for detecting patterns across rows, described as similar to regex for rows. Learners working with event-style data, such as login attempts, should know the syntax exists before building pattern detection logic another way.

READ METADATA will be available by default for workspaces with the compliance security profile enabled in mid-October 2026. The privilege gives read-only access to object metadata that BROWSE does not show, including privilege grants, row filters and column masks, and ABAC policies, so users who inspect or debug access controls should understand the difference.

ABAC row filters and column masks now extend to views, in addition to tables. Account admins can enable the ABAC on Views preview from the account console, and the feature requires Databricks Runtime 19 or above.

Window measures in metric views now support unitless offsets and trailing or leading ranges on a consecutive integer order column. Authors can use a numeric index for non-calendar periods such as fiscal weeks or 4-4-5 periods.

Databricks added the counter_diff window function for time-series counter data. It calculates differences between consecutive cumulative counter values and handles counter resets, and is available in Databricks SQL and Databricks Runtime 19 and above.

In early October 2026, Databricks will automatically enable JAR tasks on serverless compute for workspaces with the compliance security profile enabled. Users can package Java or Scala Spark code in a JAR and run it as a JAR task in a Lakeflow Job without provisioning or managing clusters.

Databricks announced on-demand state repartitioning for Apache Spark Structured Streaming on Databricks. The entry is aimed at people running stateful Structured Streaming queries in production.

When a Genie response includes a chart, Slack replies now show the visualization as an image. This is Public Preview; users who installed the Databricks Genie app for Slack before must reinstall it to enable the feature.

Databricks can automatically capture a snapshot for eligible serverless compute job runs. In Public Preview, you can restore the Python variables and Spark session from that run into a new interactive notebook to debug a failure or explore results without rerunning the whole job.

DeepSeek V4.1 Flash is available on Unity Gateway with text and image inputs. Users can access it through Foundation Model APIs and should check supported regions and applicable terms before using it.

Sharing and reading foreign Iceberg tables with OpenSharing is now generally available. Providers can add Iceberg tables federated from foreign Iceberg catalogs through Lakehouse Federation to a share, including for recipients that use external Iceberg clients.

The Jobs API can now set `performance_target` to `STANDARD` for one-time runs. Apache Airflow's `DatabricksSubmitRunOperator` also supports this setting, so users who trigger one-time runs from Airflow can use it there too.

A provider can share Iceberg tables, schemas and tables that live in a federated catalog without copying anything into Databricks. The compute and storage cost of materialising them stays with the provider. Generally available.

Lakeflow pipelines can now create managed tables and write to them with one or more append flows. The Beta also adds import_checkpoint on an append flow to move an existing Structured Streaming workload into a pipeline without reprocessing the source.

Genie Code skills now appear in Genie One and are turned on by default; individual skills can be turned on or off on the customization page. Genie One also supports backend steering while it answers and PowerPoint .pptx uploads in a conversation. The entry marks Genie Code skills in Genie One and memory confirmation prompts as Beta.

Databricks added several dashboard authoring and viewing changes: relationship-scoped filters are in Public Preview, themes can be exported and imported, and line, pivot table, and path map visualizations have new controls. Genie Code can import more Tableau calculations and now prefers governed metric views when building dashboards, while Ask Genie uses page, widget, or selected chart context on published dashboards.

Genie One and Genie Code can now use Databricks-provided MCP connectors for Google Drive, Gmail, Google Calendar, Microsoft 365, Atlassian, Slack, and GitHub. The connectors let Genie search and act on those tools while answering questions, with each connection governed through Unity Catalog and Unity Gateway and each user authenticating separately.

The Databricks Excel Add-in is now generally available. It lets Microsoft Excel connect to a Databricks workspace to browse and import governed Unity Catalog tables and views, run SQL queries, build pivot tables, use custom functions, and write data back to Unity Catalog. Live query can also be enabled for pivot tables so results update as fields change.

Each Genie Agent now supports up to 50 tables, views, or metric views as data sources, increased from 30. Each Genie Agent can store up to 200,000 conversations, increased from 10,000, so people designing Genie Agents can plan for larger data scopes and longer conversation history.

You can now set a maintenance window on a continuous Lakeflow pipeline for platform updates that require a restart, including new runtime versions and serverless infrastructure maintenance. The window is a selected day, start hour, and time zone; Databricks starts maintenance only during that one-hour window and lets you configure it in the Jobs UI, the Pipelines UI, or the Jobs REST API.

An administrator sets a weekly window — day, start hour, timezone — in which the platform applies its updates to continuous pipelines, from the Jobs UI, the Pipelines UI or the Jobs API. Generally available.

The add-in browses Unity Catalog tables, runs SQL, builds pivot tables against a live query and writes data back into Unity Catalog. Generally available, and governed like anything else in the catalog.

Account admins can now set an account-wide retention period for supported system tables. The release note says the configurable range is 30 to 3,650 days.

In early October 2026, horizontal scaling for Databricks Apps will be automatically enabled for workspaces with the compliance security profile enabled. The feature runs an app across multiple instances behind one app URL for higher availability and concurrency.

The Anaplan connector is now available in Lakeflow Connect in Beta. It ingests audit trail events and user account records from an Anaplan tenant into Databricks, using an Anaplan CA certificate and paired private key for authentication. Workspace admins can control access from the Previews page.

Lakeflow Connect now has a Beta connector for Anysphere Organization data. It uses the Anysphere (Cursor) Organization Admin API to ingest organization members, groups, and group members as daily snapshots, and workspace admins can control access from the Previews page.

Lakeflow Connect now has a Beta connector for Celigo. It ingests Celigo integrator.io audit log events, including create, update, and delete operations on resources, and uses a Celigo integrator.io API token for authentication. Workspace admins can control access from the Previews page.

External secrets connect a Unity Catalog schema to AWS Secrets Manager. They appear in Unity Catalog as read-only securable objects, with values read from AWS Secrets Manager on demand and access governed by Unity Catalog privileges.

A schema can be connected to AWS Secrets Manager, and the secrets appear as read-only securables governed by Unity Catalog privileges. Beta, so the shape may still move.

Databricks SQL version 2026.36 is rolling out to the Current channel. The release note points readers to the features listed for version 2026.32, so learners following Current channel behavior should review that linked entry.

Lakeflow Connect will soon make incremental ingestion generally available for Salesforce formula fields. Today, those fields use a full snapshot on each pipeline run by default; after the change, full snapshots will no longer be the default.

The ai_search function now has a generate_citations option. When it is turned on, the answer can show which retrieved chunks the model used to support the generated response.

ABAC DENY policies are in Beta in Unity Catalog. They can explicitly deny the MANAGE ACCESS CONTROL privilege for specified principals, including object owners, and the denial takes precedence over grants.

Databricks SQL now has the time_bucket function, which returns the start of the fixed-width interval that contains a timestamp. Use it instead of date_trunc when the bucket width or bucket origin is not a calendar unit.

A DENY policy blocks MANAGE ACCESS CONTROL for the principals it names, and it always wins over a grant that would allow it. Beta. It is the first ABAC construct that can take something away rather than hand it out.

The function returns the start of the fixed-width interval a timestamp falls into, aligned to an origin you choose. Generally available, and it removes the date_trunc-and-arithmetic dance from most time-series grouping.

Zerobus Ingest can now target tables backed by default storage, not only ones with an external location. Public preview. Arrow Flight support for the same connector went generally available on 1 September.

OpenAI GPT-6 Astra is now available on Unity Gateway. The release notes say it can be used through Foundation Model APIs pay-per-token for reasoning, structured document processing, coding, agentic search, multimodal inputs and long-context workloads.

Data Classification will scan Unity Catalog views in addition to tables. View columns will be classified for sensitive data with the same detection engine used for tables, and the release note says this is a Beta feature.

Databricks users can now share metric views with users in other metastores or accounts by using OpenSharing. Anyone who teaches or studies metric views or OpenSharing should treat this as beta, not generally available.

The base environment for serverless and standard classic compute moved to version 6. Pinning an older one keeps working, but the default a new workload gets has changed.

AI Search can now index and filter struct and map columns. It can also run filter-only queries that return rows matching a filter without vector or keyword retrieval.

Model Serving now supports Google Gemini 3.8 Flash as a Databricks-hosted model. It can be accessed through Foundation Model APIs, and customers are responsible for complying with applicable terms.

Automatic change data feed computes row-level changes at query time with row tracking, without manually enabling change data feed on each Delta Lake or Apache Iceberg v3 table. It works with batch queries, Structured Streaming, and Delta Sharing, and requires Databricks Runtime 19 LTS or above with row tracking enabled.

Unity Gateway now supports Anthropic Claude Fable 5.1 as a Databricks-hosted model. Users can access it through Foundation Model APIs, and customers are responsible for meeting the applicable terms.

Databricks released maintenance updates for supported Databricks Runtime versions. The entry says the updates include bug fixes, security patches, and performance improvements for Databricks Runtime 18 LTS, 18.2, 18.1, 17.3 LTS, 16.4 LTS, 15.4 LTS, 14.3 LTS, and 13.3 LTS.

Genie Code can now run a prompt on a recurring schedule. Each run starts a full Genie Code session and creates a Genie chat with results that can be continued later. Users can create a scheduled task from chat or from the Schedules menu.

Genie One and Genie Agents can now use models served through OpenAI on Databricks when partner-powered AI features are enabled. Anyone using these features should check which hosting provider applies when the partner-powered AI features setting is enabled or disabled.

Genie One can now search the public web when answering questions that need current information, such as recent release notes, third-party documentation, or news. Its responses include links to the external sources it used.

Apache Arrow support for Zerobus Ingest in Lakeflow Connect is now generally available. Readers studying Zerobus Ingest should know that Arrow Flight support is a released option, not only a documentation example.

Automatic change data feed reached general availability in Databricks. It computes change data feed at read time from row tracking on Delta Lake tables and row lineage on Apache Iceberg v3 tables, so qualifying Unity Catalog tables can serve row-level changes.

Git Folder Serverless entered beta in Databricks. It uses one serverless compute resource shared by every notebook and file in a Git folder, with the folder Python environment managed by a root pyproject.toml instead of per notebook.

Auto CDF derives inserts, updates and deletes at query time from row tracking, so a table no longer has to have the change data feed enabled before the fact to be read as a stream of changes. Databricks measures MERGE and UPDATE about 15% faster with it. Generally available.

Genie One can search the open web and cite the sources it used, which puts information outside the lakehouse into an answer for the first time. Beta. Genie Code got the same thing in August, switched off by default.

ABAC GRANT policies are now generally available for models, model services, model provider services, Model Context Protocol services, and agent services. They grant Unity Catalog privileges to securable objects when their governed tags match a condition, instead of granting privileges on each object one by one.

The pipeline_events system table is in Beta and records Lakeflow pipelines event log entries for pipelines in an account. It includes lifecycle transitions, flow progress, data quality metrics, errors, and other operational data across pipelines and workspaces in a region.

Lakeflow pipeline events — lifecycle transitions, flow progress, data quality metrics and errors — are now collected in a system table covering every pipeline in a region, instead of one event log per pipeline. Beta.

Databricks added the beta ai_transcribe() SQL function to transcribe an audio file into text. It returns a VARIANT value with time-stamped segments, and can label each segment with a speaker when speaker diarization is enabled for the workspace.

Service principal OAuth secrets can now be restricted to specific API scopes, such as sql or jobs, instead of covering all APIs. Tokens created from a scoped secret cannot exceed those scopes, so administrators who create these secrets can limit what a leaked secret can access.

Model Serving now supports Zhipu AI GLM 5.3 as a Databricks-hosted text-only MoE model in supported US and Americas regions. It is available through Foundation Model APIs and supports function calling, structured output, coding, agentic tool use, always-on reasoning, and the stated token windows. Customers are responsible for checking the applicable terms.

Unity Gateway can now track and cap spend for external model providers, including Amazon Bedrock and Azure AI Foundry. To use it, an account admin enables External Model Spend in Budgets (Beta) on the account console Previews page and includes external model usage when creating a budget for the Unity Gateway resource type.

A skill published under the SKILL.md specification is a first-class object in Unity Catalog, with the same permissions and audit trail as a table. Beta, and the piece that was missing for governing what an agent is allowed to know how to do.

Spend against external providers such as Amazon Bedrock and Azure AI Foundry is tracked and can be capped at the gateway. Beta. It is the answer to the agent that quietly costs a fortune over a weekend.

The Lakeflow Designer August release adds a Unique operator for removing duplicate rows, COUNT DISTINCT in the Aggregate operator, and dynamic column selection in the Select operator. It also changes the visual data prep experience with Python preview mode, streaming Python output, operator enable and disable controls, lineage highlighting, Run all, exact row counts, results-table changes that can be applied to the canvas, more filter values, @ mentions, and visual canvas diffs in version history.

Dashboard authors can hide a page from the published dashboard while keeping it in the dashboard for authors. Dashboards also added path map visualizations and separate left, center, or right alignment for table column headers.

Query text now reads <REDACTED> in system tables and APIs unless an account admin adds you to the databricks_pii_access group. Anything that parsed query text out of system tables — a cost dashboard, a lineage script — stops working until someone is granted the unmasked view.

Three new privileges allow writing rows without allowing schema changes, which is what MODIFY has always bundled together. Beta, and the least-privilege answer to a grant most people give too broadly.