Jobs & Pipelines
ReleaseWorth evaluating
Lakeflow Designer adds Unique, COUNT DISTINCT, and canvas updates
From Databricks release notes (AWS)
The Lakeflow Designer August release adds a Unique operator for removing duplicate rows, COUNT DISTINCT in the Aggregate operator, and dynamic column selection in the Select operator. It also changes the visual data prep experience with Python preview mode, streaming Python output, operator enable and disable controls, lineage highlighting, Run all, exact row counts, results-table changes that can be applied to the canvas, more filter values, @ mentions, and visual canvas diffs in version history.
Learners using Lakeflow Designer need to know that common prep steps such as deduplication, distinct counts, preview-safe Python code, and version comparison now have built-in UI support.
What to re-read
More from Jobs & Pipelines
- ReleaseLakeflow Jobs show job-level query performance metrics and insights (GA)
- PreviewServerless job tasks can configure their environment from pyproject.toml (Beta)
- ReleaseGenie Code tasks in Lakeflow Jobs are generally available
- ReleaseLakeflow system tables for pipelines reach general availability