Jobs & Pipelines
PreviewWorth evaluating
The pipeline event log becomes a system table
From Databricks release notes
Lakeflow pipeline events — lifecycle transitions, flow progress, data quality metrics and errors — are now collected in a system table covering every pipeline in a region, instead of one event log per pipeline. Beta.
What to re-read
More from Jobs & Pipelines
- ReleaseLakeflow Jobs show job-level query performance metrics and insights (GA)
- PreviewServerless job tasks can configure their environment from pyproject.toml (Beta)
- ReleaseGenie Code tasks in Lakeflow Jobs are generally available
- ReleaseLakeflow system tables for pipelines reach general availability