Product area
Jobs & Pipelines
Orchestration with Lakeflow Jobs (tasks, dependencies, triggers, parameters, repair) and declarative pipelines with Lakeflow pipelines (formerly Delta Live Tables).
Concepts
- 1Lakeflow Jobs, what a job isA job is the object Databricks uses to run work non-interactively: a set of tasks connected by dependencies, executed on a compute you choose, started by a trigger (manual, scheduled, on file arrival, on table update), and observed through runs…Beginner
- 2Tasks, dependencies, and the job graphDependencies tell a job in which order to run its tasks. The set of tasks and dependencies is a DAG (directed acyclic graph): a task starts only when every task it depends on has finished, and tasks with no dependency between them run in parallel.Intermediate
- 3Control flow: retries, if/else, for each, run jobA job's control flow is the set of mechanisms that decide whether, how many times, and over how many elements a task runs. Lakeflow Jobs has four of them:Intermediate
- 4Triggers: schedule, file arrival, table update, continuousA trigger is the rule that starts a job run without human intervention. Lakeflow Jobs offers four families of triggers, plus the manual trigger ("Run now", see jobs overview):Intermediate
- 5Serverless compute for jobsServerless compute for workflows is what runs a Lakeflow Jobs task when the task has no cluster attached. Databricks picks the instance types, the memory and the engine, turns autoscaling and Photon on for you, and keeps optimising the shape of the compute…Intermediate
- 6Job and task parameters, dynamic values, and task valuesA parameterized job is a job that receives values from the outside instead of hard-coding them in the code: the date to process, the target catalog, a check's threshold. Lakeflow Jobs gives you three tools, and the exam likes to mix them up:Intermediate
- 7Repair runs, retries, and notificationsA repair run resumes a failed run from the point where it broke: the platform reruns the tasks that didn't succeed and everything downstream of them, while keeping the already-completed tasks as-is. The run keeps the same runid; every repair adds a repairid…Intermediate
- 8Lakeflow pipelinesA declarative pipeline is a set of SQL or Python files where you declare the datasets and the query that produces each one. Instead of writing "read, transform, write, repeat," you write "this table is the result of this query," and the engine works out the…Intermediate
- 9Data quality: expectations and constraintsDatabricks offers two mechanisms for saying "this data must satisfy a rule":Intermediate
- 10Change data capture with AUTO CDCAUTO CDC is the API in Lakeflow pipelines (see pipelines overview) that takes a stream of change records and keeps a target streaming table in sync with them, as either SCD Type 1 (current state only) or SCD Type 2 (full history). You declare the target…Advanced
- 11Choosing SQL or Python for a pipelineA Lakeflow pipeline (see pipelines overview) can be written in SQL or in Python. Both interfaces compile to the same underlying dataflow graph, so for most data processing they are genuinely equivalent: the same streaming tables, the same materialized views…Intermediate
- 12Concurrent runs, queueing, and the limits behind themTwo settings decide what happens when a job is asked to start while it is already busy, or while the workspace is already full.Intermediate
- 13Sinks: writing out of a pipelineA sink is an output target for a pipeline flow that is not a dataset the pipeline manages. By default every flow in a Lakeflow pipeline (see pipelines overview) writes to a streaming table or a materialized view in Unity Catalog. A sink sends the same stream…Advanced
- 14The pipeline event logThe event log is a Delta table that every Lakeflow pipeline writes for itself. One row per event, with a timestamp, a level, an eventtype, and a details column holding a JSON payload whose shape depends on that event type. The graph you stare at after a…Advanced
- 15Continuous jobsContinuous mode is the trigger family in jobs triggers that does not wait for anything. Instead of starting a run at a time or on an event, the scheduler keeps one run alive: when the run ends, for whatever reason, a new run starts. Databricks recommends it…Intermediate
News
No news for this area.