pipelinesIntermediate
Data Engineering
Build production pipelines: ingest with Auto Loader, COPY INTO and Lakeflow Connect, transform with PySpark and SQL, orchestrate with Lakeflow Jobs, and ship it with bundles.
0 of 53 done
53 of 53 written|~60 h
01Ingest
02Transform
03Declare
04Orchestrate
05Operate
06Ship
FoundationsPlatform & governanceData engineering · Lakeflow
Drag to pan · scroll to zoom · click a task to open it · the small button says what it covers
The same path as a list53 concepts
Stage 1 · Ingest
Ingestion patterns: batch, streaming, incrementalCOPY INTOAuto LoaderLakeflow Connect: managed connectorsIngesting from JDBC and REST APIs in notebooksSemi-structured data: JSON, nested data, VARIANTStructured Streaming on DatabricksReading and writing Apache KafkaTrigger intervals in Structured StreamingWatermarks and stateful streamingZerobus IngestStage 2 · Transform
Medallion architecture: bronze, silver, goldColumns, rows, and DataFrame structureJoins and unions between DataFramesDeduplication and aggregationsGold objects: tables, views, materialized views, streaming tablesReading and writing DataFramesUpsert with MERGE INTOArbitrary sinks with foreachBatchData quality on Databricks, layer by layerDQX: data quality checks for PySparkStage 3 · Declare
Lakeflow pipelinesData quality: expectations and constraintsChange data capture with AUTO CDCChoosing SQL or Python for a pipelineThe pipeline event logSinks: writing out of a pipelineStage 4 · Orchestrate
Lakeflow Jobs, what a job isTasks, dependencies, and the job graphControl flow: retries, if/else, for each, run jobTriggers: schedule, file arrival, table update, continuousJob and task parameters, dynamic values, and task valuesRepair runs, retries, and notificationsServerless compute for jobsConcurrent runs, queueing, and the limits behind themContinuous jobsStage 5 · Operate
Monitoring runs: states, run history, trendsSpark UI: skew, shuffle, and spillOPTIMIZE, VACUUM, and file layoutSystem tablesPredictive optimizationDeletion vectorsTable history and transaction log checkpointsChange Data FeedTime travel and table historyIceberg on DatabricksData profiling and anomaly detectionThe longest path on the site and the one that maps almost one-to-one onto the Data Engineer Associate exam. Work it in order: every stage assumes the one before it.
If you have never opened a Databricks workspace, do Lakehouse Foundations first.
Full resources for this path
24 freeNothing of that kind on this path. Try the full list.