pipelinesIntermediate

Data Engineering

Build production pipelines: ingest with Auto Loader, COPY INTO and Lakeflow Connect, transform with PySpark and SQL, orchestrate with Lakeflow Jobs, and ship it with bundles.

Data Engineer Associate

0/52|52 written|~60 h

To doIn progressDonePlannedClick a concept to open its cardSwipe the map, tap a concept

Stage 1 · Ingest

Ingestion patterns: batch, streaming, incremental COPY INTO Auto Loader Lakeflow Connect: managed connectors Ingesting from JDBC and REST APIs in notebooks Semi-structured data: JSON, nested data, VARIANT Structured Streaming on Databricks Reading and writing Apache Kafka Trigger intervals in Structured Streaming Watermarks and stateful streaming Zerobus Ingest

Stage 2 · Transform

Medallion architecture: bronze, silver, gold Columns, rows, and DataFrame structure Joins and unions between DataFrames Deduplication and aggregations Gold objects: tables, views, materialized views, streaming tables Reading and writing DataFrames Upsert with MERGE INTO Arbitrary sinks with foreachBatch Data quality on Databricks, layer by layer DQX: data quality checks for PySpark

Stage 3 · Declare

Lakeflow pipelines Data quality: expectations and constraints Change data capture with AUTO CDC Choosing SQL or Python for a pipeline The pipeline event log Sinks: writing out of a pipeline

Stage 4 · Orchestrate

Lakeflow Jobs, what a job is Tasks, dependencies, and the job graph Control flow: retries, if/else, for each, run job Triggers: schedule, file arrival, table update, continuous Job and task parameters, dynamic values, and task values Repair runs, retries, and notifications Serverless compute for jobs Concurrent runs, queueing, and the limits behind them Continuous jobs

Stage 5 · Operate

Monitoring runs: states, run history, trends Spark UI: skew, shuffle, and spill OPTIMIZE, VACUUM, and file layout System tables Predictive optimization Deletion vectors Table history and transaction log checkpoints Change Data Feed Time travel and table history Iceberg on Databricks Data profiling and anomaly detection

Stage 6 · Ship

Declarative Automation Bundles and the Databricks CLI Bundles: variables, targets, and per-environment overrides Secrets and credentials The CLI and the SDKs Bundles in a CI/CD pipeline

The longest path on the site and the one that maps almost one-to-one onto the Data Engineer Associate exam. Work it in order: every stage assumes the one before it.

If you have never opened a Databricks workspace, do Lakehouse Foundations first.

Resources for this path

Courses, books and repos that cover the whole map go here. None have been added yet; per-concept resources appear on each concept page.