prerequisitesBeginner

SQL & Python Foundations

The SQL and PySpark you need before any other path: the Spark dialect, types, joins, windows, MERGE, and the DataFrame API as it differs from pandas.

Data Engineer Associate

0/15|15 written|~25 h

To doIn progressDonePlannedClick a concept to open its cardSwipe the map, tap a concept

Stage 1 · SQL on Databricks

Spark SQL, the dialect Types and casting Joins and set operations

Stage 2 · SQL that does work

Window functions MERGE, UPDATE, DELETE on Delta SQL scripting and stored procedures Pipe syntax for queries AI functions in SQL

Stage 3 · Python and PySpark

PySpark versus pandas Python in notebooks: dbutils, widgets, modules Reading and writing DataFrames

Stage 4 · Transforming data

Columns, rows, and DataFrame structure Joins and unions between DataFrames Deduplication and aggregations UDFs and when not to write one

Every other path assumes this one. If you already write SQL for Postgres or pandas for analysis, you still want the first stage: the dialect is close enough to be misleading, and the differences that matter (ANSI mode, LEFT SEMI, QUALIFY, MERGE, lazy evaluation) are exactly the ones that bite in production.

Work it in order and you will be able to read every code block on the rest of the site without stopping. Skip to Lakehouse Foundations if you want the platform picture first and are happy to come back for the syntax.

Resources for this path

Courses, books and repos that cover the whole map go here. None have been added yet; per-concept resources appear on each concept page.