operationsAdvanced

Platform & Administration

Choose the right compute, read the Spark UI, tune the handful of settings that matter, and diagnose clusters that will not start or run out of memory.

Data Engineer Associate
0/16|16 written|~25 h
To doIn progressDonePlannedSwipe the map, tap a concept
Stage 1 · Compute
Choosing compute: all-purpose, job cluster, serverless, SQL warehouseServerless computeDatabricks Runtime and PhotonAccess modes: standard and dedicated
Stage 2 · Governing compute
Cluster policies
Stage 3 · Performance
Basic Spark tuning parametersSpark UI: skew, shuffle, and spillLiquid clusteringInstance pools and autoscalingPartitioning, Z-order, and data skipping
Stage 4 · Troubleshooting
Diagnosing clusters: startup failures, libraries, out of memoryMonitoring runs: states, run history, trendsSystem tablesCost attribution and budgetsDatabricks Labs, the tools around the platformQuery tags

The operations half of the job. Read it after you have a pipeline of your own to look at, otherwise the Spark UI numbers mean nothing.

Resources for this path

Courses, books and repos that cover the whole map go here. None have been added yet; per-concept resources appear on each concept page.