operationsAdvanced

Platform & Administration

Choose the right compute, read the Spark UI, tune the handful of settings that matter, and diagnose clusters that will not start or run out of memory.

0 of 16 done
Data Engineer Associate
16 of 16 written|~25 h
Steps through the path one concept at a time, in order.100%
01Compute
02Governing compute
03Performance
04Troubleshooting
01Choosing compute: all-purpose, job cluster, serverless, SQL warehouse
02Serverless compute
03Databricks Runtime and Photon
04Access modes: standard and dedicated
05Cluster policies
06Basic Spark tuning parameters
07Spark UI: skew, shuffle, and spill
08Liquid clustering
09Instance pools and autoscaling
10Partitioning, Z-order, and data skipping
11Diagnosing clusters: startup failures, libraries, out of memory
12Monitoring runs: states, run history, trends
13System tables
14Cost attribution and budgets
15Databricks Labs, the tools around the platform
16Query tags
Platform & governanceData engineering · LakeflowData warehousing & BI
Drag to pan · scroll to zoom · click a task to open it · the small button says what it covers
The same path as a list16 concepts
Stage 1 · Compute
Choosing compute: all-purpose, job cluster, serverless, SQL warehouseServerless computeDatabricks Runtime and PhotonAccess modes: standard and dedicated
Stage 2 · Governing compute
Cluster policies
Stage 3 · Performance
Basic Spark tuning parametersSpark UI: skew, shuffle, and spillLiquid clusteringInstance pools and autoscalingPartitioning, Z-order, and data skipping
Stage 4 · Troubleshooting
Diagnosing clusters: startup failures, libraries, out of memoryMonitoring runs: states, run history, trendsSystem tablesCost attribution and budgetsDatabricks Labs, the tools around the platformQuery tags

The operations half of the job. Read it after you have a pipeline of your own to look at, otherwise the Spark UI numbers mean nothing.

Full resources for this path

12 free
Newsletter

What changed, and what it means for you

Databricks renames and deprecates things often. Get the ones that actually change how you work, plus the concepts rewritten because of them. No more than the schedule below, and one click to leave.

Choose what you get5 of 43
Your stack · Platform & governance

Tick what you run, and untick “Everything worth knowing” to get only that.

Data engineering · Lakeflow
Data warehousing & BI
AI & ML
Your learning paths

Each one is a section of the letter, and you only get the ones you tick. Every email carries a link back to this list, so you can change it later without an account.

Double opt-in: we send one email to check the address is yours. Nothing else until you confirm.