Product area

Compute

Clusters, serverless, SQL warehouses, policies, pools, runtimes, Photon: how to choose compute and how to diagnose problems.

DE Associate DE Professional Lakehouse Foundations Data Engineering Platform & Administration

Concepts

9 in this area
  1. 1Choosing compute: all-purpose, job cluster, serverless, SQL warehouseCompute is the set of resources that runs your code. On Databricks it isn't one single thing: interactive notebooks, scheduled jobs, SQL queries, and pipelines each have their own compute type, with different startup times, costs, and limits. The exam doesn't…
  2. 2Serverless computeServerless compute is Databricks running your notebook, job, or pipeline on infrastructure it owns, in its own account rather than yours. No cluster to size, no instance type to pick, no autotermination timer: you submit code, capacity attaches in seconds…
  3. 3Databricks Runtime and PhotonDatabricks Runtime is the versioned software image a classic cluster boots: an Apache Spark build, the JVM, OS packages, GPU drivers where relevant, and curated Python/Java libraries, all tested together. You pick a version — 16.4.x-scala2.12, say — when…
  4. 4Cluster policiesA cluster policy is a JSON document, attached in the admin console, restricting what a user or group can put into a cluster configuration: instance types, whether autoscaling is mandatory, the max DBU-per-hour spend, which tags must be present. Instead of…
  5. 5Instance pools and autoscalingAn instance pool is a set of VMs Databricks keeps idle and ready, sitting between "not provisioned" and "attached to a cluster." When a cluster is created against a pool, its driver and workers claim nodes straight from that idle set instead of asking the…
  6. 6Basic Spark tuning parametersSpark exposes hundreds of configuration properties. Four families explain most of the performance issues in an ETL job: how many partitions a shuffle produces, how much parallelism low-level operations get, how much memory the driver and executors have, and…
  7. 7Spark UI: skew, shuffle, and spillThe Spark UI is Apache Spark's diagnostic interface, reachable from a cluster's Spark UI tab or from a task's detail view in a run (see runs monitoring). It shows how Spark broke your code down into job → stage → task, and, for each stage, how metrics are…
  8. 8Diagnosing clusters: startup failures, libraries, out of memoryA classic cluster (see compute options) can fail at three distinct points: before it starts (the cloud can't provision the machines, an init script exits with an error, a policy blocks the configuration), while loading libraries (conflicting versions across…
  9. 9Access modes: standard and dedicatedAccess mode is the setting on a classic compute resource that decides who may attach to it and what data they can reach through it. Every all-purpose and job compute resource has one. In the UI it sits under Advanced; in the API it is datasecuritymode.

Also covered here

3 filed under another area

News

DataFrame caching on serverless compute enters Public Preview23 Oct 2026PreviewDatabricks Runtime 19 hides row filter and column mask metrics in Spark Connect19 Oct 2026ReleasePrivate Network Gateway for serverless: Private Preview on AWS, Public Preview on Azure14 Oct 2026PreviewPrivate Link from serverless compute to AWS-managed resources is generally available12 Oct 2026ReleaseDatabricks Runtime 10.4 LTS ends on 2026-11-011 Nov 2026DeprecatedVariant shredding will be added by default to new VARIANT tables22 Sept 2026ReleaseBase environments are in beta on classic compute17 Sept 2026PreviewJAR tasks on serverless compute are coming to compliance-profile workspaces14 Sept 2026ReleaseEnvironment version 6 is the new serverless base3 Sept 2026ReleaseMaintenance updates are available for supported Databricks Runtime versions1 Sept 2026Release

Resources