Product area
Compute
Clusters, serverless, SQL warehouses, policies, pools, runtimes, Photon: how to choose compute and how to diagnose problems.
Concepts
- 1Choosing compute: all-purpose, job cluster, serverless, SQL warehouseCompute is the set of resources that runs your code. On Databricks it isn't one single thing: interactive notebooks, scheduled jobs, SQL queries, and pipelines each have their own compute type, with different startup times, costs, and limits. The exam doesn't…Beginner
- 2Serverless computeServerless compute is Databricks running your notebook, job, or pipeline on infrastructure it owns, in its own account rather than yours. No cluster to size, no instance type to pick, no autotermination timer: you submit code, capacity attaches in seconds…Intermediate
- 3Databricks Runtime and PhotonDatabricks Runtime is the versioned software image a classic cluster boots: an Apache Spark build, the JVM, OS packages, GPU drivers where relevant, and curated Python/Java libraries, all tested together. You pick a version — 16.4.x-scala2.12, say — when…Intermediate
- 4Cluster policiesA cluster policy is a JSON document, attached in the admin console, restricting what a user or group can put into a cluster configuration: instance types, whether autoscaling is mandatory, the max DBU-per-hour spend, which tags must be present. Instead of…Intermediate
- 5Instance pools and autoscalingAn instance pool is a set of VMs Databricks keeps idle and ready, sitting between "not provisioned" and "attached to a cluster." When a cluster is created against a pool, its driver and workers claim nodes straight from that idle set instead of asking the…Intermediate
- 6Basic Spark tuning parametersSpark exposes hundreds of configuration properties. Four families explain most of the performance issues in an ETL job: how many partitions a shuffle produces, how much parallelism low-level operations get, how much memory the driver and executors have, and…Intermediate
- 7Spark UI: skew, shuffle, and spillThe Spark UI is Apache Spark's diagnostic interface, reachable from a cluster's Spark UI tab or from a task's detail view in a run (see runs monitoring). It shows how Spark broke your code down into job → stage → task, and, for each stage, how metrics are…Intermediate
- 8Diagnosing clusters: startup failures, libraries, out of memoryA classic cluster (see compute options) can fail at three distinct points: before it starts (the cloud can't provision the machines, an init script exits with an error, a policy blocks the configuration), while loading libraries (conflicting versions across…Intermediate
- 9Access modes: standard and dedicatedAccess mode is the setting on a classic compute resource that decides who may attach to it and what data they can reach through it. Every all-purpose and job compute resource has one. In the UI it sits under Advanced; in the API it is datasecuritymode.Intermediate
News
No news for this area.