Spark UI: skew, shuffle, and spill

What it is

The Spark UI is Apache Spark’s diagnostic interface, reachable from a cluster’s Spark UI tab or from a task’s detail view in a run (see Monitoring runs: states, run history, trends). It shows how Spark broke your code down into job → stage → task, and, for each stage, how metrics are distributed across tasks. Three patterns explain most slow stages: skew (a few tasks holding much more data than the rest), excessive shuffle (data moved across the network between executors), and spill (not enough execution memory, so data gets written to disk).

Why it exists

A DataFrame is declarative: you write a join and Spark decides how to execute it. When it’s slow, the code doesn’t tell you why. The Spark UI shows what actually happened: how many tasks, how much they read, how long the slowest one took. Without these metrics, tuning (see Basic Spark tuning parameters) is just guessing.

How it works

Job, stage, task

An action (write, count, display) creates a job. Spark cuts it into stages at every shuffle boundary (join, groupBy, repartition, window). Each stage runs as N tasks in parallel, one per partition. The Jobs tab shows the timeline; the Stages tab lists stages with their duration and volumes; a stage’s detail view has the Summary Metrics table.

Reading the summary metrics

For each metric, the table reports min, 25th percentile, median, 75th percentile, max across the stage’s tasks. The signal isn’t the absolute value but the shape of the distribution.

Rule of thumb from the docs: if the duration’s max exceeds the 75th percentile by more than 50%, suspect skew.

The three symptoms

Skew. A key (“unknown” customer, NULL, a country that accounts for half the dataset) ends up in a single partition. Most tasks finish right away, one works for minutes, and the whole stage waits on it. In the UI: low median, very high max, and the same imbalance in Shuffle Read Size.

Shuffle. Moving data between executors is the most expensive phase: serialization, network, writes. A large Shuffle Write in one stage followed by a stage with many small partitions points to repeated joins and aggregations, or an unsuitable partitioning scheme.

Spill. When a task’s execution memory isn’t enough for its partition’s sort or hash, Spark writes to disk (Spill (Disk)) the data it was holding in memory (Spill (Memory), the deserialized size). A nonzero value means the task did the work twice. Spill and skew often go together: the bloated task is the one that spills.

Fixes

Joins and their strategies are covered in Joins and unions between DataFrames.

Example

A join stage has 200 tasks. Summary metrics:

Reading it: 199 tasks finish in under a minute, one takes ten; that task reads 5 GB against a median of 60 MB, and it spills. This is skew on the join key, with spill as a consequence. The fix isn’t a bigger cluster (that would only help one task) but changing how the data is distributed:

from pyspark.sql import functions as F

# 1. check which key is heavy
orders.groupBy("customer_id").count().orderBy(F.desc("count")).show(5)

# 2a. if the "customers" side is small: broadcast
res = orders.join(F.broadcast(customers), "customer_id")

# 2b. otherwise, salting: 16 sub-keys for the large side, explode the small side
n = 16
orders_s    = orders.withColumn("salt", (F.rand() * n).cast("int"))
customers_s = customers.withColumn("salt", F.explode(F.array([F.lit(i) for i in range(n)])))
res = orders_s.join(customers_s, ["customer_id", "salt"]).drop("salt")

-- the same broadcast in SQL
SELECT /*+ BROADCAST(c) */ o.*, c.segment
FROM orders o JOIN customers c ON o.customer_id = c.customer_id;

Common mistakes

• Adding workers to a skewed stage: the slow task is still just one task, and it’s still slow; the cost just goes up.
• Looking only at the stage’s total duration and not the distribution: a 10-minute stage with 200 uniform tasks is a volume problem, not skew.
• Increasing spark.sql.shuffle.partitions to fix skew: more partitions don’t split a single key apart.
• Ignoring a “small” spill: it signals that tasks are already at the edge of their memory budget, and the stage will collapse the next time data volume grows.
• Forcing a broadcast of a table that doesn’t fit in memory: you trade a slow stage for a driver out of memory (see Diagnosing clusters: startup failures, libraries, out of memory).